Introduction#

I read 『生成AIアプリケーション評価入門』 (roughly “Introduction to Evaluating Generative AI Applications”, Shinsuke Matsuki, Gijutsu-Hyoronsha) — a Japanese book. It starts from classic evaluation metrics like the confusion matrix and BLEU / ROUGE, moves through the ISO/IEC 25059 quality model and the OWASP LLM Top 10, and finishes with a hands-on part that uses DeepEval and DeepTeam for evaluation and red teaming. As a systematic introduction it was easy to read and not too thick, which made it just right for getting the big picture of evaluation.

Shinsuke Matsuki / Gijutsu-Hyoronsha (Japanese)

生成AIアプリケーション評価入門

One book that takes you from evaluation-metric basics through quality models and the OWASP LLM Top 10 to hands-on DeepEval / DeepTeam. Easy to read and not too thick — a good systematic introduction. Tools move fast, so I recommend learning the concepts from the book and then checking the current version in the official docs.

View on Amazon →

* Contains Amazon Associates links

After finishing the book, two things bugged me. First, how far can DeepEval and DeepTeam actually go, and where are their weak spots? Second, are there other options? DeepEval can’t be the only evaluation tool out there, and without comparing, I can’t tell where DeepEval stands.

So I dug into DeepEval and DeepTeam through their official docs and the source code of the published packages, and then went exploring for alternatives. As I searched, something caught my eye before any feature differences did: this industry was being reshuffled at a furious pace in 2026. Roughly half of the major tools were acquired, shut down their service, or stalled in development within the past year.

Note that I haven’t run DeepEval / DeepTeam myself. This is a summary of what I learned from the book plus the official docs, PyPI, GitHub, and source code. Versions, star counts, and prices are all as of 2026-10-05.

Evaluation splits into two tracks#

To set things up first: “evaluation” of GenAI apps broadly splits into two. The book also covers DeepEval and DeepTeam in separate chapters.

Both are open source from the same company, Confident AI, and DeepTeam depends on DeepEval at runtime. Below, I dig into both first, then look at alternatives for each.

Digging into DeepEval#

1
# test_app.py → `deepeval test run test_app.py`
2
from deepeval import assert_test
3
from deepeval.metrics import GEval, AnswerRelevancyMetric, FaithfulnessMetric
4
from deepeval.test_case import LLMTestCase, SingleTurnParams
5

6
correctness = GEval(
7
    name="Correctness",
8
    criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
9
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
10
    threshold=0.5,
11
)
12

13
def test_refund():
14
    tc = LLMTestCase(
15
        input="What if these shoes don't fit?",
16
        actual_output="You have 30 days to get a full refund at no extra cost.",
17
        expected_output="We offer a 30-day full refund at no extra costs.",
18
        retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."],
19
    )
20
    assert_test(tc, [correctness, AnswerRelevancyMetric(threshold=0.7), FaithfulnessMetric()])

1
from deepeval.tracing import observe
2
from deepeval.dataset import EvaluationDataset, Golden
3
from deepeval.metrics import TaskCompletionMetric
4

5
@observe()
6
def agent(q): ...
7

8
ds = EvaluationDataset(goldens=[Golden(input="Plan a two-day trip to Paris")])
9
for g in ds.evals_iterator(metrics=[TaskCompletionMetric()]):
10
    agent(g.input)

Digging into DeepTeam#

1
from deepteam import red_team
2
from deepteam.vulnerabilities import Bias, PIILeakage
3
from deepteam.attacks.single_turn import PromptInjection, ROT13
4
from deepteam.attacks.multi_turn import LinearJailbreaking
5
from deepteam.test_case import RTTurn
6

7
async def model_callback(input: str, turns: list[RTTurn] | None = None):
8
    reply = await my_app.generate(input)            # the app under test
9
    return RTTurn(role="assistant", content=reply)  # returning a plain str also works
10

11
ra = red_team(
12
    model_callback=model_callback,
13
    vulnerabilities=[Bias(types=["race", "gender"]), PIILeakage(types=["direct_disclosure"])],
14
    attacks=[PromptInjection(weight=2), ROT13(), LinearJailbreaking(num_turns=3)],
15
    simulator_model="gpt-4o-mini", evaluation_model="gpt-4o",
16
    attacks_per_vulnerability_type=2,
17
    target_purpose="Retail bank support bot",
18
)
19
print(ra.overview.to_df())
20
ra.save(to="./results/")
21

22
# To run with a standards preset (cannot be combined with vulnerabilities)
23
# from deepteam.frameworks import OWASPTop10
24
# red_team(model_callback=model_callback, framework=OWASPTop10(categories=["LLM_01"]))

What else is there? Quality evaluation alternatives#

Now for the exploration part. First, quality evaluation tools that can stand in for DeepEval.

The big three clouds also have managed evaluation services (AWS Bedrock / AgentCore Evaluations, Azure Foundry Evaluation, Google Gen AI Evaluation)³⁷³⁸³⁹. I left OpenAI Evals out of the table because its API shuts down on 2026-11-30⁴⁰.

Here’s how the main ones relate to DeepEval.

• Ragas: the library that became the common vocabulary for RAG metrics; Langfuse, MLflow, Braintrust and others pull in Ragas metrics. But there have been no releases since 2026-01, and the repository moved²⁶. Depending on it alone is risky
• promptfoo: a config-driven tool that runs “prompts × models × test cases” from YAML. It handles both evaluation and red teaming, making it the closest competitor to DeepEval + DeepTeam. Assumes Node.js²⁷
• Inspect AI: built by the UK AI Security Institute. Runs agents in sandboxes and ships 200+ benchmarks²⁸⁴¹. Good for reproducibly measuring model/agent capabilities and safety; not suited to app-level RAG quality evaluation
• Langfuse: the observability platform with the largest OSS community. Since 2025-06, all features — including LLM-as-judge, annotation queues, and experiments — are MIT, and a Tokyo region launched in 2026-04³⁰⁴². Self-hosting means running ClickHouse, Postgres, Redis, and S3
• MLflow GenAI: a hub that can run DeepEval, Ragas, Phoenix, and TruLens scorers together inside mlflow.genai.evaluate()⁴³
• Opik: the most feature-complete OSS platform under Apache-2.0. Paid Cloud plans start cheap at $19/month³⁴⁴⁴
• Braintrust: a SaaS with a polished experiment-comparison UI and CI integration that comments scores on PRs and gates merges⁴⁵

Lined up like this, a division of labor emerges: DeepEval is strong at “unit tests during development”, while Langfuse, Opik, LangSmith, and Braintrust are strong at “production tracing and monitoring”. Since DeepEval’s OSS edition has no production monitoring, DeepEval and Langfuse / Opik are partners to combine rather than competitors.

What else is there? Red teaming alternatives#

Next, alternatives to DeepTeam.

How each differs from DeepTeam, in a line or two:

• promptfoo: ahead on coverage (157 plugins), standards support, and CI / web UI maturity⁴⁶. However, its advanced attack generation uses promptfoo’s (now OpenAI’s) remote inference by default, and the free tier is capped at 10k probes per month⁵¹. If you don’t want data leaving your environment, want to stay in Python, or don’t want to lean on a particular vendor, DeepTeam becomes a candidate
• PyRIT: a toolkit for experts to hand-build attack campaigns. Strong at multimodal and advanced multi-turn attacks, but no standards presets and more code to write⁴⁷. DeepTeam’s strength is that it “just runs”
• garak: the “Nmap for LLMs”, with a huge set of static probes, strong at auditing a model on its own⁴⁸. For app-level evaluation including RAG context and tool calls, DeepTeam fits better
• Giskard v3: handles quality evaluation and security scanning in one, and can call DeepTeam and garak as scan generators⁴⁹. It was just rewritten from scratch for agents in 2026-08

While searching, I found the industry being reshuffled everywhere#

As I researched the alternatives one by one, I noticed that “which company owns this tool now?” kept changing. Lined up, it looks like this.

Within a single year, in every area — quality evaluation, observability, red teaming, guardrails — major players were either acquired or wound down. Three things stood out to me.

First, OpenAI is shutting down its own evaluation service (OpenAI Evals) and pointing users to promptfoo, which it acquired, as the migration path⁴⁰⁵³. A company that builds models now owns a tool for evaluating models. promptfoo is a tool that lets you compare non-OpenAI models side by side, so I want to keep an eye on what happens to its neutrality.

Second, OSS observability is being absorbed by big data-infrastructure and monitoring players. Langfuse went to ClickHouse (the company behind the database Langfuse itself uses as its backend), Arize to Dynatrace, Galileo to Cisco (Splunk). LLM tracing is becoming one feature of existing monitoring stacks.

Third, turnover on the defensive side (guardrails). Here’s what the tools that plug holes found by red teaming in production look like:

It felt like stepping outside the tools I’d learned from the book only to find the ground itself moving. This table will probably look different again in six months, so please read the state described here as a snapshot from 2026-10.

Recommendations by use case#

Summarizing the research’s recommendations:

Building an OSS-only stack around DeepEval and DeepTeam looks like this:

1
Development : DeepEval (regression tests on pytest, run on every PR)
2
Pre-release : DeepTeam (red_team with OWASP LLM / Agentic presets, deepteam scan on the diff)
3
              + garak (static probes per model / endpoint)
4
Production  : Langfuse (tracing, online evaluation, annotation)
5
              + NeMo Guardrails or DeepTeam Guardrails
6
Feedback    : Build Goldens from production traces and feed them back into DeepEval datasets

Paying for Confident AI lets you see development, production, and red teaming in one place, but even Starter is $200 a month, and the red-teaming module is Enterprise++¹¹. If you build with OSS, handing production to Langfuse or Opik is the cost-sensible choice.

Notes for non-English (Japanese) use#

I write from Japan, so this section is about Japanese, but most of it applies to any non-English app.

• Judge prompts mostly assume English. Every tool runs in Japanese, but judging quality isn’t officially validated. Ragas can adapt its prompts to the target language with adapt()⁶⁵; with DeepEval you can write G-Eval criteria in the target language, or pin down the judging steps with DAG. Either way, check agreement between the judge and a few dozen to a few hundred human labels in your language before setting thresholds
• Cloud safety evaluation targets English. Azure Foundry’s safety evaluators aren’t available in Japanese regions, and English is stated as optimal⁶⁶
• If you need data to stay in Japan, options are Langfuse Cloud Japan (Tokyo)⁴², AWS AgentCore Evaluations (Tokyo)³⁸, Azure evaluation (Japan East / West, excluding safety evaluators)⁶⁶, and Confident AI Enterprise¹¹. LangSmith’s APAC region is Sydney; there’s no Japan region⁶⁷
• Red-team attack prompts are English-centric. DeepTeam’s Multilingual attack switches languages to slip past defenses; it is not an attack set for Japanese-language apps. For those, you need to add CustomVulnerability or your own attack prompts

A local LLM as the judge might cut API costs#

From here on, it’s my own idea, outside the research.

DeepEval’s top weakness was judge cost. QAG-style metrics call the judge many times per case, so cost grows as “test cases × metrics × calls per metric”. If you run it in CI on every PR, that multiplication runs every time.

Meanwhile, local LLMs have come a long way. DeepEval can use local OpenAI-compatible servers like Ollama or vLLM as the judge, and DeepTeam reuses DeepEval’s model layer. So for a company that already owns hardware like a DGX Spark, using an open frontier model that barely fits as the judge might cut API costs.

I have three reasons for thinking so.

• The workload shape fits. A judge’s job is to read fairly long retrieval context and answers and return a verdict with a short reason. In my post on putting Qwen3.8 Flash Next on two DGX Sparks into real work (Japanese), I wrote that jobs like code review — “read a long prompt, return something short” — feel fast even locally. What made the difference was TTFT, prefill, and aggregate throughput under parallel load, and evaluation fired in parallel with deepeval test run -n uses exactly those
• You can pin the judge. The weakness of API judges was scores shifting when the model is updated. Local weights don’t change unless you update them, so as a yardstick for release decisions, it might actually be more stable
• Data doesn’t have to leave. Evaluation data tends to include production inputs and outputs, and there are probably cases where evaluation never happened because the data couldn’t leave the company. That’s the same pattern as the “work we couldn’t do before because we couldn’t send it out” I described in the post above

There are preconditions, though. As in my post calculating the economics of two DGX Sparks (Japanese), this isn’t about buying new hardware to save on API fees. It works when the hardware is already there and you can run evaluations in its idle time. Also, if you switch the judge to an open model, calibration — checking agreement with human labels — is mandatory. Then again, that step is needed anyway with API judges if you’re evaluating in a non-English language, so it’s less extra work than a change in what the work you’d do anyway is applied to.

Adoption checklist#

• Choice of judge model and cost estimate (test cases × metrics × calls per metric)
• Judge calibration (agreement with human labels), and whether to pin the judge model
• Telemetry opt-out (DEEPEVAL_TELEMETRY_OPT_OUT, DEEPTEAM_TELEMETRY_OPT_OUT, RAGAS_DO_NOT_TRACK, etc.)
• Version pinning. DeepEval has many breaking changes, so pin it in CI, e.g. deepeval==4.2.x
• For DeepTeam, check defaults in the code, not the docs
• Acquisition / shutdown status of the tools you use (promptfoo, Langfuse, Phoenix, Weave, OpenAI Evals)

Wrapping up#

I learned about DeepEval and DeepTeam from a book, dug into them, and then went exploring.

Digging in, I found that DeepEval is still the first choice for putting evaluation into CI in Python, with the broadest metric range in open source. DeepTeam is a reasonable, easy red-teaming option for DeepEval users, with OWASP and EU AI Act presets ready to use. On the other hand, neither has production monitoring, so if you include production, pairing them with Langfuse or Opik is the realistic choice.

Exploring, I found that this field was reshuffled hard in 2026. promptfoo to OpenAI, Langfuse to ClickHouse, Arize to Dynatrace, Galileo to Cisco. OpenAI Evals stops at the end of next month. When choosing an evaluation tool, beyond the feature comparison table, be sure to check “who owns it now, and does it look likely to stay vendor-neutral?” One more thing: for judge costs, companies that own hardware like a DGX Spark might be able to hand the job to a local open model, and that’s something I’d like to try. DeepEval / DeepTeam are no exception to the reshuffle, either — nobody knows what will happen to Confident AI.

An introductory book gave me a map, and when I stepped outside, the terrain was changing every month. Better not to throw the map away — just check where you are, often.

References#

1. DeepEval GitHub https://github.com/confident-ai/deepeval
2. DeepEval PyPI https://pypi.org/project/deepeval/
3. DeepEval Releases https://github.com/confident-ai/deepeval/releases
4. DeepEval Test Cases https://deepeval.com/docs/evaluation-test-cases
5. DeepEval LLM Tracing https://deepeval.com/docs/evaluation-llm-tracing
6. DeepEval Unit Testing in CI/CD https://deepeval.com/docs/evaluation-unit-testing-in-ci-cd
7. DeepEval Metrics Introduction https://deepeval.com/docs/metrics-introduction
8. DeepEval Classifiers https://deepeval.com/docs/classifiers-introduction
9. DeepEval Synthetic Data Generation https://deepeval.com/docs/synthetic-data-generation-introduction
10. DeepEval Integrations https://deepeval.com/integrations
11. Confident AI Pricing https://www.confident-ai.com/pricing
12. DeepEval Data Privacy https://deepeval.com/docs/data-privacy
13. AgentCore Third-party Evaluators https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/third-party-evaluators.html
14. Rubin Lake Technology Radar: DeepEval https://www.rubinlake.com/en/technology-radar/data-platforms-and-mlops/deepeval
15. PromptLayer: DeepEval Review https://www.promptlayer.com/blog/deepeval-review-what-it-is-and-the-best-alternatives-in-2026/
16. DeepTeam GitHub https://github.com/confident-ai/deepteam
17. DeepTeam PyPI https://pypi.org/project/deepteam/
18. DeepTeam Red Teaming Introduction https://www.trydeepteam.com/docs/red-teaming-introduction
19. DeepTeam Adversarial Attacks https://www.trydeepteam.com/docs/red-teaming-adversarial-attacks
20. DeepTeam Risk Assessment https://www.trydeepteam.com/docs/red-teaming-risk-assessment
21. DeepTeam Code Scanning https://www.trydeepteam.com/docs/code-scanning-introduction
22. DeepTeam Guardrails https://www.trydeepteam.com/docs/guardrails-introduction
23. DeepTeam Vulnerabilities https://www.trydeepteam.com/docs/red-teaming-vulnerabilities
24. DeepTeam Frameworks https://www.trydeepteam.com/docs/frameworks-introduction
25. DeepTeam Data Privacy https://www.trydeepteam.com/docs/data-privacy
26. Ragas GitHub https://github.com/vibrantlabsai/ragas
27. promptfoo GitHub https://github.com/promptfoo/promptfoo
28. Inspect AI GitHub https://github.com/UKGovernmentBEIS/inspect_ai
29. LangSmith Pricing https://www.langchain.com/pricing
30. Langfuse Pricing https://langfuse.com/pricing
31. Arize Phoenix License https://arize.com/docs/phoenix/self-hosting/license
32. TruLens GitHub https://github.com/truera/trulens
33. MLflow GenAI Evaluation https://mlflow.org/docs/latest/genai/eval-monitor/
34. Opik GitHub https://github.com/comet-ml/opik
35. Braintrust Pricing https://www.braintrust.dev/pricing
36. Evidently OSS vs Cloud https://docs.evidentlyai.com/faq/oss_vs_cloud
37. Agent and model evaluations in Gemini Enterprise Agent Platform are now GA https://developers.googleblog.com/agent-and-model-evaluations-in-gemini-enterprise-agent-platform-are-now-ga/
38. AgentCore Evaluations GA https://aws.amazon.com/about-aws/whats-new/2026/03/agentcore-evaluations-generally-available
39. Microsoft Foundry Built-in Evaluators https://learn.microsoft.com/en-us/azure/foundry/concepts/built-in-evaluators
40. OpenAI API Deprecations https://developers.openai.com/api/docs/deprecations
41. inspect_evals GitHub https://github.com/UKGovernmentBEIS/inspect_evals
42. Langfuse Cloud Japan https://langfuse.com/blog/2026-04-27-langfuse-cloud-japan
43. MLflow Third-party Scorers https://mlflow.org/blog/third-party-scorers/
44. Comet Pricing https://www.comet.com/site/pricing/
45. Braintrust Run in CI https://braintrust.dev/docs/evaluate/run-in-ci
46. promptfoo Red Team Plugins https://www.promptfoo.dev/docs/red-team/plugins/
47. PyRIT GitHub https://github.com/microsoft/PyRIT
48. garak GitHub https://github.com/NVIDIA/garak
49. Giskard OSS GitHub https://github.com/Giskard-AI/giskard-oss
50. PurpleLlama GitHub https://github.com/meta-llama/PurpleLlama
51. promptfoo Pricing https://www.promptfoo.dev/pricing/
52. OpenAI to acquire Promptfoo https://openai.com/index/openai-to-acquire-promptfoo/
53. Moving from OpenAI Evals to promptfoo https://developers.openai.com/cookbook/examples/evaluation/moving-from-openai-evals-to-promptfoo
54. Langfuse joins ClickHouse https://langfuse.com/blog/announcing-acquisition
55. Dynatrace to acquire Arize https://www.dynatrace.com/news/press-release/dynatrace-to-acquire-arize/
56. Splunk / Galileo https://www.splunk.com/en_us/about-splunk/acquisitions/galileo.html
57. CoreWeave completes acquisition of Weights & Biases https://coreweave.com/blog/coreweave-completes-acquisition-of-weights-biases
58. Check Point acquires Lakera https://www.checkpoint.com/press-releases/check-point-acquires-lakera/
59. Palo Alto Networks completes acquisition of Protect AI https://www.paloaltonetworks.com/company/press/2025/palo-alto-networks-completes-acquisition-of-protect-ai
60. LLM Guard GitHub https://github.com/protectai/llm-guard
61. Guardrails AI joins Harvey https://www.harvey.ai/blog/guardrails-ai-joins-harvey
62. NeMo Guardrails GitHub https://github.com/NVIDIA-NeMo/Guardrails
63. Azure AI Red Teaming Agent https://learn.microsoft.com/azure/ai-foundry/concepts/ai-red-teaming-agent
64. Nejumi LLM Leaderboard https://nejumi.ai
65. Ragas Metrics Language Adaptation https://docs.ragas.io/en/stable/howtos/customizations/metrics/metrics_language_adaptation/
66. Microsoft Foundry Evaluation Regions and Limits https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-regions-limits-virtual-network
67. LangSmith Regions FAQ https://docs.langchain.com/langsmith/regions-faq