# Your LLM Evals Passed. Why Did the Agent Still Fail in Production?

Source: https://converra.ai/blog/why-passing-llm-evals-fails-in-production
Published: 2026-09-02. Updated: 2026-09-02. Sources reviewed: 2026-09-02.
Publisher: Converra (https://converra.ai), the autonomous improvement loop for production AI agents.

Passing LLM evals does not prove that a production agent works. It proves that one identified version met defined thresholds on a specific dataset, under one evaluator and test environment. The agent can still fail when production traffic, tools, permissions, deployed artifacts, or business outcomes differ from that test contract.

Teams still need evals. But evals are pre-deployment evidence, not the finish line. The reliability loop begins where the green eval ends: confirm what shipped, observe the same failure on fresh traffic, and issue a bounded verified, not fixed, or confounded verdict.

> A green eval is a bounded test result, not a production verdict. Use evals to select a candidate; use comparable post-deployment evidence to determine whether the change actually worked.

## A green eval is a bounded result, not a production verdict

An eval turns an expectation into a testable contract: the scenario, the system version, the behavior that counts, the evaluator, the release threshold, and the evidence retained for review. That repeatability is valuable precisely because generative systems are variable.

But an eval suite is still a sample and a measurement system. A green result means the tested version passed declared checks on the tested cohort. It does not mean every production input is covered, every label is correct, every dependency is healthy, or the tested artifact is the one serving users.

## What a passing eval actually proves

Read every pass conditionally: version V met threshold T on dataset D, using evaluator E in environment H at time R. Remove one identifier and the result becomes harder to reproduce. Generalize beyond them and the claim becomes weaker.

*The boundary around a green LLM eval*

| Contract element | What it controls | What a green result does not establish |
| --- | --- | --- |
| Version | Prompt, model, tools, retrieval, routing, code | That the tested version reached production |
| Dataset | Inputs, scenarios, slices, reference labels | Coverage of unseen production traffic |
| Evaluator | Rules, human labels, model judge, threshold | Ground truth or unbiased measurement |
| Environment | Tool state, permissions, latency, dependencies | Equivalent production conditions |
| Decision rule | Pass threshold, pairwise lift, safety gates | Business impact after deployment |
| Review date | Freshness of the entire contract | Future validity after traffic or model drift |

## Six reasons green evals still fail in production

1. The suite measured the wrong distribution. Curated cases can overrepresent clean, common, English-language, single-turn, or easily labeled interactions. Production introduces longer histories, incomplete context, unusual goals, tool errors, and slices the suite omitted.

2. The evaluator measured a proxy. A format checker can confirm valid JSON while the agent chooses the wrong action. A model judge can reward a polished answer that failed the user's task. The test passes because the evaluator and the real outcome are not identical.

3. The offline environment omitted real dependencies. Production tools time out, permissions differ, retrieval indexes change, and orchestration paths carry state that happy-path fixtures often replace.

4. The team optimized to the visible suite. Repeated prompt tuning against the same small set can improve the reported score without improving general behavior. Tuning evidence and acceptance evidence need separate cases.

5. The deploy broke the evidence chain. The tested prompt may not be active, a model alias can move, or a tool schema can change between test and deploy. A correct pull request is still only a proposed change until the runtime identity is known.

6. The production mix changed. New customer segments, campaigns, languages, model releases, policies, and seasonal demand can create failure modes that were absent from the test window.

## So, do you still need LLM evals?

Yes—when variable behavior can create a costly failure and repeatable testing is cheaper than discovering that failure through users. You may not need a large evaluation platform. You do need a testable release contract for decisions that are too expensive to make from a demo, a few handpicked prompts, or a reviewer saying the output looks fine.

A small team can start with one costly failure mode, reviewed positive and clean-control cases, a deterministic check where possible, and a comparison against the current version. Add semantic grading only for behavior code cannot evaluate directly.

*When an LLM workflow needs repeatable evals*

| Question | If no | If yes |
| --- | --- | --- |
| Can the workflow produce variable behavior? | Ordinary deterministic tests may be sufficient. | Assess the consequence of wrong behavior. |
| Could a failure harm a user, violate a rule, spend money, block revenue, or create review work? | Keep examples and spot-check; revisit as volume grows. | Define the expected outcome precisely enough to review. |
| Can reviewers state what acceptable behavior means? | Define the policy first; an evaluator cannot rescue an undefined requirement. | Build a small, versioned regression set. |
| Does the workflow change frequently? | A small regression set plus release review may be enough. | Run scoped offline evals on material changes and feed reviewed production failures back into the suite. |

## Offline evals, online evals, and production evidence do different jobs

Offline evidence is cheaper and safer for candidate selection. Online evaluation reveals traffic the suite missed. Production outcome measurement closes one specific change. Combining them creates a loop: observe a failure, adjudicate it, add a regression, test a fix, deploy under policy, and measure the same outcome.

*The evidence ladder for an AI agent change*

| Layer | Primary job | Typical action |
| --- | --- | --- |
| Deterministic tests | Enforce schemas, permissions, tool arguments, and invariants | Block a broken build |
| Offline behavioral evals | Compare versions before user exposure | Select or reject a candidate |
| Regression suite | Protect behavior the current version already handles | Block or flag a tradeoff |
| Online evals | Detect quality patterns on live traffic | Alert, sample, or create a review queue |
| Production outcome measurement | Determine whether the deployed change worked | Issue a bounded verdict |

## Human review should calibrate the system, not inspect everything forever

Nothing replaces judgment when the requirement itself is ambiguous. Domain owners define policy and acceptable tradeoffs. Reviewers label representative positives, clean controls, and ambiguous cases. Automated evaluators triage repeatable behavior. Humans then concentrate on consequential disagreements, new patterns, and sampled ordinary cases.

Manual review becomes a calibration and escalation system instead of a ritual pass through every output. Repeated reviewer overrides are evidence that either the judge contract or the agent's behavior needs to change.

## The reliability loop starts where the green eval ends

A candidate that wins offline has earned consideration for deployment, not a claim that the problem is solved. Preserve the tested version identity, deploy under an explicit policy, and measure the same failure definition against fresh post-change traffic with a comparable baseline.

The result can be verified, not fixed, or confounded. That last state matters: if traffic mix, another release, or missing coverage prevents attribution, the honest answer is not success or failure. It is that the evidence cannot isolate the change yet.

Start with one sentence: if the agent does X in situation Y, the consequence is Z. Turn it into reviewed cases, compare the current and candidate versions, preserve what shipped, and keep measuring after deployment.

## Sources

- Evaluation best practices (OpenAI): https://developers.openai.com/api/docs/guides/evaluation-best-practices
- LangSmith evaluation concepts (LangChain): https://docs.langchain.com/langsmith/evaluation-concepts
- LangSmith evaluation workflow (LangChain): https://docs.langchain.com/langsmith/evaluation
- Challenges to the monitoring of deployed AI systems (NIST): https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-4.pdf

## Frequently asked questions

### Why do LLM evals pass when production fails?

LLM evals can pass because the dataset missed a production slice, the evaluator measured a proxy, the test environment omitted dependencies, the team tuned to the visible suite, the deployed version differed, or traffic changed.

### Do small teams need LLM evals?

Small teams need LLM evals when variable behavior can create a meaningful cost and repeatable testing is cheaper than discovering failures through users. Start with one costly failure and a small versioned case set.

### Are golden sets enough for LLM evals?

Golden sets protect known expectations but are not sufficient by themselves. Labels can be wrong, coverage becomes stale, and production reveals new behavior, so keep clean controls, protected holdouts, and a path for reviewed failures to enter later suite versions.

### Do I need online LLM evals?

Use online evals when you need to detect quality patterns, edge cases, or drift in live interactions. Online scores still need calibration and do not automatically establish that a deployed change caused an outcome.

### What replaces manual QA for LLM applications?

Layered evaluation replaces repetitive checking, not human judgment. Deterministic tests handle rules, automated evaluators triage repeatable semantic checks, and humans define policy, calibrate labels, and review consequential disagreements.

## Related reading

- [Your LLM judge is not ground truth](https://converra.ai/blog/llm-as-a-judge-pitfalls): Measure evaluator precision, recall, bias, and drift instead of trusting one score.
- [Where improvement loops stop](https://converra.ai/blog/why-ai-agent-improvement-loops-stop-before-production-verification): Why a tested change still needs a fix-linked production verdict.
- [AI agent testing](https://converra.ai/ai-agent-testing): How simulation, regression protection, and production verification fit together.

Markdown edition of https://converra.ai/blog/why-passing-llm-evals-fails-in-production. The HTML page is canonical; this file is for agents and answer engines.