Your LLM Evals Passed. Why Did the Agent Still Fail in Production?
Passing LLM evals does not prove that a production agent works. It proves that one identified version met defined thresholds on a specific dataset, under one evaluator and test environment. The agent can still fail when production traffic, tools, permissions, deployed artifacts, or business outcomes differ from that test contract.
Teams still need evals. But evals are pre-deployment evidence, not the finish line. The reliability loop begins where the green eval ends: confirm what shipped, observe the same failure on fresh traffic, and issue a bounded verified, not fixed, or confounded verdict.
The short version
A green eval is a bounded test result, not a production verdict. Use evals to select a candidate; use comparable post-deployment evidence to determine whether the change actually worked.
A green eval is a bounded result, not a production verdict
An eval turns an expectation into a testable contract: the scenario, the system version, the behavior that counts, the evaluator, the release threshold, and the evidence retained for review. That repeatability is valuable precisely because generative systems are variable.
But an eval suite is still a sample and a measurement system. A green result means the tested version passed declared checks on the tested cohort. It does not mean every production input is covered, every label is correct, every dependency is healthy, or the tested artifact is the one serving users.
What a passing eval actually proves
Read every pass conditionally: version V met threshold T on dataset D, using evaluator E in environment H at time R. Remove one identifier and the result becomes harder to reproduce. Generalize beyond them and the claim becomes weaker.
- Contract element
- Version
- What it controls
- Prompt, model, tools, retrieval, routing, code
- What a green result does not establish
- That the tested version reached production
- Contract element
- Dataset
- What it controls
- Inputs, scenarios, slices, reference labels
- What a green result does not establish
- Coverage of unseen production traffic
- Contract element
- Evaluator
- What it controls
- Rules, human labels, model judge, threshold
- What a green result does not establish
- Ground truth or unbiased measurement
- Contract element
- Environment
- What it controls
- Tool state, permissions, latency, dependencies
- What a green result does not establish
- Equivalent production conditions
- Contract element
- Decision rule
- What it controls
- Pass threshold, pairwise lift, safety gates
- What a green result does not establish
- Business impact after deployment
- Contract element
- Review date
- What it controls
- Freshness of the entire contract
- What a green result does not establish
- Future validity after traffic or model drift
Six reasons green evals still fail in production
1. The suite measured the wrong distribution. Curated cases can overrepresent clean, common, English-language, single-turn, or easily labeled interactions. Production introduces longer histories, incomplete context, unusual goals, tool errors, and slices the suite omitted.
2. The evaluator measured a proxy. A format checker can confirm valid JSON while the agent chooses the wrong action. A model judge can reward a polished answer that failed the user's task. The test passes because the evaluator and the real outcome are not identical.
3. The offline environment omitted real dependencies. Production tools time out, permissions differ, retrieval indexes change, and orchestration paths carry state that happy-path fixtures often replace.
4. The team optimized to the visible suite. Repeated prompt tuning against the same small set can improve the reported score without improving general behavior. Tuning evidence and acceptance evidence need separate cases.
5. The deploy broke the evidence chain. The tested prompt may not be active, a model alias can move, or a tool schema can change between test and deploy. A correct pull request is still only a proposed change until the runtime identity is known.
6. The production mix changed. New customer segments, campaigns, languages, model releases, policies, and seasonal demand can create failure modes that were absent from the test window.
So, do you still need LLM evals?
Yes—when variable behavior can create a costly failure and repeatable testing is cheaper than discovering that failure through users. You may not need a large evaluation platform. You do need a testable release contract for decisions that are too expensive to make from a demo, a few handpicked prompts, or a reviewer saying the output looks fine.
A small team can start with one costly failure mode, reviewed positive and clean-control cases, a deterministic check where possible, and a comparison against the current version. Add semantic grading only for behavior code cannot evaluate directly.
- Question
- Can the workflow produce variable behavior?
- If no
- Ordinary deterministic tests may be sufficient.
- If yes
- Assess the consequence of wrong behavior.
- Question
- Could a failure harm a user, violate a rule, spend money, block revenue, or create review work?
- If no
- Keep examples and spot-check; revisit as volume grows.
- If yes
- Define the expected outcome precisely enough to review.
- Question
- Can reviewers state what acceptable behavior means?
- If no
- Define the policy first; an evaluator cannot rescue an undefined requirement.
- If yes
- Build a small, versioned regression set.
- Question
- Does the workflow change frequently?
- If no
- A small regression set plus release review may be enough.
- If yes
- Run scoped offline evals on material changes and feed reviewed production failures back into the suite.
Offline evals, online evals, and production evidence do different jobs
Offline evidence is cheaper and safer for candidate selection. Online evaluation reveals traffic the suite missed. Production outcome measurement closes one specific change. Combining them creates a loop: observe a failure, adjudicate it, add a regression, test a fix, deploy under policy, and measure the same outcome.
- Layer
- Deterministic tests
- Primary job
- Enforce schemas, permissions, tool arguments, and invariants
- Typical action
- Block a broken build
- Layer
- Offline behavioral evals
- Primary job
- Compare versions before user exposure
- Typical action
- Select or reject a candidate
- Layer
- Regression suite
- Primary job
- Protect behavior the current version already handles
- Typical action
- Block or flag a tradeoff
- Layer
- Online evals
- Primary job
- Detect quality patterns on live traffic
- Typical action
- Alert, sample, or create a review queue
- Layer
- Production outcome measurement
- Primary job
- Determine whether the deployed change worked
- Typical action
- Issue a bounded verdict
Human review should calibrate the system, not inspect everything forever
Nothing replaces judgment when the requirement itself is ambiguous. Domain owners define policy and acceptable tradeoffs. Reviewers label representative positives, clean controls, and ambiguous cases. Automated evaluators triage repeatable behavior. Humans then concentrate on consequential disagreements, new patterns, and sampled ordinary cases.
Manual review becomes a calibration and escalation system instead of a ritual pass through every output. Repeated reviewer overrides are evidence that either the judge contract or the agent's behavior needs to change.
The reliability loop starts where the green eval ends
A candidate that wins offline has earned consideration for deployment, not a claim that the problem is solved. Preserve the tested version identity, deploy under an explicit policy, and measure the same failure definition against fresh post-change traffic with a comparable baseline.
The result can be verified, not fixed, or confounded. That last state matters: if traffic mix, another release, or missing coverage prevents attribution, the honest answer is not success or failure. It is that the evidence cannot isolate the change yet.
Start with one sentence: if the agent does X in situation Y, the consequence is Z. Turn it into reviewed cases, compare the current and candidate versions, preserve what shipped, and keep measuring after deployment.
Frequently asked questions
Why do LLM evals pass when production fails?
LLM evals can pass because the dataset missed a production slice, the evaluator measured a proxy, the test environment omitted dependencies, the team tuned to the visible suite, the deployed version differed, or traffic changed.
Do small teams need LLM evals?
Small teams need LLM evals when variable behavior can create a meaningful cost and repeatable testing is cheaper than discovering failures through users. Start with one costly failure and a small versioned case set.
Are golden sets enough for LLM evals?
Golden sets protect known expectations but are not sufficient by themselves. Labels can be wrong, coverage becomes stale, and production reveals new behavior, so keep clean controls, protected holdouts, and a path for reviewed failures to enter later suite versions.
Do I need online LLM evals?
Use online evals when you need to detect quality patterns, edge cases, or drift in live interactions. Online scores still need calibration and do not automatically establish that a deployed change caused an outcome.
What replaces manual QA for LLM applications?
Layered evaluation replaces repetitive checking, not human judgment. Deterministic tests handle rules, automated evaluators triage repeatable semantic checks, and humans define policy, calibrate labels, and review consequential disagreements.
Related reading
Your LLM judge is not ground truth
Measure evaluator precision, recall, bias, and drift instead of trusting one score.
Where improvement loops stop
Why a tested change still needs a fix-linked production verdict.
AI agent testing
How simulation, regression protection, and production verification fit together.
Stop reading dashboards. Ship the fix.
Converra diagnoses the failure, tests the fix in simulation, and verifies it worked on your real traffic. Connect your production data and see it on your own agent.