How to Find the AI Agent Failures Your Evals Miss and Prove the Fix

Three gaps account for most of it. Coverage: the suite tests the requests you imagined, and production users phrase goals, combine tasks, and interrupt themselves in ways you did not. Drift: the mix of requests changes after launch, and a case that was rare when the suite was written becomes common. Turn depth: most suites score single responses, while production failures accumulate across turns, tool calls, and handoffs between agents.

The result is a suite that stays green while the failure rate in production moves. The guide to why passing LLM evals fails in production covers the mechanics; this one covers what to do about it.

Start with a sample of real conversations, including ones your monitoring marked as fine. For each, find the first turn where the agent's behavior diverged from what the user needed: it misread the goal, answered the wrong question for that point in the conversation, ignored context it already had, or called the wrong tool. Record the turn, the failure mode, and the root-cause class: prompt, configuration, context, code or routing, or model fit.

Step-level diagnosis is what makes the rest possible. A five-turn conversation that fails at turn two and one that fails at turn five need different fixes, and a conversation-level score cannot tell them apart. Converra scores every agent response on intent recognition, relevance, context use, and tool use, and traces across agent boundaries so a broken handoff names both agents; see step-level diagnosis.

Group the diagnosed steps by failure mode and root cause. The output is a short list of patterns, each with a count and a denominator. Two rules keep this honest. First, one definition of failure everywhere: if the pattern view, the failure rate, and the fleet summary use different predicates, they will disagree and one of them is wrong. Second, exclude what you could not diagnose from the denominator instead of counting it as a pass; a thin sample should read as insufficient data, not as a low rate.

Sanity-check the distribution before trusting it. If nearly every conversation lands in the same bucket, suspect the detector or the threshold before suspecting the agent; a metric pinned near its floor or ceiling has no headroom to show a change.

Take the pattern with the highest count times severity and write it as a scenario: the user's goal, the context the agent had, the turn where it broke, and what a correct response would have done. Add it to the regression suite before you change anything. That is the piece your evals were missing, and it should never leave the suite once it is in.

Build the scenario from the production conversation, not from memory of it. The guide to building a regression suite from production traces covers redaction, generalizing a single conversation into a scenario family, and keeping the suite from going stale.

Fix the layer the diagnosis named. A prompt-layer failure gets a prompt change that is as small as will remove the behavior; a configuration failure gets a threshold or tool-setting change; a code or routing failure gets a described change with the evidence attached for the engineer who owns that code. Resist the rewrite. A fix that touches ten things cannot be verified for any one of them.

Converra generates prompt and configuration fixes automatically from the diagnosis and provides a PR-ready description with evidence for code and routing causes; it does not merge code on its own.

Run the current agent and the candidate on the same simulated conversations, including the new scenario and the existing suite, and score both with the same rules. Compare them per persona, not by averaging two unrelated runs. A candidate is worth approving only if it beats the baseline on the conversations both ran and does not regress the rest of the suite.

Converra does this as a paired comparison on shared personas, with a regression gate before any deployment. The number of simulated conversations needed depends on the effect size you are trying to detect; see how many simulated conversations agent testing needs.

Shipping the fix does not end the loop. Count the targeted pattern in production conversations after the release with the same detector you used before, with enough conversations on each side, and check that nothing else changed the agent in the window. The table summarizes the outcomes Converra reports, as of October 2026.

The numbers are illustrative, not customer data. A support agent's eval suite of 140 cases has been green for six weeks. Step-level diagnosis of 500 production conversations finds a pattern the suite never covered: when a user asks about a refund and an order lookup returns nothing, the agent promises a refund anyway. It occurs at turn three or four in 31 of the 500 conversations (6.2%), always classified as a prompt-layer failure.

The team writes the scenario, adds it to the suite, and generates a prompt change that requires a found order before any refund commitment. In paired simulation the candidate removes the behavior on the new scenario family and matches the baseline on the other 140 cases. It ships under approval. In the seven days after release the detector finds the pattern in 9 of 580 conversations (1.6%). Under the rules above that reads as improving, not verified, because 9 recurrences remain, so the next step is to diagnose those 9. One turns out to be a different failure: the order lookup tool timed out, a code-layer cause the prompt fix could not touch.

The one published Converra example is the Salespeak case study: unsupported pricing, VAT, and infrastructure claims did not recur on live traffic after a deployed fix, and routing failures fell 74% after a second deploy. Limits: one customer, one agent, and percentages rather than counts.

Run as a one-off, this method finds one missed failure. Run continuously, it changes what the eval suite is: a record of every production failure that has been diagnosed and fixed, with the proof attached. New failures still appear, because users keep finding new paths, but each one enters the suite once and is measured once. That is the difference between maintaining an agent by hand and having a loop that does it; the agent optimization hub describes the full loop.