Prompt Regression Testing for AI Agents

It is the check that an improvement did not cost you something else. Agent behavior is coupled: a rule added to fix cancellations can change how refunds are handled, and a model upgrade can change everything at once. Regression testing protects the cases the agent already handles well while you change it.

Two things make it different from regression testing in ordinary software. Outputs are non-deterministic, so one bad run is not proof of a regression. And quality is judged, not asserted, so the scoring itself has to be stable. Both are why each scenario is run several times and compared against the current agent rather than against a fixed expected answer.

Before release, keep a small suite of protected scenarios built from production: conversations the agent handles well, plus the failures you already fixed. Run the candidate and the current agent on each scenario several times and compare scores in pairs. The guide to building a regression suite from production traces covers how to choose and de-identify the cases.

After release, count the failure patterns you care about on live traffic with the same detection rule you used before, and compare the overall failure rate as well as the one you targeted. The fix verification calculator tells you whether a change in those counts is bigger than chance.

Track a few behavioral measures per scenario before release and the matching pattern counts after it. Business metrics such as conversion are the outcome you care about, but they move slowly and for many reasons; behavioral measures tell you sooner which change caused the drop.

Make it automatic and keep it small. Run the suite on every candidate change instead of before big releases only, and keep it to the scenarios that matter, refreshed from production as the agent and its users change. A large stale suite is slower and catches less than a small current one.

Drop scenarios the current agent cannot pass reliably; an unstable baseline produces false alarms that teach people to ignore the suite. Gate automatic deployment on a passing suite, and let a person decide when a candidate trades a small regression for a larger gain.

The procedure is the same for both: run the current and the candidate configuration on the same protected scenarios, several times each, and compare in pairs. For a model change, keep the prompt fixed in the first comparison so you know which change caused a difference, then tune the prompt for the new model as a separate step.

The guide to why AI agents regress after prompt or model changes lists the regressions that model changes typically cause and a step-by-step investigation process.

Suite. Converra generates protected scenarios from your prompt and its analysis findings: behaviors the agent handles well, cases that trigger known weaknesses, and up to three scenarios reproduced from diagnosed production failures. A scenario is kept only if the current agent passes it stably, with a median score of at least 50 across five runs and little variation. Suites hold 3 to 15 scenarios by default. You can add your own scenarios; they skip the stability check and can be dropped later if they prove unstable.

Runs. Each scenario is a three-turn exchange. Converra runs five paired baseline and candidate runs per scenario, seven when the baseline is noisy, and another batch when results are mixed.

Gate. A scenario fails when the candidate's median drops by more than the larger of 7 points and the baseline's own variation, or by more than 3 points on a safety-critical scenario. The suite fails if any safety-critical scenario fails, if more than two scenarios fail, or if scores decline broadly across the suite. A candidate that fails the gate cannot be chosen as the winner, and automatic deployment, which is off by default, also requires a passing suite. Results list each scenario's baseline score, candidate score, and difference.

After release. Converra counts the targeted failure on live traffic and marks the fix verified, not fixed, or confounded; a verified fix whose failure returns is marked regressed. Automatic rollback on regression is off by default. When enabled, it triggers on a regressed verdict after at least 15 post-fix conversations: a direct deployment is restored, and a GitHub deployment gets a revert pull request for someone to merge.

This example is constructed to show the gate; it is not customer data. A candidate softens the agent's tone for frustrated users. It wins its head-to-head comparison on the target scenarios. In the regression suite, nine scenarios hold steady, but the cancellation scenario's median drops from 82 to 71: the new instruction to acknowledge frustration first pushes the cancellation steps out of the three-turn window.

An 11-point drop exceeds the 7-point allowance, so that scenario fails. One failed scenario is within the default limit of two and nothing is marked safety-critical, so the suite can still pass while showing the drop. A reviewer sees the trade-off before anything ships, and the cancellation case stays in the suite for the next attempt.