How to Build an AI Agent Regression Suite From Production Traces

Production traces contain realistic inputs, state, tool calls, latency, and outcomes. They can also contain personal data, ambiguous intent, transient dependency failures, incorrect user assumptions, and behavior that succeeded for the wrong reason. Copying the whole trace imports that ambiguity and risk into the suite.

The durable unit is the failure mechanism: the triggering condition, the rule or outcome that should have applied, the causal step, acceptable alternatives, and the evidence needed to decide pass, fail, or unobserved. LangSmith's current evaluation model makes the underlying distinction explicit: production runs generally lack reference outputs, while curated offline examples can include them. A trace therefore needs review before it becomes a regression contract.

Every proposed case should clear the same gates. A failure at rights, privacy, adjudication, or expected-behavior review pauses the case; it does not become a lower-confidence executable test.

Start with a confirmed harmful or costly failure, a repeat pattern, a production escape, a critical policy or tool boundary, or clean behavior adjacent to one of those failures. Frequency alone is not the rule: a rare authorization failure may deserve a permanent gate. Severity alone is not the rule either: an expired sandbox credential may belong in infrastructure testing rather than the agent behavior suite.

Record why the trace was selected. Keep frequency, severity, business impact, and confidence separate. If one was not measured, mark it UNOBSERVED instead of inventing a priority score. OpenAI's current evaluation guidance recommends combining production, historical, synthetic, and expert-curated data, covering typical, edge, and adversarial cases, and growing the set over time.

Production data should not move into an evaluation repository by default. Establish who can authorize test reuse, where the derived case may live, how long the source may be retained, and whether customer or vendor terms add limits. Publication and cross-customer reuse are separate decisions.

Keep only the turns and tool evidence required for the claim. Remove names, contact details, account numbers, secrets, and unnecessary free text. Transform contextual details that could reidentify a person or company. When exact wording is not causal, replace it with a behaviorally equivalent scenario and preserve only a restricted lineage pointer.

NIST's de-identification guidance warns that some de-identified data can still be reidentified. Removing direct identifiers reduces risk; it does not guarantee anonymity, establish consent, or create reuse rights.

Write a test card first. The failure claim should be bounded, the policy or desired outcome should be named, and the expected behavior should be observable without demanding one magic sentence. Exact matching is useful for structured tool arguments, required disclosures, and bounded classifications; it is usually brittle for open-ended conversation quality.

When reviewers disagree, preserve the disagreement and route it to the policy or outcome owner. Do not compress source evidence, reviewer opinion, evaluator output, and later production outcome into one truth label.

Preserve the user goal, relevant prior state, tool availability or error mode, agent or handoff boundary, required policy context, and the point where the outcome becomes observable. Keep conversation length or interruption patterns only when they are causal. Remove incidental wording, identities, dates, and unrelated turns.

Converra's current regression-scenario path instructs its drafting step to recreate the conditions of a production failure rather than copy exact words. It prioritizes production-grounded reproduction candidates alongside synthetic candidates and validates candidates against the baseline before inclusion. That is implementation evidence for this workflow, not proof that every trace label is correct or that every production environment is supported.

A failure case by itself can reward a degenerate fix. If the agent issued an unauthorized refund, a version that refuses every refund might pass the narrow failure test while making the product worse.

Add a nearby clean control: the same intent under an authorized state, the same tool with a successful response, the same wording with different account context, or the same handoff with the correct destination. Add a counterfactual when it tests the causal hypothesis by removing the suspected trigger while keeping the rest stable. If the failure persists, the proposed cause may be wrong.

Version the source type, transformation, failure label, expected behavior, evaluator model and rubric, threshold, tool fixtures, and the prompt, model, routing, and orchestration under test. Preserve creation, review, supersession, and retirement records. Never edit a failed case silently to rescue a candidate.

Separate development cases from protected acceptance cases. Repeatedly tuning against every visible case encourages overfitting. Use a reviewed holdout or another independent acceptance cohort whose labels do not change merely because a candidate missed them.

Run baseline and challenger against the same scenario contract, tool fixtures, and relevant seed or stochastic settings. Record both outcomes. Baseline-only and challenger-only runs do not belong in the primary paired comparison.

Ask two questions separately: did the challenger resolve the preserved failure, and did it preserve clean controls plus unrelated protected behavior? Aggregate scores help with triage, but a critical confirmed regression can remain a hard gate. The companion sample-size guide explains how pair count, variance, rare-case coverage, and stop rules fit together.

After an approved version reaches production, retain its version and deployment marker and monitor the same failure pattern on fresh traffic. A new escape can become a candidate case only after it clears rights, privacy, and adjudication again.

The loop is trace to reviewed claim to versioned case to matched replay to governed deploy to fresh production outcome. Each transition has its own evidence owner. A regression-suite pass selects a candidate; a pull request records governance; fresh production evidence determines whether the deployed behavior was verified, not fixed, or confounded.

Refresh or retire cases when traffic, policy, models, prompts, tools, routing, evaluators, or data rights change. Retire only with a reason such as removed workflow, superseded policy, invalid source label, duplicate mechanism, expired rights, or stronger replacement coverage. Keep result history when retention rules allow it.