AI Agent Drift Detection Is Not Drift Correction

A useful drift signal has an eligible denominator, a failure rule, and a time window. For example: among refund requests where the order record is available, the agent gives a refund policy that conflicts with the current policy source. Keep requests without an order record, unknown policy versions, and unscorable conversations visible. Hiding them as passes makes an apparent rate too clean to act on.

Segment the signal by the conditions that can change its meaning: model and prompt version, tool or retrieval version, route, user request type, locale, and time. A total failure rate can move because traffic changed, not because the agent did. The point is not to create a dashboard for every field. It is to retain enough evidence to ask whether the same behavior moved in a comparable population.

When a signal moves, record the first affected time window and list every relevant boundary: prompt edits, model or model-version changes, tool schemas, retrieval sources, routing rules, safety policy, and upstream services. A model provider can change behavior, while a new knowledge-base article can alter answers without any model change. The calendar narrows the search; it does not prove causation.

LangSmith distinguishes offline regression testing from online evaluations that monitor production traces. That is a useful division of labor: online evaluation can identify a changed pattern or an unusual run, and the affected examples can become a focused offline comparison. Do not promote the alert directly into a prompt change. First inspect the failing step and decide which layer is actually eligible to change.

A correction should name the layer and the expected behavior. If a refund agent skips a lookup after a model upgrade, a candidate might change only the lookup instruction and its placement. If the tool contract changed, repairing the tool description or schema is a different candidate. Rewriting the whole prompt, changing the model, and changing retrieval together may improve an average while leaving no answer about why.

Run the candidate and current agent on the same drift-family conversations plus the nearby behavior that was already working. Score the named failure and the regressions that matter for the route. LangSmith's regression-testing guidance likewise frames comparisons as a way to identify improvements and regressions across versions. A passing set supports a release decision; it does not establish what will happen on later customer traffic.

Before release, record the candidate identifier, the current baseline, the failure rule, eligible denominator, simulation result, approval, and release time. Also record known overlapping changes. Without that provenance, a post-release chart cannot distinguish a repaired behavior from a traffic shift, model migration, tool outage, or a second prompt edit.

After the approved change is live, apply the original failure definition to new eligible conversations. Compare like with like and keep insufficient or unscorable cases explicit. If the failure recurs at least as often, the candidate is not fixed. If another material change overlaps the result, it is confounded. If there is too little eligible evidence, the honest result is insufficient evidence, not a silent pass.

A coding agent can retrieve the saved failure definition, affected versions, representative examples, and missing evidence; it can also prepare a bounded candidate for review. It should not diagnose a cause from a rate alone or declare a correction verified because a test passed. The decision boundary stays with the evidence and the person approving the release.

Converra connects the diagnosed behavior, a bounded prompt or configuration candidate, a head-to-head simulation, approval, and post-release verification. Through Converra MCP, a coding agent can inspect authorized diagnosis, comparison, deployment-impact, and verification evidence. State-changing actions remain governed by the connected account's scopes and the user's authority.