What an AI Agent Trace Must Contain for a Fix to Be Verifiable

Start with the customer request and the outcome that makes the interaction eligible for one failure pattern. A support agent that promised a refund without checking an order is not interchangeable with every refund conversation. Its denominator is refund requests where the tool path is available, and its numerator is requests where the promise happened before the lookup.

Store an outcome signal with the trace: a completed task, a handoff, a correction, a policy violation, or a reviewed label. Keep an explicit incomplete bucket when the relevant step or outcome is absent. Filling that bucket with presumed successes makes the measured rate look better than the evidence supports.

Keep the ordered agent, model, tool, handoff, retrieval, and guardrail steps that could have caused the behavior. For a tool call, retain the tool identity, arguments, result or error, and timestamps. For retrieval, retain the source identity and the material context used in the response. For a handoff, retain the sending and receiving agent plus the boundary between them.

A fluent final answer cannot reveal whether the model ignored a correct tool result, the wrong tool was called, retrieval was stale, or a route selected the wrong specialist. The OpenAI Agents SDK illustrates the shape: it records generations, tool calls, handoffs, and guardrails as spans under one trace. Your framework can use a different schema, but it needs the same causal path for the question you are asking.

Record the prompt or configuration version, model identifier, routing policy, tool definition version, and retrieval or knowledge-base revision when each can affect the target behavior. The version in a repository is not enough when a prompt is edited in an application setting or a tool schema changes outside the code deploy.

Tie the candidate to one named layer. If the observed failure is an incorrect tool argument, a revised tool description is a different hypothesis from a prompt rewrite or a service fix. A trace that cannot identify the layer can still be useful for triage, but it cannot attribute a later improvement to one bounded change.

Use stable internal IDs to join related turns, release markers, and outcome records. Redact or tokenize personal data before it leaves the system that needs it, and set a retention policy for the raw payload. Preserve the evidence required to understand the failure, not a permanent copy of every customer conversation.

Tracing settings can remove inputs and outputs while retaining span structure. That improves privacy, but it can make a content-dependent failure impossible to diagnose later. Decide deliberately which fields need protected access, which can be redacted, and which absence means the trace is incomplete for a particular investigation.

Once you name the failure, freeze the eligible pre-change examples, the denominator rule, and the known-good behaviors that must not regress. Run the candidate and the current agent on comparable scenarios. Preserve both outputs and the evaluator or reviewer judgment, rather than replacing the baseline record with the candidate result.

This is the boundary between diagnosis and proof. A trace can suggest a candidate, but a single replay does not establish that the candidate improves the target pattern or preserves adjacent behavior. Keep unresolved cases visible instead of using them to pad either side of the comparison.

Record the approved release time, deployed version, and every overlapping change that could affect the target behavior. Then apply the same eligibility rule and detector to new production conversations. Compare the targeted failure with a comparable pre-change window and inspect the regression checks chosen before release.

A production verdict answers a narrower question than an observability dashboard: did this specified change move this specified failure under the recorded conditions? If the failure recurs, call it not fixed. If another prompt, model, routing, or policy change overlaps the window, call the result confounded. If there are too few eligible records, retain insufficient evidence.

A coding agent can retrieve the saved run, show the step-level diagnosis, identify missing evidence, and prepare a bounded prompt or configuration candidate. It should preserve trace IDs, versions, the denominator, and unresolved records in its handoff. Do not ask it to infer a production result from a changed prompt or a successful simulation alone.

With Converra MCP, inspect the account and agent inventory, retrieve the failing run's diagnosis, and use the authorized optimization and simulation workflow. After an approved release, inspect deployment impact and verification evidence. The connected account's scopes and the responsible user's approval still govern state-changing actions.