AI Agent Observability vs Optimization: From Trace to Fix

The two disciplines answer different questions and produce different artifacts. Most confusion comes from the overlap in the middle: modern observability platforms also run evaluations and experiments, and some now propose fixes.

A trace is the execution record of one request. OpenTelemetry's generative AI conventions, still in development status, define spans such as invoke_agent for an agent run and execute_tool for each tool call, with attributes for the model, the provider, token counts, duration, and errors. Platforms such as LangSmith and Langfuse capture the same lifecycle with inputs and outputs attached.

That record answers questions you cannot answer any other way: which tool the agent called and with what arguments, what the tool returned, which specialist a router handed off to, how long each step took, and where an exception was thrown.

A trace records what happened, not whether it was right. A tool call that returned successfully can still have been the wrong tool. A fluent answer can contradict the policy in the system prompt. Judging correctness needs a rubric, a reference, or a person, which is why observability platforms add evaluators.

A trace also does not say why the agent chose what it chose, which instruction to change, or whether a change would help. And a trace from last week cannot tell you whether this week's fix worked; only new traffic, compared with a baseline, can.

The boundary has moved. LangSmith runs online evaluations and automation rules on traces, and its Insights feature clusters traces into usage patterns and failure modes. LangSmith Engine goes further: it groups failures into issues, diagnoses root causes against your code, proposes a fix as a pull request, and reopens an issue if it resurfaces; its fix validation is in private beta. Langfuse pairs tracing with LLM-as-a-judge evaluation, datasets, and experiments.

So the useful question is no longer whether a tool only observes. It is which steps it owns between a failure and a verified outcome: counting the failure, generating a bounded change, testing it head-to-head against the current agent before release, gating the release on approval, and returning a verdict from live traffic.

Each step hands the next one a specific artifact. If a step has no artifact, that is where the loop breaks.

Converra connects to traces from LangSmith, Langfuse, or its SDK and API, and runs these steps as one loop. The hub guide to AI agent optimization covers each step, and production verification explains the verdict rules.

This example is constructed to show the hand-offs; it is not customer data. A support agent's trace shows a lookup_order tool call that returned no order, followed by a reply telling the customer their refund was on its way. Observability made the failure visible: the tool span, its empty result, and the final message are all in the trace.

Optimization starts by counting. Diagnosis finds the same pattern, a confident claim after an empty tool result, in a share of refund conversations last week. That count is the baseline. The fix is bounded: one instruction telling the agent what to say when a lookup returns nothing. Simulation runs refund conversations with missing orders against both versions, plus regression cases where the order exists. A reviewer approves the pull request. Over the following conversations, the pattern either stops recurring (verified), recurs at least as often (not fixed), or someone edited the same prompt in the meantime (confounded).

In the Salespeak case study, an orchestrator agent routed users to specialist agents. The routing decision was recorded in each conversation, but the record alone did not fix anything. Converra diagnosed the wrong-specialist pattern from production conversations, generated fixes aimed at routing behavior, tested them in multi-turn simulation, and Salespeak's CTO reviewed and applied the winning change. Routing failures fell 74% on live traffic after the April 25 deploy.

Limits: this is one customer and one agent. The published case study reports the percentage, not the underlying conversation counts. Under Converra's current verdict rules, a reduction that does not reach zero recurrences is reported as improving rather than verified.

Can you count how often a specific failure pattern occurred last week, not just find an example of it? Who decides what to change, and how is the change scoped to that failure? Is every candidate tested against the current agent on the same conversations before release? Is there a record of who approved the release and when it shipped? After release, does anyone measure the same failure pattern again, and would you notice if another change shipped in the same window?

Any question you answer with no is a gap between observability and a verified fix. If the gaps are in the middle, between a visible failure and a tested change, that is the work regression testing and simulation testing cover.