What Is AI Agent Optimization? From Failure to Verified Fix

Optimization is defined by its output: a shipped change, plus evidence about whether it fixed a specific failure. Several adjacent practices produce inputs to that output, but each stops short of it.

Observability records what the agent did, so you can see where a conversation broke. Evaluation scores outputs against chosen cases, so you can compare candidates. Prompt engineering edits instructions. Optimization connects them: it starts from production failures, produces the change, tests it, gets it approved, and measures the result on the same kind of traffic that exposed the failure. The guide to AI agent observability versus optimization covers that boundary in detail.

Start from conversations that went wrong, not from a list of best practices. Diagnosis answers three questions: which step failed (routing, retrieval, a tool call, a policy, or the final answer), what kind of failure it was, and how often the same pattern recurs.

Converra diagnoses at the step level. It records the failure types it finds in each conversation, so a recurring pattern becomes a counted target instead of an anecdote. That count is also the baseline the production verdict uses later. See step-level diagnosis and the taxonomy of agent failure modes.

Prioritize by how often a pattern occurs and what it costs when it does. A routing mistake that sends users to the wrong specialist on a common request usually matters more than a rare formatting slip.

A good fix changes one behavior for one diagnosed reason. Rewriting a whole system prompt to fix a routing error makes the result impossible to attribute and risks breaking behavior that already works.

Converra generates prompt or configuration variants aimed at the diagnosed root cause, as small incremental edits rather than rewrites. Each candidate records which failure it targets and what it changes, so the reviewer can judge it as a specific change, not a new prompt.

Before anything reaches users, test the candidate head-to-head against the current agent on the same simulated users and scenarios. Running both versions on identical conversations isolates the change. Comparing a new variant with last month's average does not.

Converra runs multi-turn conversations with synthetic personas derived from production patterns. Only paired comparisons count as evidence, and a variant can win only when its head-to-head lift over the current agent is positive. Regression scenarios, cases the agent already handles well, check that fixing one behavior did not break another.

Simulation chooses a candidate. It does not prove the fix works in production, because simulated users are not your users.

Someone accountable should decide what goes live. In Converra, deployment is review-first by default: the winning change ships as a GitHub pull request or as a direct update that a person applies. Teams can turn on automatic deployment. It stays off until they do, and when it is on, a change must pass its regression test before it ships.

Record exactly what shipped and when. That deployment marker is what lets the next step separate conversations before the change from conversations after it.

After deployment, measure the targeted failure in new production conversations, with the same detection criteria used for the baseline, and classify the result. The table shows how Converra decides for fixes it ships, as of October 2026.

Before there is enough evidence, Converra reports the fix as monitoring (fewer than 5 diagnosed conversations since the change), likely fixed (no recurrence yet, but fewer than 15 conversations), or improving (fewer failures than before, but not zero). A verified fix whose failure later returns is marked regressed. The guide to production verification explains the evidence behind each verdict.

Salespeak ran an orchestrator agent that decided which specialist agent should handle each user. Converra analyzed its production conversations and found two recurring problems: users routed to the wrong specialist, and the orchestrator stating pricing, VAT rules, and infrastructure details that were not true.

Converra generated fixes aimed at those two behaviors rather than a general rewrite, tested the variants in multi-turn simulation before any change touched production, and Salespeak's CTO reviewed and applied the winning changes. Results were measured on post-deployment live traffic for each target failure: routing failures fell 74% after the April 25 deploy, and no hallucinated pricing, VAT, or infrastructure claims recurred after the April 23 deploy. Read the Salespeak case study.

Limits of this evidence: it is one customer and one agent, so it is not a rate you should expect on yours. The case study, published in May 2026, reports both results as verified; under the current rules in the table above, a 74% reduction that does not reach zero recurrences would be reported as improving, and only the eliminated claims would be verified. The two changes shipped two days apart, which is why each is reported against the failure pattern it targeted rather than as one combined number. The published case study reports percentages, not the underlying conversation counts, and covers the period after deployment rather than a long-term follow-up.

The loop pays off when four things are true. Failures recur in patterns you can name and count; a one-off failure is better handled as an incident. The agent has enough traffic to judge a change, since a verdict needs diagnosed conversations after deployment, so low-volume agents wait longer. A named person owns approvals and has time each week to review changes. And the agent's prompt or configuration can be changed through a repository or a direct update.

If an agent has stopped working after a change, start with first checks for an agent that stopped working, then bring the recurring pattern into the loop.

AI agent reliability is not one uptime score. It is repeatable task success across changing inputs, stable tool and routing behavior, recovery from failures, regression protection, and evidence that deployed fixes worked.

The loop connects those checks: measure the failure, change the behavior, protect what already works, and verify the production outcome. The production agent reliability methodology describes how comparable reliability evidence is collected and corrected.