How Converra Evaluates AI Agent Performance

A support agent, sales agent, research agent, and routing orchestrator should not share one universal definition of quality. A support agent may need grounded answers and correct escalation. A router may need to send each request to the right specialist. A research agent may need complete source coverage and faithful citations.

Converra starts with the agent's objective, customer-defined metrics or rules, and observed production behavior. The useful unit is a named failure pattern: what happened, where in the trajectory it happened, why it matters, and what evidence would show that it stopped happening.

This prevents Goodhart's-law behavior in which the system improves a convenient score while missing the actual job. An aggregate can summarize evidence, but it cannot create the outcome definition.

Absolute averages can compare different traffic. A baseline might receive harder personas than a challenger and appear worse even when the prompt change did nothing. Converra's winner-selection contract uses persona-level head-to-head pairs: baseline and challenger must both have a scored conversation for the same persona before that evidence enters the comparison.

Unpaired conversations do not select the winner. Raw conversation count does not substitute for matched evidence. A challenger must show strictly positive paired lift to beat the baseline; if every challenger is flat or worse, the baseline remains the winner.

This makes a simulated win narrower and more honest. It means the challenger beat the current version on the matched test design. It does not mean the change is production-verified.

Prompt fixes can be locally correct and globally harmful. A stronger escalation rule might reduce hallucinations while sending routine questions to a human. A concise answer style might improve one support flow while omitting required detail elsewhere.

Converra carries forward protected scenarios and checks the challenger against behavior the baseline already handled. A material regression can disqualify a candidate even when its target metric improves. This is why finding the highest score is not the same as selecting a safe change.

Some agent qualities can be checked deterministically: a tool was called, required JSON parsed, a cited source exists, or a forbidden field appeared. Semantic qualities such as whether the response addressed the user's intent may require an LLM judge or a human reviewer.

Converra does not answer judge-reliability questions with an unsupported universal precision number. The stronger evidence chain is visible: defined metrics, matched comparisons, regression cases, transcripts, and real production results after deployment. Customers can bring their own metrics, ground truth, and evaluation rules when those better express the job.

A simulation winner has earned the right to be considered for deployment. The engineer or configured deployment policy decides whether it goes live. Converra then evaluates new production conversations tied to that change and returns one of three outcomes.

Salespeak's production orchestrator routes conversations to specialist agents. In that one documented deployment, approved fixes reduced routing failures by 74 percent and eliminated the observed hallucinated pricing, VAT, and infrastructure claims. Converra generated and tested the changes; Salespeak's CTO reviewed and applied them; production traffic supplied the outcome evidence.

That case establishes that the loop worked for one agent and defined failure set. It is not an industry benchmark, an average expected lift, or a promise that every candidate wins. One attempted fix was also caught as not fixed, which is exactly the kind of negative outcome a trustworthy loop must preserve.

The customer owns the outcome definition, data boundary, deployment policy, and final accountability. Converra automates repetitive diagnosis, candidate generation, simulation, regression, and follow-up measurement. It does not replace the engineer who integrates the system, reviews the evidence, and decides what the agent is allowed to do.

The clean starting scope is one production agent, one high-value failure pattern, one agreed metric, and one accountable owner. That is enough to learn whether the evidence chain produces a defensible result without pretending one dashboard score describes the whole system.