How Does Converra Know Whether an AI Agent Is Doing a Good Job?
Converra knows whether an AI agent is doing a good job by testing it against that agent's defined outcome and rules, not a universal score. It combines deterministic and semantic evaluation, paired baseline-versus-challenger simulation, regression protection, governed deployment, and evidence from new production conversations.
Each layer answers a different question. Evaluation identifies behavior worth changing. Simulation selects a candidate. Regression checks protect behavior that already works. Human governance decides what ships. A production verdict determines whether the deployed fix actually worked.
The short version
“Good” is an evidence contract for one agent and one job. Converra keeps outcome definition, candidate selection, deployment approval, and production proof separate so no single score can silently stand in for all four.
First define the job, the failure, and the decision
A support agent, sales agent, research agent, and routing orchestrator should not share one universal definition of quality. A support agent may need grounded answers and correct escalation. A router may need to send each request to the right specialist. A research agent may need complete source coverage and faithful citations.
Converra starts with the agent's objective, customer-defined metrics or rules, and observed production behavior. The useful unit is a named failure pattern: what happened, where in the trajectory it happened, why it matters, and what evidence would show that it stopped happening.
This prevents Goodhart's-law behavior in which the system improves a convenient score while missing the actual job. An aggregate can summarize evidence, but it cannot create the outcome definition.
| Layer | Question it answers | Evidence | What it cannot prove |
|---|---|---|---|
| 1. Outcome contract | What does good mean for this agent? | Customer objective, rules, metrics, and failure cost | Whether current behavior meets the contract |
| 2. Conversation evaluation | What happened and where did it fail? | Trace, deterministic checks, semantic evaluation, and diagnosis | That a proposed fix will work |
| 3. Paired simulation | Is a challenger better than the current version on matched cases? | Baseline and challenger scored on the same personas | That production traffic will behave the same way |
| 4. Regression gate | Did the challenger damage behavior the baseline handled? | Protected cases and material regression rules | That every unknown edge case is safe |
| 5. Deployment governance | Is this change authorized to go live? | Review-first PR or another explicitly configured policy | That the deployed change improved outcomes |
| 6. Production verdict | Did this shipped fix work on new traffic? | Fix-linked post-apply conversations and confounder handling | A universal future guarantee |
Why Converra compares the same personas head-to-head
Absolute averages can compare different traffic. A baseline might receive harder personas than a challenger and appear worse even when the prompt change did nothing. Converra's winner-selection contract uses persona-level head-to-head pairs: baseline and challenger must both have a scored conversation for the same persona before that evidence enters the comparison.
Unpaired conversations do not select the winner. Raw conversation count does not substitute for matched evidence. A challenger must show strictly positive paired lift to beat the baseline; if every challenger is flat or worse, the baseline remains the winner.
This makes a simulated win narrower and more honest. It means the challenger beat the current version on the matched test design. It does not mean the change is production-verified.
Regression tests protect what already works
Prompt fixes can be locally correct and globally harmful. A stronger escalation rule might reduce hallucinations while sending routine questions to a human. A concise answer style might improve one support flow while omitting required detail elsewhere.
Converra carries forward protected scenarios and checks the challenger against behavior the baseline already handled. A material regression can disqualify a candidate even when its target metric improves. This is why finding the highest score is not the same as selecting a safe change.
The judge is evidence, not an oracle
Some agent qualities can be checked deterministically: a tool was called, required JSON parsed, a cited source exists, or a forbidden field appeared. Semantic qualities such as whether the response addressed the user's intent may require an LLM judge or a human reviewer.
Converra does not answer judge-reliability questions with an unsupported universal precision number. The stronger evidence chain is visible: defined metrics, matched comparisons, regression cases, transcripts, and real production results after deployment. Customers can bring their own metrics, ground truth, and evaluation rules when those better express the job.
Simulation selects the candidate; production decides the result
A simulation winner has earned the right to be considered for deployment. The engineer or configured deployment policy decides whether it goes live. Converra then evaluates new production conversations tied to that change and returns one of three outcomes.
| Verdict | Meaning | Next action |
|---|---|---|
| Verified | The production evidence supports that the targeted fix worked without a disqualifying regression. | Preserve the win and its regression coverage. |
| Not fixed | The targeted failure persisted or worsened under the measured change. | Reopen diagnosis with the failed attempt attached. |
| Confounded | The observed result cannot be attributed cleanly because other changes or evidence gaps interfere. | Create a cleaner comparison or gather sufficient evidence. |
A bounded production example: Salespeak
Salespeak's production orchestrator routes conversations to specialist agents. In that one documented deployment, approved fixes reduced routing failures by 74 percent and eliminated the observed hallucinated pricing, VAT, and infrastructure claims. Converra generated and tested the changes; Salespeak's CTO reviewed and applied them; production traffic supplied the outcome evidence.
That case establishes that the loop worked for one agent and defined failure set. It is not an industry benchmark, an average expected lift, or a promise that every candidate wins. One attempted fix was also caught as not fixed, which is exactly the kind of negative outcome a trustworthy loop must preserve.
What the customer still owns
The customer owns the outcome definition, data boundary, deployment policy, and final accountability. Converra automates repetitive diagnosis, candidate generation, simulation, regression, and follow-up measurement. It does not replace the engineer who integrates the system, reviews the evidence, and decides what the agent is allowed to do.
The clean starting scope is one production agent, one high-value failure pattern, one agreed metric, and one accountable owner. That is enough to learn whether the evidence chain produces a defensible result without pretending one dashboard score describes the whole system.
Frequently asked questions
Does Converra use one AI agent reliability score?
Converra does not use one universal reliability score as the final definition of agent quality. It combines the agent's outcome contract, component evaluation, paired comparisons, regression gates, and production evidence.
Can customers bring their own AI agent metrics or ground truth?
Customers can bring their own metrics, evaluation rules, and ground-truth examples when those better represent the agent's job. Converra uses that evidence inside the same comparison, regression, governance, and verification workflow.
How reliable is Converra's LLM judge?
Converra does not claim a universal judge-reliability number because performance depends on the rubric, failure mode, traffic, and reference labels. Judge output is one evidence layer alongside matched simulation, regression cases, visible transcripts, and production outcomes.
What happens if a Converra simulation winner fails in production?
If a simulation winner fails to improve the targeted production behavior, Converra marks the deployed fix not fixed and returns it to diagnosis with the failed attempt attached. A simulation win never guarantees a production win.
What does a confounded Converra production verdict mean?
A confounded verdict means the available production evidence cannot isolate the effect of the fix. Simultaneous changes, traffic differences, instrumentation gaps, or insufficient evidence can all prevent clean attribution.
Can engineers inspect the evidence behind Converra's decision?
Engineers can review the proposed change, simulation evidence, regression result, and the conversations and measurements supporting the production outcome. Human review remains a deployment policy, while the production verdict remains a separate evidence decision.
Stop reading dashboards. Ship the fix.
Converra diagnoses the failure, tests the fix in simulation, and verifies it worked on your real traffic. Connect your production data and see it on your own agent.