What Counts as Parity: Correct Answers, Useful Follow-Ups, Safe Behavior, or All Three?

A single composite score invites a single question, which is whether the candidate's number is close enough to the incumbent's. That question has no defensible answer, because the composite blends behaviors that have different consequences. A candidate can be two points better at answering supported requests and ten points worse at declining unsupported ones, and the composite will call that a wash. The customer whose agent starts inventing answers will not.

The fix is to state parity as a set of separate claims, each with its own measurement and its own margin. The agent answers correctly when it has the evidence. The agent takes the right next action: asks the right follow-up, calls the right tool, escalates when it should. The agent behaves safely when evidence is partial or absent. And the agent does all of that in the required format, within latency, at the expected cost. Each claim is checked on its own, and parity holds only when all of them do.

Supported requests are the ones where the evidence in the prompt, the retrieved context, or the tool results is enough to answer. This is the dimension most benchmarks measure and the one fine-tuning improves most reliably. It is also the easiest to overfit, because supported requests are the bulk of any training set. Measure it with the strictest available check: exact match against an answer key where one exists, a deterministic field comparison for structured outputs, a calibrated grader only where meaning genuinely cannot be checked by code.

The margin here can be tight, because the incumbent's performance on supported requests is usually stable and the candidate's failures are visible. A candidate that falls short on this dimension is not close to ready, and the gate should say so without weighing it against anything else.

Agents do not only answer. They ask clarifying questions, choose tools, pass arguments, decide to escalate, and decide to stop. A model that answers well but asks the wrong follow-up, or calls the right tool with a malformed argument, or escalates a routine request, produces a worse agent even when every final answer is correct. These behaviors have to be scored as their own dimension, from the trajectory rather than from the final message.

Scoring next actions needs an expected-action label per case, which means the evaluation set has to carry more than an answer key. For a case where the right move is a clarifying question, the label is the question's intent, not its wording. For a tool call, it is the tool and the argument contract. For an escalation, it is the fact of escalation and the reason. Part 39 covers how to build these labels when only the model changes and the surrounding agent stays fixed.

This is where most open-model replacements fail and most evaluations fail to notice. When the evidence is incomplete, the right behavior is a bounded answer that says what is and is not known. When there is no evidence, the right behavior is to say so and route the request somewhere useful. A smaller model that has been trained mostly on supported requests learns that every request has an answer, and produces one.

Measure this dimension on cases deliberately constructed for it, with labels for the acceptable behaviors. A candidate can be at parity on correct answers and clearly below it here, and that combination is a regression, not a trade. Part 18 explains how to keep these three evidence conditions separate in the evaluation set so that strength on one cannot mask weakness on another.

The margin for each dimension answers the question: how much worse can the candidate be here before the customer would not accept the swap? That is a product decision, and it has to be made before anyone looks at the candidate's output. A margin chosen after seeing that the candidate is four points behind on abstention will be five points, and the gate will pass. A margin chosen before will be whatever the consequence of a bad abstention actually justifies, and the gate will decide honestly.

Write the margins, the dimension weights if there is a composite for reporting, the seeded critical cases, and the missing-data rule into one file, hash it, and record the hash before the candidate's first evaluation call. Regulated non-inferiority trials work the same way: the margin is specified in advance and justified from what the comparator has been shown to do, precisely because a margin chosen after the data would be meaningless.

Margins allow small losses. Some losses cannot be small. A supplier-risk agent that misses one sanctioned entity, a support agent that resets the wrong account, a billing agent that quotes an invented discount: each is a single case, and a per-dimension margin of five points would let it through. So the gate needs a second mechanism: a set of seeded adverse cases, each carrying a required finding, and a rule that the candidate must produce every one of them.

Zero allowed misses on a seeded set is a strong rule, and it should be stated with its honest limit. Suppose the seeded set has 25 cases and the candidate misses none. The rule of three says the 95 percent upper bound on the true miss rate is about 3 in 25, roughly 11 to 12 percent. The gate can say the candidate missed nothing on 25 seeded cases and that the true miss rate is probably under 12 percent. It cannot say the candidate never misses. Reporting the bound instead of the perfect score is what keeps the claim defensible.

Suppose a paired evaluation on 120 cases shows the candidate three points ahead of the incumbent on correct answers to supported requests. On the 30 partial-evidence cases, the incumbent hedged correctly on 26 and the candidate on 19. On the 20 no-evidence cases, the incumbent abstained on 18 and the candidate on 11, answering the other 9 with confident fabrications. On the 25 seeded adverse cases, the candidate missed one.

The composite might come out slightly in the candidate's favor, because supported requests dominate the count. The gate fails it three times over: below the margin on partial-evidence behavior, below the margin on abstention, and vetoed by the seeded miss. That is the correct outcome. The candidate is a better answerer and a worse agent, and the team now knows exactly which training data to add: bounded answers and abstentions, which part 9 covers.

Cost and latency are part of the parity bundle because a replacement that meets every quality margin at twice the latency is not a replacement the customer can deploy. They are measured per arm from the actual calls, reported as distributions rather than means, and gated only when the customer has stated a limit. A p95 latency ceiling on a user-facing agent is a gate. A vague preference for speed is not.

Cost has a different role. A model production test that decides on non-inferiority should also require strictly lower cost, because parity at equal cost is a change with no upside and real migration risk. Cost measured in the gate is an estimate from token counts and list prices; cost measured in production, after hosting and fallbacks, is the number that matters, and part 26 works through the difference.

A candidate that passes every dimension on the frozen set has earned a switch under approval, not a conclusion. The traffic mix in production differs from the evaluation set, the repair layer behaves differently, and the customer's users find requests the set never contained. Parity has to be checked again on real traffic after the switch, on the same dimensions, with the deployment boundary preserved so the comparison is between before and after rather than between two different weeks.

The production result belongs in one of four states: parity verified, regressed, confounded by a concurrent change, or insufficient data to decide. Any claim of parity that stops at the gate is a claim about the evaluation set. Part 32 covers what a production verdict has to contain.

Converra benchmarks candidate models against scenarios generated from the agent's own instruction, runs every model on every scenario three times with median scoring, measures real cost and latency from the calls, and ships the winning switch as a pull request with a comparison table. For a fine-tuned candidate, the fine-tuning workflow curates reviewed production runs into versioned datasets with protected splits, records the training job and its gate result against a dataset version, and holds the production switch behind an explicit approval. Training runs on the customer's provider or infrastructure; Converra records it rather than running it.

After the switch, Converra's model production test decides on non-inferiority plus strictly lower cost, and the model-swap verdict on real traffic is parity verified, regressed, confounded, or insufficient data, with the cost delta per arm. The customer owns the parity definition, the margins, and the deployment policy. Converra does not use customer data to train any model without a written, tenant-exclusive election, and no open-weight production verdict for a customer agent has been published yet; the outcome on a given agent's traffic is unobserved until measured.