How Do You Compare Two Models Without Output Repair Hiding Their Failures?

Most production agents carry a layer of quiet fix-ups between the model and the caller. A JSON parser that tolerates trailing commas. A retry when the schema fails. A default value filled in when a field is absent. A truncation of an answer that ran past the limit. A regular expression that pulls the answer out of surrounding prose. Each one exists because some model, at some point, produced something the downstream code could not use.

The problem for a comparison is that these fix-ups were tuned for the incumbent. The closed model's typical mistakes are the ones the repair layer already knows how to absorb. The open candidate makes different mistakes, and the same repair layer either misses them or fixes them differently. Two models scored through the same repair code are not scored under the same conditions. The repair layer is a third participant in the comparison and it is on the incumbent's side.

Repair erases evidence in four ways. First, structural failures vanish: an output that did not parse becomes an output that did, and the metric never learns the model cannot hold a schema. Second, omissions become defaults: a missing risk tier becomes low, a missing citation becomes an empty list, and the model gets credit for a field it never produced. Third, truncation becomes brevity: a cut-off answer scores as a short answer rather than as a failure to finish. Fourth, retries become single attempts: the third try counts and the two failures do not, so a model that succeeds one time in three looks reliable.

Each of these matters more for a smaller model than for a frontier one, because smaller models fail structurally more often. A comparison that repairs before scoring will systematically understate the gap on exactly the dimension where the candidate is weakest, which is the dimension that will bite in production.

Not all repair is wrong. The question is which failures the model can reliably avoid and which it cannot. A model cannot reliably count characters, so an answer that overflows a 200-character field by twelve characters is a limitation to truncate and record, not a reason to discard the output. A model can reliably produce a required field, a valid enum, or a closing brace, so a missing one is a real failure and should fail the output rather than being patched.

Written down, the rule is: truncate and record what the model cannot reliably avoid, and hard-fail what it can. Apply it identically to both models. A comparison built on this rule keeps structural failures visible, where they belong, and stops a length overflow from being counted as a wrong answer, where it does not belong. The rule has to be written before the comparison runs, because deciding it afterward is deciding which model wins.

Run every output through two scoring passes. The first pass takes the model's raw text, parses it strictly, and scores whatever survives. An output that fails strict parsing scores zero on the structural dimension and is marked so. The second pass applies the repair layer exactly as production would, then scores the repaired result. Both scores are stored per case per model.

The raw score tells you what the model can do on its own. The repaired score tells you what the deployed system would do. The difference between them, per model, is the size of the repair layer's contribution, and that number is the one a comparison usually needs most. Suppose the incumbent gains two points from repair and the candidate gains eleven. The candidate's repaired score may look close to parity, but the gap in raw scores says the candidate is leaning on a crutch that was built for someone else.

A repair that is only mentioned in a log line or a rationale string is invisible to every aggregate. The metric reads fields, not sentences. So each repair must land as a structured field on the scored record: which repair fired, on which output, what it changed, and what the original value was. Drops need the same treatment: a case that was excluded from the comparison must carry a reason code, not a note.

This is what makes the comparison auditable. A reviewer can ask how many candidate outputs needed enum coercion, or how many incumbent outputs were truncated, and get a count instead of a search through free text. It also makes the repair layer itself measurable over time. If the count of a particular repair rises after a model swap, the model has changed its failure mode, and that is a signal the production monitor should carry.

Every comparison has missing data, and how it is handled decides more outcomes than most teams realize. The rule has two halves. Model-output failures, meaning empty responses, schema-invalid output after strict parsing, refusals where an answer was expected, and truncation past the point of usability, score zero for that case in whichever arm produced them. The incumbent included. A frontier model that refuses a legitimate request is failing that case, and the comparison should say so.

Infrastructure failures are different. Timeouts, server errors, rate limits, and connection resets say nothing about the model's quality. They are retried a fixed number of times under a rule written before the run, and if they still fail, that case is excluded from the pair on both sides and counted per arm. The exclusion count is part of the report. The two halves must be applied to both models with the same thresholds, or the rule is not symmetric and the result is not a comparison.

An arm that loses more than a few percent of its cases to infrastructure failures is not the arm you meant to test. Suppose the candidate's endpoint dropped eight percent of calls to timeouts during the run, and those cases were excluded. The remaining ninety-two percent skew toward the requests that produced short outputs quickly, which are the easy ones. The candidate's score on that subset is not comparable to the incumbent's score on the full set.

Pre-register the threshold, and make the consequence a third outcome. A run that exceeds it is invalid: not a pass, not a fail, but a result that has to be repeated after the fault is fixed. Treating an invalid run as a fail punishes the candidate for an operations problem. Treating it as a pass because the surviving cases looked fine is the mistake this whole article is about. The third outcome exists so that neither temptation is available.

With repair recorded and missing data handled, the comparison itself should be paired. Each case is scored for both models under identical inputs and identical pre-fetched tool results, and the difference per case is what gets analyzed. An average across all cases for one model, compared to an average for the other, hides the cases where they disagree, and those are the cases that carry the decision.

Pairing also exposes repair asymmetry directly. Sort the per-case differences and look at the tail where the candidate loses badly. If most of those losses are structural failures that the repaired pass would have absorbed, the candidate's weakness is format discipline, which more training data or a tighter output schema can address. If they are wrong answers in valid structure, the weakness is judgment, which is a different problem with a different fix.

A clean, repair-aware comparison on a frozen evaluation set tells you the candidate's raw and repaired quality relative to the incumbent on those cases. It does not tell you how the candidate will behave on the production traffic mix, under production latency limits, or after the repair layer is tuned for its failure modes rather than the incumbent's. Those are separate measurements, and the production verdict in part 32 is where they land.

It also does not tell you that the repair layer should stay as it is. Once the raw failures are visible, the right move is often to fix the output contract so fewer repairs are needed, then re-run the comparison. A repair layer that grows to accommodate a new model is a sign the model was not ready, not a sign the integration is done.

Converra benchmarks candidate models on scenarios generated from the agent's own instruction, running every model on every scenario three times with median scoring and measuring cost and latency from the actual calls. The winning switch ships as a reviewed pull request with a comparison table. For a fine-tuned candidate, the fine-tuning workflow records the training job and its gate result against a versioned dataset with protected splits, and holds the production switch behind an explicit approval.

The production side is where the repair question resolves. Converra's model production test decides on non-inferiority plus strictly lower cost, and the model-swap verdict on real traffic comes back as parity verified, regressed, confounded, or insufficient data, with the cost delta per arm. Converra does not run training and does not use customer data to train any model without a written, tenant-exclusive election. No open-weight production verdict for a customer agent has been published yet; what a given agent's traffic will show is unobserved until it is measured.