What Should an Evaluation Set Contain Before You Claim Closed-Model Parity?

Before choosing cases, write down the request types the agent handles in production and roughly how often each appears. Include the ones that are rare and costly. A parity claim that silently excludes the hardest tenth of traffic is a claim about the easy nine tenths, and that is not what a customer hears when they are told the open model matches the closed one.

Then decide which of those types the transfer is meant to cover. If the plan is routing rather than full replacement, the evaluation set can be scoped to the routed share, and the claim scopes with it. Either way, the set's composition should be a written decision with a rationale, not whatever cases happened to be lying around from an earlier prompt experiment.

Coverage is not one dimension. The table lists the components a parity set needs, why each is there, and what its absence hides. Most failed transfers that looked fine in evaluation were missing two or three of these.

The three evidence conditions have three different correct behaviors: answer, qualify, abstain. Blending them into one accuracy number lets a model that is excellent on supported cases and dangerous on no-evidence cases post a good average. Score each condition as its own row per model, and set the pass rule so a candidate has to hold parity on each row, not on the blend. Part 18 works through the mechanics.

Seeded adverse cases get an even stricter rule. They are the findings whose absence causes real harm: a sanctions hit, a policy violation, a safety instruction. Require every one to appear in the candidate's output, with zero allowed misses. With 25 seeded cases and zero observed misses, the rule of three puts the 95 percent upper bound on the true miss rate near 11 to 12 percent; state that bound rather than claiming perfect recall, and add cases if a tighter bound is needed.

Parity with a model that is failing the task is meaningless. Before comparing any candidate, score the closed incumbent on the frozen set and require it to clear a floor: for a classification-shaped task, something like ninety percent on the primary decision and full recall on the seeded adverse cases. If the incumbent does not clear the floor, the gate is invalid, not passed or failed, and the right next step is fixing the task or the answer keys, not proceeding to the candidate.

This check also protects the answer keys. An incumbent that scores poorly on cases a reviewer believes are easy is often pointing at wrong keys or ambiguous cases. Resolve those before the freeze, on a development set, so the protected holdout is not opened for repair.

The comparison is paired: each case is scored for the incumbent and the candidate, and the quantity of interest is the per-case difference. Non-inferiority asks whether the mean difference is no worse than a margin you choose in advance, the same logic the FDA's guidance on non-inferiority trials describes for clinical designs. The margin is a business decision: how many points of quality the customer will trade for the cost reduction.

The sample size then follows from the margin and the spread of per-case differences. For a one-sided test at five percent significance with eighty percent power and an assumed true difference of zero, the required number of pairs is roughly (1.645 plus 0.842) squared, times the variance of the paired differences, divided by the margin squared. Suppose the margin is five points and the standard deviation of paired differences is estimated at 15 on a development set; that gives about 56 pairs. If the standard deviation is 20, it is about 99. The number grows with the square of the spread, which is why a development estimate of that spread is worth getting before freezing.

The variance of paired differences is unknown before you run anything, and the honest place to estimate it is a development set disjoint from the protected holdout, using a replay of the incumbent against a cheaper catalog model. That replay is a proxy for candidate-versus-incumbent variance and should be labeled as one. Document the estimate, its source, and how sensitive the sample size is to it.

Then size the protected set to at least the calculated number of pairs, with a floor that reflects the adverse-case and template requirements, and freeze it. If the calculation says more families are needed than exist, generate or collect more before the freeze. Never lower the margin or the sample size after opening candidate outputs; a holdout that was used to retune the gate is no longer a holdout.

The set is only comparable if every model sees the same inputs, the same pre-fetched tool results, the same instruction, and the same output schema, and is scored by the same frozen scorer. Serve tool results from fixtures rather than live tools, so a flaky lookup does not become a quality difference between arms. Run each case several times per arm and take the median, so one bad sample does not decide a case.

Handle missing data symmetrically. A model-output failure, meaning empty, schema-invalid, refused, or truncated, scores zero for that case in whichever arm produced it, including the incumbent. An infrastructure failure, meaning a timeout, a rate limit, or a connection error, is retried under a pre-registered rule and then excluded from the pair and counted per arm. If more than a small share of any arm's runs are excluded, the run is invalid and repeated after the fault is fixed.

The cases are one input to the score. The scorer's code, its weights, the answer-key table, the adverse-case matcher, the report schema version, the incumbent's exact model id, the fixture tool results, and the candidate's serving receipt all move the result. Record a content hash of each in a preregistration file before the first candidate inference. At evaluation time, recompute every hash and refuse to run on a mismatch.

Persist the verdict as one immutable record: the dataset version, the preregistration hash, the sample-size calculation and its inputs, the per-case scores per arm, the exclusions per arm, the result of each rule, and the overall pass, fail, or invalid. That record is what a customer can audit, and it is what makes a later production result comparable to the gate that preceded it.

A frozen evaluation set proves parity on the cases it contains, scored the way it scores them. It does not prove parity on the traffic that arrives next month, on request types that were rare enough to be excluded, or on outcomes the scorer cannot see, such as a customer's downstream satisfaction. It also cannot see cost and latency at production concurrency, which are measured, not gated, at this stage.

Passing the gate earns a production test, and the production test issues the verdict. Part 32 describes what it takes to show the open model works on real traffic, and part 21 sets out the definition of parity the gate and the production test should share.

Converra benchmarks candidate models and the incumbent on the same scenarios generated from the agent's instruction, three runs per scenario with median scoring, with real cost and latency captured from the calls, and ships a winning switch as a reviewed pull request. For a fine-tuned candidate, its workflow curates reviewed production runs into versioned datasets with protected splits, records the training job and gate result against that dataset version, and holds the production switch behind an explicit approval. The model production test then decides on non-inferiority plus strictly lower cost on live traffic.

Converra does not run training and does not use customer data to train any model without a written, tenant-exclusive election. No open-weight production verdict for a customer agent has been published yet. The evaluation set described here is what makes a gate result meaningful; the outcome on a given agent's traffic stays unobserved until the verdict is measured there.