What Should Customers See in a Closed-Versus-Open Model Comparison?

The most common presentation is a table with the closed model's average score in one column and the open model's in the other. It answers the wrong question. Two averages can be equal while the models disagree on a third of the cases, and the cases where the open model loses are exactly the ones that decide whether a customer's hardest requests still get handled.

A comparison worth showing is paired. Each case ran on both models under the same inputs, the same pre-fetched tool results, and the same scoring, and the customer can see the distribution of per-case differences: how many cases the open model matched, how many it lost, by how much, and which ones. The average is a summary of that distribution, not a substitute for it.

The table below lists what a customer should be able to find on the comparison without asking. Each row names the field, why it changes the decision, and the failure that its absence hides. A comparison missing several of these is a demo, not a decision document.

An agent's requests fall into conditions the evaluation should keep apart: requests the available evidence fully supports, requests it partially supports, and requests with no relevant evidence at all. The closed model and the open model often match on the first and diverge sharply on the third, where the right behavior is to say so rather than to answer.

A single pooled score averages across those conditions in whatever proportion the evaluation set happened to contain. Show the paired result within each condition, with its own case count. A customer whose traffic is heavy on partial-evidence requests needs that row, not the pooled one. Part 18 of this series describes how to construct the three conditions; the comparison should present them as they were constructed.

Every agent has a class of request where a miss is expensive: a compliance flag, a refund limit, a safety escalation, an adverse finding. Seed the evaluation set with a known number of those cases and report how many each model caught. The number should be shown as a count, not a percentage, because the customer needs to know both the recall and how thin it is.

Be honest about what zero misses means. If the open model caught all 25 seeded cases, the 95 percent upper bound on its true miss rate is still about 12 percent by the rule of three. State that bound on the comparison. A customer who sees 100 percent recall without the case count will assume a certainty the evidence does not support.

A comparison that lists the closed model's per-token price beside the open model's per-token price has not measured anything. Show cost per case for each arm computed from the tokens each model actually used on each case, because the open model may produce longer or shorter outputs than the closed one and the difference changes the bill. Show latency percentiles per arm from the same calls, at the concurrency the evaluation ran at, with that concurrency stated.

Label the serving configuration the numbers came from: hardware, quantization, engine, and whether it was a dedicated replica or a per-token endpoint. Part 28 covers why a number from one configuration does not transfer to another; the comparison should make the configuration visible so the customer can ask whether it matches what will be deployed.

A comparison that says an 8B open model matched the closed model has named a family, not an artifact. The customer needs the base model identifier and revision, the adapter identifier, the training job, the dataset version the adapter was trained on, and the manifest that assigned cases to training, development, and protected splits. Those identifiers are what let anyone check that the evaluation cases were never seen in training and that the artifact evaluated is the artifact proposed for deployment.

The same applies to the closed arm. Closed models change under their aliases. Record the exact model version the comparison ran against and the date, because a comparison against last quarter's version of the closed model is a comparison against a model the customer may no longer be running.

The comparison should show the rule that was fixed before any candidate output was scored: the non-inferiority margin, the recall requirement on seeded cases, the mode checks for collapse, how missing data was treated on each side, and how many cases the rule required. A reader should be able to see that the rule existed first and that the result was judged against it, not that a rule was chosen to describe the result.

Show the outcome against each element of the rule separately. A candidate can pass the margin and fail recall, or pass both and fail a mode check. A single pass or fail label hides which element was close, and the close elements are where the customer's follow-up questions belong.

Cases that timed out, produced invalid output, or hit an infrastructure error have to appear on the comparison per arm, with the rule that was applied to them. A comparison that dropped twelve failed cases from the open arm and none from the closed arm has changed its denominator in a direction that favors the candidate, and the customer cannot see that unless the exclusions are listed.

Finally, label every number with the state it came from. Development numbers come from cases the team iterated on and can be optimistic. Acceptance numbers come from the frozen protected set and are the ones a switch decision should rest on. Production numbers come from live traffic after a switch and are the only ones that describe the customer's actual outcome. A comparison that mixes the three without labels invites the reader to treat a development score as a production result.

Converra's model benchmarks run every candidate on every scenario three times with median scoring and report quality, cost, and latency per model from the real calls, with the comparison table attached to the pull request that proposes the switch. For a fine-tuned candidate, the workflow records the gate result against a specific dataset version with protected splits, so the artifact identity and the split manifest travel with the result, and the production switch waits for an explicit approval on that candidate.

After a switch, the model-swap verdict on real traffic reports parity verified, regressed, confounded, or insufficient data with the cost delta per arm, which is the production row of the comparison above. No open-weight production verdict for a customer agent has been published yet; the fields described here are what a customer should demand from any comparison, whichever tool produces it.