What Would Prove the Open Model Works on Real Production Traffic?

The first is the offline gate. A candidate that passed a frozen holdout earned the right to serve traffic. It did not demonstrate anything about the traffic, because the holdout was built from history and production is not obliged to repeat history. Request mix moves, users phrase things differently, tools return new shapes, and the rare cases that the gate could only sample a few of arrive in volume.

The second is the merged pull request. A PR proves that a change was proposed, reviewed, and accepted into a repository. It does not prove that the change deployed, that the runtime picked it up, or that the model string in production is the one in the diff. The third is the quiet dashboard. No alerts after a switch means no alert fired. It says nothing about whether the outcome you cared about held, because most dashboards were never built to measure that outcome in the first place.

Before the switch, the gate scored the candidate on a specific definition of success: the report names the right risk tier, the support reply cites a real order, the router hands off to the right specialist. Production proof has to score that same definition. If the gate measured task correctness and the production check measures customer satisfaction ratings, you have two studies of two different things, and neither confirms the other.

Write the outcome definition down once, version it, and reuse the same evaluator for both the gate and the production check. When the evaluator is a code check, that is easy. When it is a semantic grader, pin the grader version and its prompt, because a grader that changed between the gate and the production run can manufacture a regression or hide one.

Most production verification machinery is built around prompt changes, so it splits history by a prompt content hash. A model swap leaves the prompt hash unchanged. If your pre-and-post split uses that hash, every post-switch run lands in the pre-switch bucket and the comparison is same-versus-same. The result looks like perfect parity for the wrong reason.

The fix is to record the served model identifier on every run at the point of inference, from the actual response metadata rather than from configuration, and to split by that field. Configuration says what should have served the request. Response metadata says what did. When a fallback fired or a deploy lagged, the two disagree, and only the second one is evidence.

Before and after are only comparable if the population is the same. Compare the same request types, the same channels, the same customer segments, and where possible the same time-of-week pattern. A switch on Friday evening that is measured against Monday morning traffic compares two different workloads, and the model is only one of the differences.

The cleanest design is a concurrent split: route a fixed share of live traffic to the open model and the rest to the closed model at the same time, using a deterministic assignment so a given conversation stays in one arm. Concurrent arms share the traffic mix by construction. A sequential before-and-after design can still work, but it needs a longer window and an explicit check that the mix did not shift.

A model swap rarely happens alone. Someone edits the instruction the same week. A tool gets a new response format. A marketing campaign changes who is asking. Each of these can move the outcome by more than the model did. If the record does not show they happened, the analysis will credit or blame the model for them.

Keep a change log that is keyed to the same time window as the measurement: prompt versions, tool contract versions, routing rules, retrieval index refreshes, and known traffic events. When a material change lands inside the window, the honest verdict is confounded. That is not a failure of the experiment. It is the experiment reporting that attribution is not defensible, which is exactly what you need it to do.

Pre-register the decision: the outcome, the minimum number of scored runs per arm, the non-inferiority margin, and how missing or unscorable runs are handled. A rule written after the data is in will drift toward whatever the data supports. The margin should be the one the customer accepted when they approved the switch, expressed in the outcome's own units.

Missing data has to be handled symmetrically. If a run in the open-model arm could not be scored because the evaluator timed out, the same rule applies to the closed-model arm. If more than a small share of either arm is unscorable, the run is invalid and should be repeated after the fault is fixed, not reported as a pass with a caveat.

Suppose the agreed margin is five points on a hundred-point outcome score, and the paired per-run difference has a standard deviation of about twenty points on the development set. A one-sided non-inferiority test at the usual power needs on the order of a hundred paired observations to distinguish parity from a five-point loss. At three hundred requests a day, that is a day or two of traffic in a fifty-fifty split. At thirty requests a day, it is weeks, and the window is long enough that concurrent changes become likely.

Low volume does not make the question unanswerable; it makes the answer insufficient data until the runs accumulate. That is the correct verdict for a thin sample. Reporting parity from twelve runs because the numbers happened to line up is the same mistake as calling a coin fair after three flips.

The economic half of the claim is measured the same way as the quality half: per arm, from real usage. Token counts and wall-clock time come from the provider responses and the serving layer. A cost delta between arms is a measured number with a denominator, not a difference between two rate cards. Part 47 goes through why the rate-card version overstates savings.

Latency belongs in the verdict too, because a model that holds quality while doubling the p95 response time may fail the customer's requirement even when it passes the outcome rule. Report the distribution per arm, not the mean, and check it against whatever latency bound the customer set when they approved the test.

Parity verified means the pre-registered rule passed on comparable traffic with no material concurrent change in the window: the switch can be promoted to full traffic under the same approval. Regressed means the rule failed: the switch rolls back, and the evidence goes into the next dataset version as reviewed failures. Confounded means a concurrent change makes attribution indefensible: nothing is promoted, the window is rerun after the change settles. Insufficient data means the runs have not reached the minimum: the split continues.

None of the four is a judgment about the model in general. Each is a statement about one agent, one workload, one window. A parity verdict on a support agent's routing does not transfer to the same model on a different agent, and a regression on one workload does not mean the model is unusable elsewhere.

Converra's production A/B test splits live traffic deterministically between the incumbent and a challenger and rolls back when the challenger underperforms. For a model swap, its model production test keys each run on the model that served it, decides on non-inferiority plus strictly lower cost, and reports the verdict as parity verified, regressed, confounded, or insufficient data, with the measured cost delta per arm. The switch itself stays behind an explicit approval.

Two boundaries apply. No open-weight production verdict for a customer agent has been published yet; the machinery is available, and the result on any given agent's traffic is unobserved until it is measured there. And the customer owns the outcome definition and the margin, because a verdict on an outcome nobody agreed to is not proof of anything.