What Percentage of Calls Can Safely Move to the Open Model?

Teams often begin with a number: move 70 percent of calls to the open model and keep the rest on the frontier. The number comes from a cost model, and the cost model needs it to hit a savings figure. But the open model's ability to handle a request has nothing to do with what fraction of traffic a spreadsheet needs it to handle. A target chosen this way forces the router to send requests the open model fails, or it silently misses the target when the router does its job.

The right order is the reverse. First find out which requests the open model handles at parity. Then find out what share of real traffic those requests are. That share is the answer, and if it is below what the cost model needed, the conclusion is that the candidate is not ready for that savings figure, not that the router should be loosened.

The parity gate should already report results per request class, not just in aggregate, because a candidate at parity on average can be far below it on one class. Define the classes by something the router can see before inference: the request's source, its intent as labeled by a lightweight classifier, its evidence condition, its input size. Each class gets its own paired comparison against the incumbent, its own non-inferiority check, and its own pass or fail.

Classes that pass are routable to the open model. Classes that fail stay on the closed model. Classes with too few cases to decide are neither; treat them as failing until the evaluation set has enough of them, because moving a class on insufficient evidence is exactly the kind of projection this article is trying to prevent. Part 17 covers how many cases per class a gate needs before a pass means anything.

Pull the class distribution from a recent window of production traffic, using the same classifier the router will use. The window should be long enough to include weekly patterns and any seasonal cycle the workload has: a month is a reasonable floor, and the choice should be recorded. Each class's share of that window, multiplied by whether it passed, gives its contribution to the movable share.

The result is a percentage of calls, and also a percentage of tokens, and the two differ. A class of short classification requests may be 40 percent of calls and 10 percent of tokens; a class of long document summaries may be the reverse. Report both, because cost follows tokens and quality risk follows calls. A transfer that moves 70 percent of calls but 30 percent of tokens saves less than the call share suggests.

Suppose an agent's traffic falls into five classes. Status lookups are 35 percent of calls and the candidate passed at parity. Order changes are 25 percent and passed. Policy questions with retrieved context are 20 percent and passed. Policy questions without retrievable context are 12 percent and failed on abstention. Multi-account reconciliations are 8 percent and had only nine gate cases, too few to decide.

The movable share is 35 plus 25 plus 20, or 80 percent of calls. The 12 percent that failed stays on the closed model. The 8 percent that could not be decided also stays, pending more evaluation cases. If the customer has additionally decided that anything touching a refund goes to the frontier model regardless of gate results, and refunds are a third of order changes, another 8 percent comes off. The safe share is then 72 percent of calls. By tokens, if reconciliations and unsupported policy questions are the long ones, it might be closer to 55 percent, and the savings projection should use that.

Two things reduce the share below what the gate supports. The first is the customer's own decision to keep certain categories on the frontier model regardless of results, which the example handled by removing refunds. The second is the router's accuracy. The gate results say what the open model does on each class; the router decides which class a live request belongs to, and it is wrong some of the time. A router with 90 percent precision on class membership will send some failing-class requests to the open model, and the effective quality on the moved share is lower than the gate said.

Fold router error into the estimate. If 12 percent of traffic is a failing class and the router misses 10 percent of it, about 1.2 percent of all calls go to the open model when they should not. Whether that is acceptable depends on the cost of a wrong answer on that class, which the customer supplies. If it is not acceptable, the fix is a better signal for that class or a more conservative router, not a lower target percentage.

The traffic mix in the derivation window is a snapshot. Product launches, seasonal demand, a new integration, and a change in who uses the agent all shift the class distribution. If the unsupported policy questions grow from 12 to 25 percent of calls because a new product line has thin documentation, the safe share falls, and the hybrid's escalation rate rises with it. The cost projection built on the old mix is now wrong in the customer's disfavor.

Re-derive the share on a schedule and whenever the class distribution moves beyond a recorded threshold. The router's escalation rate per cause is the early warning: a pre-inference escalation rate that climbs means the mix is shifting toward routed-away classes; a post-inference rate that climbs means the open model is failing more on the classes it keeps, which is quality drift rather than mix drift, and part 45 covers detecting it.

A derived share is still a projection until live traffic confirms it. Ramp the open model in stages: a small percentage of the movable classes first, then more, with each stage held long enough to accumulate enough scored conversations to compare against the incumbent's on the same classes. The comparison at each stage uses the same dimensions the gate used: correct answers, next actions, safe behavior on thin evidence, format, latency, and cost.

Write the rollback rule before the ramp starts: which measure, what threshold, over how many conversations, and who is notified. A rollback rule that is decided during an incident is decided under pressure and usually too late. A production A/B split that sends a fixed fraction of live traffic to the challenger and rolls back automatically when it underperforms is the cleanest mechanism, because it makes the comparison concurrent rather than before-and-after and removes the temptation to explain away a regression as a busy week.

The number that goes to the customer should read like a measurement: 72 percent of calls and 55 percent of tokens moved to the open model as of a stated date, derived from gate results on a stated dataset version and a stated traffic window, at a stated router precision, with a stated set of policy-routed categories. Each of those inputs can change, and the report should say which ones changed when the number is updated.

A share reported without those inputs is a marketing number. It cannot be checked, it cannot be reproduced, and when it turns out to be wrong, no one can say why. Part 47 extends the same discipline to the savings figure itself, which is the share times the cost difference, and is wrong for the same reasons whenever the share is.

Converra benchmarks candidate models on scenarios generated from the agent's own instruction, three runs per scenario with median scoring and measured cost and latency, and opens a pull request with the winning switch and a comparison table. For a fine-tuned candidate, the workflow curates reviewed production runs into versioned datasets with protected splits, records the training job and its gate result against a dataset version, and holds the production switch behind an explicit approval. Training itself runs on the customer's provider or infrastructure.

The ramp described here is what Converra's production A/B test does for a shipped change: it splits live traffic between the incumbent and the challenger and rolls back when the challenger underperforms. For a model swap, the model production test decides on non-inferiority plus strictly lower cost, and the verdict is parity verified, regressed, confounded, or insufficient data, with the cost delta per arm. Converra does not run training and does not use customer data to train any model without a written, tenant-exclusive election. No open-weight production verdict for a customer agent has been published yet; the share that can move on a given agent is unobserved until measured there.