When Is Full Replacement Better Than Routing Between Open and Closed Models?

Full replacement swaps the model in the agent's seat and changes nothing else. There is one inference path, one output contract, one failure mode to monitor, one bill. Routing puts a decision in front of the model call: this request goes to the open model, that one goes to the closed model, and sometimes the open model's answer goes to the closed model for a second opinion. There are two inference paths, a router with its own accuracy, two failure modes, and a blended bill that depends on how often the router escalates.

Teams often treat routing as the cautious default, since it keeps the closed model available. It is cautious about quality and expensive about everything else. The decision should be made from evidence about the workload, not from the feeling that keeping both models is safer.

The strongest signal for full replacement is a candidate that passes the parity gate separately on supported requests, partial-evidence requests, and no-evidence requests, with zero misses on the seeded adverse set. That result says the open model handles the whole distribution the evaluation set represents, including the thin-evidence cases where smaller models usually fail. If the evaluation set is representative, there is no slice left to route.

The second signal is a narrow workload. An agent that classifies support tickets into twelve categories, extracts fields from a fixed document type, or drafts one kind of report from structured inputs has a bounded space of requests. The open model either covers it or does not, and the gate can tell. An agent that fields open-ended questions across a product line has a long tail that no evaluation set fully covers, and that tail is where routing earns its keep.

Routing is the right answer when the gate results show a specific request class where the candidate falls below the margin, and something about the request lets a router recognize that class before the answer is produced. Suppose the candidate matches the incumbent on nine request classes and falls short on one: multi-document reconciliation, which needs longer reasoning chains than it learned. If a lightweight classifier can flag reconciliation requests with high recall, routing them to the closed model preserves quality on that slice and saves money on the other nine.

The condition that matters is identifiable in advance. A slice the candidate fails but nothing can predict is not routable; it is a reason the candidate is not ready. Routing on a signal with poor recall sends failing requests to the open model and produces the same bad outcomes at a lower price, which is not the trade anyone meant to make.

Some workloads carry a small number of requests whose failure is very expensive: a regulatory question in a support flow, a medication query in a wellness agent, a request that touches money. Even if the open model passes the gate on them, a customer may reasonably want the frontier model on that slice for the added margin. Routing on request category is how that preference is expressed without paying frontier prices for the routine 95 percent.

The other clear case is knowledge. Fine-tuning transfers behavior, not knowledge, so an agent whose value on some requests comes from the closed model knowing things outside the prompt and the tools cannot hand those requests to a smaller model. If retrieval can supply the knowledge, the slice becomes routable to the open model after a retrieval change. If it cannot, the slice stays with the closed model, and routing is the only design that keeps the agent whole.

Suppose the closed model costs 10 units per request on this workload and the open model costs 1 unit including hosting. Full replacement takes cost from 10 to 1 per request. Routing takes it to 1 plus 10 times the escalation rate, plus the router's own cost. At a 10 percent escalation rate that is about 2 per request, at 30 percent about 4, and at 50 percent about 6. Routing that escalates half the time has given up most of the savings while keeping all of the complexity.

The escalation rate is not a design choice; it is a measurement. It comes from the share of production traffic that falls into the classes the router sends to the closed model, plus the share the router sends there by mistake. A hybrid that looks attractive at a projected 10 percent and runs at 35 percent in production is a common outcome, because request mixes shift and routers are tuned on samples that flatter them. Part 24 shows how to derive the safe share from gate results and the real traffic mix.

Two models means two sets of prompts to keep in sync, because the closed model and the open model respond differently to the same instruction and teams end up with per-model variants. It means two output contracts to test, two latency profiles to meet, two sets of failure modes to monitor, and a router whose accuracy has to be measured and re-measured as the traffic shifts. Every evaluation runs three arms instead of two. Every incident asks which path the request took.

None of that is prohibitive, and for a workload with a clearly failing slice it is worth carrying. But it is real cost that a per-token comparison never shows. Full replacement's strongest argument is simplicity: one model, one contract, one thing to verify. A team should choose the hybrid because the evidence demands it, not because it feels like hedging.

There is a third design that is often confused with routing: the open model handles everything, and the closed model is called only when the open model fails outright, meaning a timeout, an invalid output, or an explicit abstention. That is a failure handler, not a router. It does not try to predict which requests the open model will get wrong; it catches the ones where the open model visibly did not produce an answer.

Fallback-only is closer to full replacement than to routing. The escalation rate is the open model's hard failure rate, usually a few percent, so the cost stays near the open model's. The quality risk is the requests where the open model produced a confident wrong answer, which fallback cannot catch. If the gate shows those are rare, fallback-only is a good design. If it shows they cluster in an identifiable class, that class needs a real router. Part 46 covers what the failure handler should do automatically.

Start with the gate results broken down by request class and evidence condition. If every class passes and the seeded adverse set is clean, and the workload is bounded, choose full replacement with a fallback-only handler for hard failures. If one or more classes fail and a signal available before inference identifies them with high recall, choose routing on that signal and measure the projected escalation rate against real traffic. If a class fails and nothing predicts it, the candidate is not ready; add training data for that class and re-gate.

Then check the arithmetic. If the projected escalation rate puts the hybrid's cost above roughly half the closed model's, the savings are thin and the complexity is not paying for itself; either improve the candidate or defer the transfer. Finally, ramp whichever design you chose gradually, with a rollback rule tied to the same dimensions the gate measured, and let the production verdict decide whether the choice was right.

Choosing between replacement and routing on gate results assumes the evaluation set represents production. It does not settle how the request mix will shift, how the router will behave on requests the development set never contained, or whether the customer's cost target survives hosting and idle capacity. Those are production measurements. A routing design chosen for one customer's traffic may also not transfer to another customer's, which part 25 examines.

It also does not settle the question permanently. A workload that needs routing today may be fully replaceable after two more rounds of training data on the failing slice. The decision should be revisited whenever the candidate is retrained or the traffic mix changes materially, and part 33 covers when retraining is warranted.

Converra benchmarks candidate models on scenarios generated from the agent's own instruction, runs every model on every scenario three times with median scoring, and measures real cost and latency from the calls, so the per-model, per-scenario picture that this decision depends on is produced before anything ships. The winning switch opens as a pull request with a comparison table. For a fine-tuned candidate, the workflow curates reviewed production runs into versioned datasets with protected splits, records the training job and gate result, and holds the switch behind an explicit approval.

Once a design is chosen, Converra's production A/B test splits live traffic between the incumbent and the challenger and rolls back if the challenger underperforms, and the model production test decides on non-inferiority plus strictly lower cost. The model-swap verdict on real traffic is parity verified, regressed, confounded, or insufficient data, with the cost delta per arm. Converra does not run training and does not use customer data to train any model without a written, tenant-exclusive election. No open-weight production verdict for a customer agent has been published; the result on your traffic is unobserved until measured.