When Should the Open Model Escalate a Request to the Closed Model?

A routing policy answers one question for every incoming request: will the open model handle this one as well as the closed model would? Answering yes sends the request down the cheap path. Answering no sends it to the frontier model. Every wrong yes is a quality failure the customer will see; every wrong no is money spent for nothing. That makes the escalation rule a binary classifier, and it deserves the same discipline as any other: a labeled development set, a held-out test, and measured precision and recall.

The labels come from the parity gate. Each evaluation case already has a paired result: the open model matched the incumbent, or it did not. Those pairs are the training and validation data for the escalation rule. A signal is useful exactly to the extent that it separates the matched cases from the unmatched ones on data the rule has not seen.

The best triggers fire before the open model is called, because they avoid the cost of a wasted inference and the latency of a second call. Evidence condition is the first: a request with no supporting evidence in the prompt, the retrieved context, or the tool results is one where smaller models most often fabricate, and a deterministic check can often detect the absence. Request class is the second: a lightweight classifier, or a rule on the request's source and fields, identifies the classes the candidate failed in the gate. Context size is the third: requests whose input approaches the open model's effective context limit degrade predictably, and the token count is known before the call.

Policy category is the fourth. Some request types are escalated not because the open model fails them, but because the customer wants the frontier model on them for the added margin: anything touching money, health, legal exposure, or a regulated disclosure. This trigger is a product decision expressed as a rule, and it should be listed separately in the escalation report so it is not mistaken for a quality finding.

Some failures are only visible after the open model has produced something. A tool call with malformed arguments, a required tool that was not called, an output that fails strict parsing, an explicit abstention, or an answer that a deterministic check can falsify against the evidence. These are post-inference triggers, and they cost one open-model call plus one closed-model call whenever they fire.

The hardest post-inference trigger is a confident wrong answer with valid structure. Nothing in the output flags it. The only defenses are a verifier, meaning a second check that compares the answer to the evidence, or a self-reported uncertainty signal from the model, which the next section addresses. A team that cannot build a verifier for a request class and cannot predict that class before inference should not be routing that class to the open model at all.

Asking the model how confident it is, or reading a token-probability score off its output, gives a number. That number is not a probability of being right until you have checked it against outcomes. On a development set, bucket the model's confidence scores and compute the actual match rate in each bucket. If the bucket labeled 90 percent matches the incumbent 60 percent of the time, the score is uncalibrated, and a threshold set on it will escalate the wrong requests.

Calibration is also model-specific and data-specific. A confidence threshold tuned for one fine-tune does not carry to its successor, and a threshold tuned on one customer's request mix does not carry to another's. Treat the threshold as a per-model, per-tenant configuration value with a recorded calibration date, and re-check it whenever the model, the prompt, or the traffic mix changes. A fine-tuned model that always reports high confidence, which is common, gives this trigger no discriminating power at all, and the router should not lean on it.

On the held-out gate results, count four outcomes. True escalations: requests sent to the closed model that the open model would have failed. False escalations: requests sent to the closed model that the open model would have handled. True keeps: requests kept on the open model that it handled. False keeps: requests kept on the open model that it failed. Recall is true escalations over all requests the open model fails; precision is true escalations over all escalations.

Then attach a cost to each error. A false escalation costs the closed-model price for that request. A false keep costs whatever a wrong answer costs on that request class, which is the number the customer has to supply, and it is usually much larger. The router's operating point should be chosen on those costs, not on a balanced accuracy score. For most agent workloads, that pushes toward high recall and accepted low precision: escalate generously on the classes where a miss is expensive, and accept that some of those escalations were unnecessary.

Suppose the gate ran 400 paired cases and the open model failed 60 of them. A router built on evidence condition and request class flags 90 cases for escalation. Of those 90, 51 are cases the open model failed, and 39 are cases it would have handled. Recall is 51 of 60, or 85 percent; precision is 51 of 90, or about 57 percent. Nine failing cases slip through as false keeps.

Now price it. At 10 cost units per closed-model call, the 39 false escalations cost 390 units across 400 requests, or about one unit per request. The nine false keeps are wrong answers the customer sees. If a wrong answer on this workload costs 200 units in rework and trust, the false keeps cost 1,800 units. Raising recall to 95 percent by escalating more generously might add 40 false escalations, another 400 units, while cutting false keeps to three, saving 1,200. On these numbers the more generous router is clearly cheaper. The arithmetic changes with the cost of a wrong answer, which is why the customer has to supply it.

In practice a router combines both kinds of trigger. Pre-inference rules handle the classes the gate flagged and the deterministic conditions: no evidence, oversized context, policy categories. Post-inference checks catch what the pre-inference rules missed: structure failures, tool failures, verifier disagreements. Every request carries a record of which path it took and which trigger fired, so an incident can be traced and the escalation rate can be broken down by cause.

Keeping the paths separate also keeps the measurement honest. Pre-inference escalations are a cost decision the router made on incomplete information; post-inference escalations are a quality decision made on the open model's actual output. A rising pre-inference rate means the traffic mix is shifting toward the classes the router routes away. A rising post-inference rate means the open model is failing more often on the requests it keeps, which is a drift signal that part 45 covers.

A router only helps with failures it can see coming or catch afterward. It does not help with the confident wrong answer in valid structure on a request class the gate never contained, because there is no signal to route on. It does not help when the escalation rate climbs past the point where the hybrid saves money, which part 22 works through. And it does not replace training data: a class the open model fails should be a candidate for the next dataset version, not a permanent escalation.

It also does not make the closed model a safe fallback by default. The closed model has its own failure modes on thin evidence, and an escalation policy that treats every frontier answer as correct will pass those through. The verifier that checks the open model's answers should check the escalated answers too.

Converra's benchmarks run candidate models against scenarios generated from the agent's own instruction, three runs per scenario with median scoring and measured cost and latency, which produces the per-scenario paired results an escalation rule is built from. The winning switch ships as a pull request with a comparison table. For a fine-tuned candidate, the workflow curates reviewed production runs into versioned datasets with protected splits, records the training job and its gate result, and holds the production switch behind an explicit approval.

Converra's step-level diagnosis locates the step and turn where a production conversation broke, which is how post-inference failures on the open path become labeled cases for the next dataset version rather than permanent escalations. The model production test decides on non-inferiority plus strictly lower cost, and the model-swap verdict on real traffic is parity verified, regressed, confounded, or insufficient data. Converra does not run training and does not use customer data to train any model without a written, tenant-exclusive election. No open-weight production verdict for a customer agent has been published yet.