Can a Routing Policy Generalize Beyond the Customer Used to Develop It?

Published work on learned routers has shown that a router trained to choose between a stronger and a weaker model on general preference data can transfer to different model pairs without retraining. That is a real and useful result: the notion of a hard request has some model-independent structure, and a router can learn it. It supports the idea that routing is not hopelessly specific to one model.

It does not say that a router built on one company's support traffic will work on another company's. The published routers are trained on broad, general-domain data and evaluated on public benchmarks. A customer-specific routing policy is trained on one tenant's request mix, vocabulary, tools, and failure pattern. The question of whether that policy transfers across tenants is a different question, and the answer has to be measured per policy.

The policy was built from a parity gate run on one customer's evaluation set, weighted by one customer's traffic mix, with a request classifier trained on one customer's phrasing. Every threshold in it was set where that customer's failing and passing cases separated. Move it to a second customer and three things shift at once: the classes the open model fails may be different classes, the phrasing that identifies each class is different, and the base rates that made the thresholds sensible are different.

The result is a policy that looks confident and routes wrong. A class that was rare and safe on customer A is common and failing on customer B, and the router has never seen enough of it to notice. A confidence threshold calibrated on A's outputs sits in the wrong place for B's model, especially if B's fine-tune is a different candidate. None of this shows up until B's production traffic exposes it, unless the policy was validated on B first.

Some escalation triggers describe the request itself and hold for any tenant. Whether the retrieved context or tool results contain anything relevant to the request is a property of that request. Whether a required tool was called and returned successfully is a property of the trajectory. Whether the input token count is near the model's effective limit is a property of the input. Whether the raw output parses is a property of the output. These signals were not tuned to a customer's vocabulary, and they carry over.

They carry over because they are deterministic checks on structure, not learned judgments about content. That also makes them the safest part of a policy to ship as product code: they can be tested with fixtures, they behave the same on every tenant, and when they misfire the reason is visible. Part 23 lists these triggers and when each is available.

The request taxonomy is the first customer-specific component. Customer A's classes might be status, change, policy, and reconciliation. Customer B's might be triage, scheduling, eligibility, and appeal. Even where the names overlap, the phrasing that identifies each class does not, and a classifier trained on A's phrasing will misassign B's requests. The per-class gate results are the second: they describe what A's candidate did on A's evaluation set, and they say nothing about B's candidate on B's cases.

Calibrated thresholds are the third and the most easily forgotten. A confidence cutoff, a context-size limit tied to a particular model's degradation point, a minimum evidence score: each was set where one tenant's data separated. Treat all of these as configuration values with a tenant identifier, a model identifier, and a date attached. A policy that stores them as constants has quietly decided that every customer is customer A.

Take the policy as configured for customer A and apply only its transferable components to customer B. Rebuild B's taxonomy from B's traffic, run B's parity gate, and set B's thresholds from B's development data. Then measure the router on B's held-out gate results: precision and recall of escalation, and the resulting quality on the kept share, exactly as part 23 describes. If the product-code components hold and only the configuration had to change, the split is right.

If a second customer is not available, a held-out time window on the same customer is the fallback, and it is weaker. Take the policy tuned on months one through three and measure it on month four without retuning. It catches drift in the request mix and phrasing over time, which is a partial test of generalization. It does not catch the shift to a different taxonomy or a different model, so a policy validated only this way should be described as validated on one tenant over time, not as a product.

Suppose the policy for customer A routed on evidence condition, three request classes, and a confidence cutoff, with 85 percent escalation recall on A's held-out set. On customer B, the evidence-condition check performs the same, because it reads the context rather than the customer. The class classifier, retrained on B's phrasing, reaches similar accuracy after B's taxonomy is rebuilt. The confidence cutoff, carried over unchanged, escalates almost nothing, because B's fine-tune reports uniformly high confidence and the cutoff sits above all of it.

That is the typical shape: the structural signals transfer, the taxonomy transfers as a method but not as data, and the calibrated threshold fails silently. The fix is not a better threshold; it is a policy that refuses to use a confidence signal until it has been calibrated on the current tenant and model, and records that it did so. Part 34 covers the parallel problem when the closed teacher model changes and every calibration built against it goes stale.

The practical consequence is an architecture. The product ships the deterministic request-level checks, the mechanism for defining a taxonomy and training a class classifier from a tenant's traffic, the mechanism for reading per-class gate results into routing rules, and the mechanism for calibrating a confidence threshold with a recorded date. The tenant configuration holds the taxonomy, the classifier weights, the per-class rules with the gate version they came from, the thresholds, and the policy-routed categories the customer chose.

Every routing decision then carries the configuration version it was made under. When the escalation rate changes, the first question is whether the configuration changed; when a customer disputes a routing decision, the record shows which rule fired and which gate result justified it. A policy built this way generalizes because the parts that could not generalize were never pretended to.

A routing policy is a product when the same product code, with only configuration changed, has produced acceptable escalation precision and recall on at least two tenants with different taxonomies, and the configuration for a new tenant can be produced by a repeatable procedure from that tenant's gate results and traffic. Until then it is a project delivered to one customer, and describing it as a product overstates what has been shown.

That standard is deliberately modest. Two tenants do not prove universality, and a third with a very different workload may still break an assumption. The point is to have crossed from one to more than one with the product-versus-configuration split intact, which is the evidence that the split was drawn in the right place. Part 36 applies the same test to the whole transfer workflow.

Converra benchmarks candidate models on scenarios generated from each agent's own instruction, three runs per scenario with median scoring and measured cost and latency, and opens a pull request with the winning switch and a comparison table. Because scenarios are generated per agent, the per-class evidence a routing policy needs is produced per tenant rather than copied between them. For a fine-tuned candidate, the workflow curates reviewed production runs into versioned datasets with protected splits, records the training job and its gate result against a dataset version, and holds the switch behind an explicit approval.

The model production test decides on non-inferiority plus strictly lower cost, and the model-swap verdict on real traffic is parity verified, regressed, confounded, or insufficient data, with the cost delta per arm, again per tenant. Converra does not run training and does not use customer data to train any model without a written, tenant-exclusive election. No open-weight production verdict for a customer agent has been published yet, and no routing policy has been shown to transfer across Converra customers; both remain unobserved until measured.