What Should Happen Automatically When the Open Model Fails or Times Out?

An open model on a dedicated endpoint fails in more ways than a closed API does, and each way needs a different automatic response. A timeout means the endpoint is slow or saturated. A server error means it is down or restarting. A rate limit means capacity is exhausted. Invalid output means the model answered but the answer cannot be used. Truncation means it ran out of output budget. Context overflow means the input never fit. A refusal means the model declined a request it should have handled.

Treating all of these as one generic error produces a fallback policy that is wrong for most of them. A retry helps with a transient server error and hurts with a saturated endpoint. Repairing invalid output helps with a stray character and hides a model that has stopped producing the schema. The table below pairs each class with detection, the automatic action, and what to record.

The fallback target is usually the closed model the agent ran on before the transfer, with the same prompt and tools, because that path is already proven on the workload and its cost is already known. It is not the only option. Some requests can be served from a cached or templated answer. Some should fail explicitly with a message the user can act on. Some should go to a human queue. The choice depends on the request class and on what a wrong answer costs.

Whatever the target, it has to be exercised regularly or it will rot. A closed-model fallback that has not received traffic in a month may have a stale prompt version, a revoked key, or a changed tool contract. Route a small steady share of requests through the fallback path on purpose, or run a synthetic canary against it, so that the first real failover is not also the first test.

The open model's per-call timeout is the user's total latency budget minus the time the fallback needs. If the user experience allows four seconds and the closed model answers in two, the open model gets at most two seconds before the system must give up on it and start the fallback. A timeout set from the open model's typical latency ignores the fallback and pushes the worst cases past the budget.

Measure the fallback's own p95, not its median, when computing the remainder. Under a burst that saturates the open endpoint, the fallback also sees a burst. Leave headroom for that. If the arithmetic leaves the open model with less time than its own p95 at production concurrency, the transfer has a latency problem that part 40 should have caught, and the honest response is to route a smaller share rather than to widen the budget.

A single immediate retry recovers transient failures such as a dropped connection or a restarting replica. A second retry rarely helps and costs budget. Retrying a timeout is almost always wrong: the endpoint was slow because it was busy, and a retry adds load to the thing that is failing. Retrying a rate limit is wrong for the same reason. Retry server errors once; fail over on everything else.

Use a circuit breaker so retries cannot amplify an outage. When failures on the open endpoint exceed a threshold over a short window, open the circuit: send all traffic to the fallback for a cooling period, probe the endpoint with a trickle of requests, and close the circuit only when the probes succeed. This turns a degraded endpoint into a clean switch to the closed model rather than a cascade of slow failures.

An open model that returns a response has not necessarily returned an answer. Validate the raw output against the agent's schema and contract before anything downstream uses it: required fields present, enumerations valid, identifiers well-formed, numbers within range, cited tool results actually present in the run. A response that fails validation is a model failure and should fail over like any other.

The temptation is to repair. Strip the stray backtick, fill the missing field with a default, truncate the over-long string, and move on. Repairs that fix length are fine when recorded. Repairs that fix structure hide the model's failure from every metric, and part 20 explains how that distorts comparisons. If repair is unavoidable, record it as structured metadata on the run, count the run as repaired in the open model's quality, and alert when the repair rate rises.

A request that exceeds the open model's context limit should never reach the endpoint. Count tokens before the call, compare against the served limit with headroom, and route oversized requests to the closed model or to a path that condenses the input deliberately. Serving stacks that silently drop early tokens produce an answer from a context the agent did not intend, and that answer can look confident and complete while missing the instruction that mattered.

Track the overflow rate as its own signal. A rising share of oversized requests means the traffic is changing shape, which is the drift condition part 45 describes, and it usually means the closed model is quietly absorbing the hardest requests. That is fine as a policy; it is not fine as an accident.

A fallback is a request the open model did not serve. Its cost is the closed model's cost plus the wasted open-model attempt plus the added latency. Its quality is whatever the fallback produced. If the reporting counts the closed model's answer as an open-model success, or drops the request from the open arm entirely, the open model's numbers describe a system that does not exist.

Record fallbacks as structured events with cause, timing, and both model identities, and roll them into two views. The open model's own rate, over requests it actually answered, is the quality claim. The blended rate, over all requests, is the user experience and the cost basis. Part 47 uses both to prove savings; a fallback share that drifts upward is the fastest way for projected savings to disappear.

Set a threshold on the fallback share per request class over a rolling window. Below it, the system runs as designed. Above it, the router reduces the open model's share for that class automatically and alerts an owner. Above a higher threshold, the circuit opens for the class and all of its traffic goes to the closed model until someone looks.

This is the same logic as the drift responses in part 45, applied to reliability instead of quality. It removes the incident-time debate about whether the endpoint is bad enough to pull. The thresholds come from the gate and the load test: a fallback share the economics can absorb, and a share above which the transfer is no longer delivering what was approved.

Before moving traffic, exercise every row of the failure table on purpose. Kill the endpoint and confirm the circuit opens and traffic fails over inside the budget. Return a malformed response and confirm validation catches it and no repaired output reaches the user. Send an oversized request and confirm it never hits the endpoint. Throttle the endpoint and confirm retries do not pile on. Read the recorded events after each drill and check that cause, timing, and model identities are all present.

Repeat the drills after any serving change. A new quantization, a new batch configuration, or a new serving stack version can change latency and error behavior, and the fallback policy tuned for the old configuration may no longer fit. Part 44 treats the served artifact as a new candidate for quality; the same rule applies to its failure behavior.

Converra's production A/B test splits live traffic between the incumbent and the challenger, measures each arm, and rolls back automatically when the challenger underperforms, which is the circuit-breaker behavior this part describes applied to quality. Its verification records model identity per run, so fallbacks to the closed model land in the correct arm and the open model's parity verdict is computed over the requests it actually served, with the cost delta per arm reported alongside.

The serving path itself, including timeouts, retries, output validation, and the fallback target, runs in the customer's infrastructure, and Converra does not operate the endpoint or run training. Whether a given agent's fallback share stays within what its economics can absorb is unobserved until the switch is live and measured; no open-weight production verdict for a customer agent has been published yet.