How Do You Detect Quality Drift After Moving Traffic to the Open Model?

A closed model changes when the provider ships a new version. A self-hosted open model with pinned weights does not change at all. So drift after a transfer has a narrower set of causes: the mix of requests shifts, a new product feature sends request shapes the training data never contained, upstream tools return different results, the prompt or retrieval layer is edited, or the customer's own users change behavior. The model answers a different population than the one it was gated on.

That narrowing is useful. It means drift detection is mostly about watching the inputs and the outcome by segment, not about diffing the model. It also means a regression that appears without any traffic change points at something in the serving path, such as a quantization or configuration change, rather than at the weights.

The gate scored a specific outcome: risk tier matched the answer key, the required findings were present, the tool result was cited, nothing unsupported was asserted. Production monitoring has to score the same outcome, defined the same way, or the two will disagree and one of them will be wrong. A dashboard that reports a generic quality score alongside a gate that measured evidence-bound correctness cannot tell you whether the gate's promise still holds.

Write the outcome definition once, version it, and stamp the version on every scored run. When the definition changes, backfill or start a new series; do not let a live recompute silently diverge from persisted scores. If the outcome cannot be scored for a run, mark it unscored and keep it out of the denominator. A rate computed over runs that could not be measured is not a rate.

Post-switch comparisons only work when each run records which model and which adapter produced it. A prompt hash is not enough, because a model swap keeps the prompt identical. If the runtime does not stamp the served model identity on the trace, the pre and post populations blur together, escalations to the closed model get counted as open-model wins, and the verdict comes back confounded.

Record model identity, adapter version, serving configuration, and whether the run was a fallback. Then every drift metric can be computed per arm. The open model's rate on requests it actually answered is the number that matters; the blended rate across both models measures the routing policy, which is a different question.

Drift shows up in a segment before it shows up in the total. If the traffic mix shifts toward a request class the model handles poorly, the overall rate drifts slowly while that class's rate was bad from day one. If a new class of request appears that the training data never contained, the overall rate barely moves while the model fails every one of the new requests. Segment rates catch both; the total catches neither in time.

Use the same classes the gate used: request type, evidence condition (supported, partial, none), context-length band, and tool-use pattern. Track each class's volume share and outcome rate over time. A share moving is a leading indicator; an outcome rate moving is the drift itself. The table below lists the signals worth computing and what each one means when it changes.

A fine-tuned model that starts meeting unfamiliar traffic often fails by retreating to its default: the most common risk tier, the most common tool, a generic no-findings report, or a refusal. The outcome score catches this late, because default answers are right often enough to look acceptable in aggregate. The default-answer rate and the distinct-output ratio catch it immediately, because they measure whether the model is still discriminating between cases.

Set these up before launch with the values the gate observed. If the gate saw a no-findings rate of one in five, a production rate of one in two on the same request class is a drift alarm regardless of the score. Length distributions belong in the same family: a model that goes terse on hard cases is often abstaining without saying so, and that shows up as a shrinking p10 output length long before anyone reads a transcript.

Most production scoring uses an automated judge, and the judge drifts too. It was calibrated against reviewed labels on the traffic that existed at launch. New request shapes can push its errors in either direction: false alarms on requests it has not seen, or silence on failures it does not recognize. A judge that goes quiet looks exactly like a model that is doing well.

Draw a fresh reviewed sample on a fixed cadence and on every alert. Sample two ways: at random, so confident judge decisions get audited, and by uncertainty, so borderline cases get resolved. Measure the judge's precision and recall per request class against those labels and compare to the launch calibration. If the judge moved, fix the instrument before acting on the reading. If it held, the drift is real.

A change in the outcome rate after launch has several candidate causes, and only some of them are the model's problem. A prompt edit, a tool schema change, a retrieval index refresh, a product release that changes who sends requests, or a serving configuration change can each move the rate. If any of these happened in the window, the result is confounded and the honest report says so.

Keep a change log with timestamps for every component that touches the agent, and join it to the metric series. When the rate moves, the first question is what else moved. When nothing else moved and the traffic mix is stable, the second question is whether the serving artifact changed. Only after those two are ruled out does the answer become the model itself, which for pinned weights means the traffic reached cases the model cannot handle.

An alert with no agreed response becomes a discussion, and a discussion during an incident is where rollbacks get delayed. Before the switch, write down for each signal the threshold that triggers an alert, the threshold that triggers an automatic ramp-down of open-model traffic, and the threshold that triggers a full rollback to the closed model. Name the owner who can execute each one and confirm the rollback path has been rehearsed.

Use the gate's margin as the anchor. If the gate passed at a non-inferiority margin of five points, a production rate five points below the launch window on a class with enough runs is a rollback signal, not a watch item. Smaller moves are alerts. A rising share of a class the model has never been gated on is a routing change: send that class to the closed model until the gate covers it, then decide whether retraining is warranted, which is part 33's question.

Every drift metric needs its denominator next to it. A class with twelve runs this week cannot support a rate, and a dashboard that renders it as a percentage invites a wrong decision. State the minimum runs per class per window that the pre-registered rule requires, render anything below it as insufficient data, and never fill the gap with a pass.

The same discipline applies to the arms. If most traffic escalated to the closed model this week, the open model's rate is computed over a small and unusual population. Report the population size, the escalation share, and the rate together. A number that looks fine over a tiny denominator is the most expensive kind of reassurance.

Converra's production verification measures a shipped change against the same outcome definition on fresh comparable traffic, with the deployment boundary preserved and concurrent changes recorded, and returns verified, not fixed, or confounded for fixes and parity verified, regressed, confounded, or insufficient data for a model swap. Its fleet view aggregates failure patterns across agents with business-impact scoring, which is the segment-level view this part asks for, and its tool-calling analysis scores how the agent uses its tools from real call evidence.

Thresholds, the rollback policy, and the definition of the outcome that matters remain the customer's decisions, and the model itself runs on the customer's provider or infrastructure. Converra records model identity per run and per switch so the arms stay separable, but it cannot detect drift on traffic it does not receive. No open-weight production verdict for a customer agent has been published yet; what a given agent's traffic does after a switch is unobserved until it is measured there.