How Should Customers Approve, Monitor, and Roll Back a Model Replacement?

An approval should name the exact thing being approved: the base model and revision, the adapter identifier, the training job, the dataset version, and the gate result those produced. It should also name what it authorizes: a switch of one agent's model, on one date, under one rollback policy. An approval of an open model in general, or of a candidate whose gate result is described but not attached, cannot be checked later against what actually shipped.

Scope it to the candidate the gate evaluated. If a second training attempt produces a new adapter, the old approval does not carry over, even if the second attempt was meant as a fix. Each candidate gets its own gate result and its own approval, and the workflow should refuse a switch whose serving model does not match the approved identity.

The comparison from part 30 is the input, and the approver should be able to answer four questions from it without asking anyone. Did the candidate pass every element of the pre-registered gate, and by how much? What is the measured cost and latency on the configuration that will be deployed? What traffic will the candidate receive first, and what fraction? Who owns rollback, and what triggers it? An approval recorded without those answers is a signature, not a decision.

Record the answers with the approval. Later, when the production measurement comes back, the record is what lets the team say whether the outcome matched the expectation or drifted from it.

The gate measured something specific: correctness on a task, recall on a seeded class, a mode check, a cost per case. Production monitoring should measure the same thing on live traffic. Switching to a different metric after deployment, such as overall user satisfaction or a generic quality score, breaks the chain between what was approved and what is being watched.

Measure before and after on comparable traffic, with the switch boundary preserved so that runs can be assigned to the old or new model without ambiguity. If the candidate is served to a fraction of traffic first, the comparison is between arms on the same period; if it replaces the old model outright, the comparison is between periods, and concurrent changes to the prompt, tools, or routing have to be recorded because any of them can confound the result. Part 32 covers what the production measurement has to look like to count as proof.

A rollback threshold is a rule written in advance: if the measured outcome on the new model falls below the old model's by more than the agreed margin over a defined window, or if any seeded class shows a miss, or if error rate or latency exceeds a stated bound, the switch is reverted. The thresholds should mirror the gate's elements so that production is judged by the same standard the candidate passed.

Assign an owner. Someone specific is responsible for watching the measurement and executing rollback, and that person's authority to do so without further approval is part of the switch record. A rollback that needs a meeting is a rollback that happens after the damage.

Rollback should be the same operation as the switch, run in reverse: change the model identifier the agent resolves to, and confirm the runtime picked it up. That is only possible if the previous model is still deployed and warm. Tearing down the closed-model integration or the previous adapter to save cost turns rollback into a redeployment with its own startup time and its own chance to fail.

Keep the previous model deployable for as long as the rollback window lasts, and define that window in advance. Once the production measurement has returned a verdict and the window has closed, the old model can be retired as a separate, recorded decision.

Run the rollback once, on purpose, before the switch is trusted. Switch to the candidate, confirm the runtime serves it, switch back, confirm the runtime serves the previous model, and time both operations. A rollback that has never been executed is a plan, and plans fail in ways that only execution reveals: a stale cache, a config that is read at startup rather than per request, a resolver that falls back to a default the team forgot about.

Record the rehearsal alongside the rollback policy. The time it took is the answer to how long a regression will persist once detected, which is an input to how much traffic the candidate should receive first.

A pull request that changes a model identifier proves that a change was proposed, reviewed, and merged. It does not prove the change deployed, that the runtime read it, or that traffic is reaching the new model. Configuration can be cached, overridden by an environment variable, or ignored by a code path that resolves the model somewhere else. The same is true in reverse: reverting the pull request does not prove traffic returned to the old model.

Confirm the runtime directly. Sample live requests after the switch and after any rollback, and read the model identifier from the call records or the provider's usage data. Until that confirmation exists, the switch is a merged change, and the monitoring window has not started.

Where the serving path supports it, send the candidate a fraction of traffic first and compare arms over the same period. This makes the before-and-after comparison a same-time comparison, which removes most confounders, and it bounds the damage a regression can do before the threshold triggers. The fraction should be large enough to reach the case counts the gate needed within a reasonable window, and small enough that a regression at that fraction is tolerable.

Promote to full traffic only after the fraction has returned a result against the thresholds. Promotion is a second switch, with its own runtime confirmation, and the rollback policy continues to apply after it.

Converra's fine-tuning workflow records the gate result against a specific dataset version and holds the production switch behind an explicit approval on that candidate identity. The model production test splits live traffic between the incumbent and the candidate, decides on non-inferiority plus strictly lower cost, and promotes by switching the model; production A/B tests roll back if the challenger underperforms. The model-swap verdict on real traffic reports parity verified, regressed, confounded, or insufficient data, with the cost delta per arm.

The approval itself, the rollback thresholds, the rehearsal, and the runtime confirmation on the customer's own serving path remain the customer's; Converra records the decision and measures the outcome, and it does not run the training. No open-weight production verdict for a customer agent has been published yet, so a model replacement's result on a given agent is unobserved until the measurement returns.