When Should You Retrain the Student, and When Should You Keep the Current Model?

Nothing about a fine-tuned model degrades on its own. The weights that passed the gate in March are the same weights in September. What changes is the world around them: the requests, the tools, the instruction, the evidence users bring. If none of those has moved, the model that was at parity is still at parity, and a retrain can only introduce risk without a corresponding reason.

Teams retrain on a schedule because it feels like maintenance. It is not maintenance; it is a release. Every retrain produces a different model that has to earn its place again. The right frame is that the current model is the incumbent, and a candidate only gets built when there is evidence the incumbent is losing.

The strongest reason to retrain is a class of request the current model handles badly and the training data never covered. It has to be a class, not an incident. One bad conversation is a review item. Twenty conversations that share a mechanism, confirmed by a reviewer who looked at the evidence rather than a score, are a gap in the dataset.

The examples that document the gap are exactly the examples the next dataset version needs. Adjudicate them, record the intended behavior, and hold some back as protected evaluation cases so the next gate can check whether the candidate actually learned the class rather than memorizing the examples it was shown. Part 6 covers how those examples become trustworthy.

A student was trained to behave under a specific instruction, with specific tools returning specific shapes. When the instruction gains a new rule, or a tool's response schema changes, the training examples describe a system that no longer exists. Some of the old behavior still applies. Some of it is now wrong, and the model will keep doing it confidently.

Not every edit needs a retrain. A wording change that does not alter any decision the model makes can be validated with the existing gate. A new rule that changes what a correct output looks like needs new examples that show the rule being followed, and a gate that checks it. The test is whether the frozen evaluation set still describes correct behavior; if the answer keys would change, the model needs to change too.

After the switch, keep scoring production runs on the same outcome the gate used. When that score moves outside the band the parity verdict was based on, and stays there across a window long enough to rule out a traffic blip, drift is real. Part 45 covers how to detect it without chasing noise.

Drift is only a retrain trigger when its cause is something training can fix. If the requests changed and the model's behavior on the old requests is unchanged, retraining on the new requests can help. If a tool started returning worse evidence, the model is faithfully reporting worse evidence, and the fix belongs in the tool. Diagnose the mechanism before deciding the remedy.

Production produces reviewed examples continuously: corrected failures, confirmed successes, edge cases a reviewer resolved. When the pool of new, admitted examples is large enough to change the dataset's coverage, a new version is worth building. Twenty new examples on top of six hundred is not a new dataset. Two hundred that cover three previously thin classes might be.

Coverage matters more than count. The question is whether the new examples add distinct families of inputs, or more variants of families the model already handles. A dataset that grows only in its densest region trains a model that is more confident on what it already knew and no better on what it did not.

The calendar. Quarterly retraining is a habit from a world where the model was a statistical fit to a moving distribution and retraining was cheap and safe. Here the model is a behavior contract, and retraining is a release with gate cost and regression risk. Do it when there is a reason.

Unreviewed traffic. Production conversations are candidate learning signals, not gold data. A retrain on unadjudicated transcripts teaches the model whatever it and the users were doing, including the failures. The incumbent's own outputs are the worst case: a model trained on its own responses converges on its own mistakes and loses the corrections the original teacher supplied. Part 7 explains how hallucinations enter a dataset this way.

Every retrain gets a new model identifier, a new dataset version with lineage to the examples it added, and a fresh run against the same frozen gate the incumbent passed. The incumbent stays deployed until the candidate passes the gate and then passes a production parity check of its own. A candidate that fails is recorded as a failed candidate, and the incumbent keeps serving.

This is what stops a retrain from silently replacing a working model with a worse one. If the pipeline can overwrite the serving model with a new training output, it eventually will, and nobody will know which version produced the regression. The identity chain is the control: dataset version, candidate id, gate result, approval, deployment marker, verdict.

The temptation on the second retrain is to fold the old holdout into the training data because it is well-reviewed and the model has already seen the gate. Do not. The holdout is the only evidence that connects the new candidate to the old one. Once it enters training, the next gate result cannot be compared to the last, and the series of releases loses its baseline.

Add new protected cases as new failure classes appear, so the holdout keeps up with the workload. Retire cases only when the contract they test no longer exists, and record the retirement. A holdout that grows in coverage and never leaks into training is what lets you say the fourth release is at least as good as the first.

Most of what looks like a model problem in production is not. The instruction is ambiguous, retrieval returns stale documents, a tool times out and the model works around it, a new customer segment asks questions the agent was never scoped for. Retraining the model on any of these teaches it to compensate for a defect that should have been fixed at its source.

Before building a candidate, ask what the reviewed evidence says the mechanism is. If the model made the right decision given what it was shown, the model is not the problem. Keeping a passing model and fixing the prompt, the tool, or the routing is cheaper, faster, and does not put a new set of weights in front of users. Part 4 is the decision procedure for which fix to reach for.

Converra's diagnosis and production verification exist to answer the question that precedes a retrain: is this a model problem, a prompt problem, or a tool problem, and is it real on live traffic? Its fine-tuning workflow curates reviewed production runs into versioned datasets with protected splits and per-example admission decisions, so a new dataset version has lineage to the evidence that justified it, and records each training job and gate result against that version. Training runs on the customer's provider or infrastructure; Converra does not run it.

A new candidate goes through the same explicit approval and the same model production test as the first one, with the verdict reported as parity verified, regressed, confounded, or insufficient data. No customer retrain verdict has been published, and the outcome of any retrain on a given agent's traffic is unobserved until it is measured there.