What Happens When the Closed Teacher Model Changes?

In closed-to-open transfer, the training targets come from the closed model, either directly as its responses or indirectly through a reviewer who edited them. That makes the closed model a labeling function: the same input produces a specific output, and the dataset is a sample of that function. When the provider updates the model, the function changes. Some inputs get the same output. Some get a different one. You do not know which without re-running them.

Providers often expose an alias that resolves to whichever version is current. An alias in the dataset manifest is a claim that the labels came from a moving target. Record the specific version identifier the provider returned at labeling time, plus the date, and treat a missing version as a defect in the dataset rather than a detail to fill in later.

Once a student passed the gate and the production verdict said parity, its weights are fixed. A provider update cannot reach into them. Whatever the closed model does next, the agent on the open model keeps doing what it was verified to do. For a team that has lived through a silent provider update breaking a tuned prompt, this stability is not a side benefit. It is one of the main reasons to do the transfer.

The stability has a cost. The student also does not get whatever the new teacher version does better. If the update fixed a failure the student inherited, the student still has it. Whether to chase the improvement is a retrain decision, covered in part 33, and it should be made on reviewed evidence about the student's behavior, not on the provider's release notes.

Three things move. First, the label distribution for any new examples: the answers the new teacher gives to fresh production inputs may differ in tone, length, structure, or substance from the ones the old teacher gave. Second, the gate's incumbent: if the gate compares the student to the closed model, the closed model's own scores on the frozen holdout change, and a parity margin measured against the old scores may no longer mean the same thing. Third, any judge that uses the closed model to grade outputs, which now grades differently.

None of these is visible from inside the student. All of them are visible if you re-run the incumbent and the judge on the frozen holdout and compare to the receipts from the last gate. That re-run is the first thing to do after a teacher change, and it is cheap relative to a retrain.

The tempting shortcut after an update is to re-run the old inputs through the new teacher and refresh the labels. That destroys the dataset's identity. The examples a reviewer adjudicated under the old labels now have different targets the reviewer never saw. The protected holdout, whose answer keys were checked once, would need checking again. And every gate result in the history was measured against a dataset that no longer exists.

Keep the old version frozen with its teacher receipt, and build a new version for new examples. If the old labels are genuinely wrong in light of what the new teacher shows, that is a review finding: a reviewer decides, example by example, whether the intended behavior changed, and the decision is recorded with lineage. Part 12 is about that lineage.

A dataset that grows across a teacher change will contain examples labeled by two versions. That is fine if every example carries its own teacher version, so a later analysis can ask whether the student's failures cluster on one teacher's labels. It is not fine if the version lives only in the dataset's header, because then the two populations are indistinguishable and a systematic difference between them is invisible.

Per-example teacher lineage also makes it possible to check whether the two versions disagree on the same inputs. Run a sample of old inputs through the new teacher without changing any labels, and measure the disagreement rate. A low rate means the update was mostly cosmetic for your workload. A high rate means the new examples are teaching something different from the old ones, and the gate needs to know.

A parity gate is only meaningful if the incumbent is competent on the task. A good gate checks this first: the closed model has to reach some minimum on the holdout before anyone compares the student to it. When the teacher changes, that check has to be repeated, because an update can make the incumbent worse on your specific workload even if it is better on public benchmarks.

If the incumbent's competence dropped, the gate is invalid for that teacher version, not passed or failed. If it rose, the parity margin got harder to meet, and a student that passed against the old baseline might not pass against the new one. Neither outcome says anything about the student. Both say the comparison changed, and the record should show it.

Retraining because the teacher improved assumes the improvement matters for your agent's requests. Often it does not. A general reasoning upgrade may change nothing about how a support agent looks up an order. The evidence that a retrain is worthwhile comes from the student's measured behavior on production traffic and from reviewed failures, not from the provider's changelog.

The reverse is also true. If the new teacher regressed on your workload, the frozen dataset and the accepted student are unaffected, and the right move may be to stop using the new teacher for labeling until it is checked. Pin the teacher version for labeling the way you pin a dependency, and upgrade it deliberately.

If part of the traffic still routes to the closed model, either as a permanent split or as a fallback for hard requests, that arm changes under a provider update while the open arm does not. A production comparison between the two arms measured across the update is comparing the student to two different incumbents. Record the served version on every run in the closed arm so the analysis can split the window.

The same applies to any judge that grades production runs using the closed model. A judge version change in the middle of a measurement window can produce an apparent drift that is entirely in the grader. Pin judge versions the way you pin teacher versions, and treat a change as a boundary in the data.

Converra records model identity on every run and every deployment, which is what makes a provider update visible as a boundary rather than a mystery. Its benchmarks re-run candidate and incumbent models on the same scenarios with median scoring and measured cost and latency, so a changed teacher can be re-measured on the agent's own workload rather than assumed from release notes. Its fine-tuning workflow freezes each dataset version with its lineage, so a new teacher produces a new version instead of overwriting an old one.

Converra does not run training, and it does not use customer data to train any model without a written, tenant-exclusive election. No customer verdict across a teacher change has been published; how a specific agent's student compares to a specific new teacher version is unobserved until it is measured on that agent's traffic.