How Do You Know Whether a Fine-Tune Learned the Task Rather Than the Test Set?

A fine-tune that memorized is near-perfect on evaluation cases that resemble its training templates and falls apart on cases that do not. The fall is not gradual. Change the wording of a request, reorder the evidence, or introduce a scenario type the training set never contained, and the model reverts to the base's default answer or to the majority class from training.

Two other signs travel with it. Free-text fields reuse phrases from training targets instead of details from the current input. And confidence stays high on the cases it gets wrong, because the model is matching a shape it has seen, not weighing evidence it has not. Any one of these can appear for other reasons; all three together on unseen shapes is memorization.

The common practice is to shuffle examples and hold out a fraction. When training data is generated from templates, or collected from a production stream where many conversations share a structure, a random split puts siblings of every training example into the holdout. The model then scores well on the holdout by recognizing the template, and the split reports generalization that does not exist.

The same problem appears with real customer data. Conversations from the same end customer, the same ticket, the same document, or the same week share vocabulary and structure. Splitting them at random leaks the training distribution into the test distribution. The unit of the split has to be the thing that carries shared structure, not the individual row.

Identify the scenario templates or families the data came from, and assign a share of them wholesale to the protected holdout. Every example instantiated from a held-out template stays out of training, out of validation, and out of any tuning. A reasonable floor is to hold out at least a fifth of the templates and no fewer than three, so the gate measures behavior on scenario types the model has never seen.

Cap the share any single template can contribute to either side. If one template supplies half the training families, the model learns that template; if one template supplies half the holdout, the gate measures that template. A cap of roughly a tenth of holdout cases and a bit more of training families per template keeps both sides varied enough for the comparison to mean something.

Define a family key that groups every example sharing an entity, document, customer, or ownership graph, and assign whole families to a split with a deterministic rule such as sorting by a hash of the key. Do this before agent outputs are generated and before any reviewer sees a case. Splitting after review lets reviewer attention and correction effort leak toward the cases that will later be scored.

Enforce it mechanically. A dataset build should refuse to complete if any family appears in more than one split, if any held-out template has a family in training, or if any holdout content hash appears in the training export. These are fail-closed checks, and part 19 goes through the full list of leakage vectors they need to cover.

Take cases the model gets right and rewrite the input without changing the correct answer: different wording, reordered evidence, a longer preamble, a different customer name. A model that learned the task should give the same decision. A model that memorized will often change its answer, because the shape it matched is gone. Paraphrase is also how contamination hides from string matching, which the rephrased-samples contamination paper documents for benchmarks and which applies just as well to a customer dataset.

Keep the probes out of training and out of the protected holdout. They are a development instrument, useful for diagnosing a candidate, and they should not be the gate. Score them as a separate row: agreement rate between the original and the paraphrased case, per model.

The strongest probe is a case that looks almost exactly like a training example and has a different correct answer. Change one fact that flips the decision: a sanction that is lifted, a policy that no longer applies, a missing document that is now present. A model that reads the evidence changes its answer. A model that memorized the training example repeats the training target.

These adversarial near-duplicates are cheap to write from an existing training set and expensive for a memorizing model to survive. Report the flip rate per model. An incumbent closed model with a good prompt should flip nearly every time; a candidate that flips on fewer than most of them has not learned the rule the flip depends on.

With the controls in place, the informative number is the difference between the candidate's score on cases from seen templates and its score on cases from held-out templates, compared with the same difference for the base and the incumbent. A candidate that is ten points better than the base on seen templates and no better on unseen ones has learned the templates. A candidate that is better on both has learned something that transfers.

The table below is the layout that makes the gap visible. Every model gets the same rows, and the pass decision rests on the unseen rows.

All of this only works if the holdout templates, family assignments, probes, and scoring weights are fixed and hashed before any candidate output is opened. A team that sees a suspicious result and then adds unseen templates has built a second test that the first result already influenced. A team that moves a template into training because the candidate failed on it has retrained on the holdout in all but name.

Record the preregistration: the list of held-out templates, the family split rule, the probe set, the scoring composite, and the sample size, each with a content hash. Evaluate. If the candidate fails on unseen templates, the answer is more distinct training families at the complexity of the unseen ones, and a new candidate against the same frozen gate. One corrected rerun for a diagnosed execution defect is reasonable; a third attempt against the same holdout is tuning on it.

Suppose a supplier-risk agent's data came from 14 scenario templates. Three templates, say a sanctioned parent reached through an intermediary, a clean supplier with stale certification, and an ownership graph with a circular reference, are withheld entirely. The holdout has 100 families, 25 seeded adverse, with no template above ten percent of cases. Two candidates and the base are scored.

Candidate A scores 88 on seen templates and 61 on unseen; the base scores 64 and 62. Candidate B scores 84 and 81. A's twenty-seven point gap, against a two point gap for the base, is memorization, and its seen-template win is worthless. B's small gap and uniform improvement over the base is task learning, and B is the only one that should proceed to the non-inferiority comparison with the incumbent. These numbers are illustrative; the point is that without the unseen slice, A would have looked like the better model.

Converra's fine-tuning workflow curates reviewed production runs into versioned datasets with protected splits, so the families a candidate is scored on are assigned before review and never enter training. It records the training job and its gate result against that dataset version and holds the production switch behind an explicit approval. Its model benchmarks run every candidate and the incumbent on the same scenarios generated from the agent's instruction and ship a winning switch as a reviewed pull request.

Converra does not run training, and it does not use customer data to train any model without a written, tenant-exclusive election. No open-weight production verdict for a customer agent has been published yet. Passing a memorization-resistant gate is what earns a candidate a production test; whether it holds parity on real traffic is unobserved until that test returns parity verified, regressed, confounded, or insufficient data.