How Do You Prevent Training Data From Contaminating the Holdout?

The naive picture of contamination is a test row that also appears in training. That is the easiest leak to catch and the rarest in practice. The leaks that matter share structure without sharing bytes: two documents about the same supplier, two conversations from the same end customer, two cases instantiated from the same scenario template, or a paraphrase of a training input. A model learns the structure, and the holdout rewards it for having seen it.

So the split unit has to be the thing that carries shared structure. For an agent over entities, that is the connected component of the entity graph: a supplier plus every parent, subsidiary, owner, and address linked to it. For a support agent, it is the end customer or the ticket thread. Every document, tool result, and run about anything in that unit carries the same family key, and the family goes to one split as a whole.

Leakage has technical vectors and process vectors, and the process vectors are the ones teams miss. The table lists both, with the control that closes each.

The order of operations decides whether the split is clean. Generate or collect the families, assign the family keys, assign splits by a deterministic rule such as sorting families by a hash of the key, and only then generate agent outputs, run the teacher, or open a review queue. A split assigned after review inherits whatever attention the reviewers gave to the cases that will later be scored. A split assigned after teacher runs invites filtering the holdout by teacher agreement, which quietly removes the cases the incumbent finds hard.

Deterministic assignment also makes the split reproducible. Anyone can recompute which family belongs where from the keys and the rule, and a build that produces a different assignment from the same inputs is a bug rather than a mystery.

Family-level splitting stops entity leakage but not template leakage: a family in the holdout can still be a new instance of a template the model trained on hundreds of times. Withhold a set of templates wholesale, so the holdout includes scenario types the model has never seen. At least a fifth of the templates and no fewer than three is a workable floor.

Then cap what any single template contributes. No template supplies more than roughly a tenth of holdout cases, and no template supplies more than a modest share of training families. Without the cap, one prolific template dominates whichever side it lands on, and the gate measures that template rather than the task. Part 16 shows how the seen-versus-unseen gap that this control creates is read.

Content-hash deduplication catches exact and near-exact duplicates and is worth running, but it must not be the mechanism that separates splits. Two documents about the same supplier have different hashes and identical learning content. Use hashes to remove duplicate fixtures within a family and to assert that no holdout content hash appears in the training export. Use the family key for everything else.

String matching also misses paraphrase. The contamination research on rephrased samples showed that simple variations of test data bypass conventional decontamination and that a meaningful share of a well-known benchmark existed in pretraining corpora in rephrased form. A customer dataset is far smaller and the same failure is far easier to produce by accident.

A dataset build should refuse to produce an export unless every check passes, and the checks should run in an order that reports the most fundamental failure first. One, every run carries a valid family key, template key, and fixture hash, and the hash matches the manifest; anything missing or mismatched is rejected and counted. Two, no two runs share a source content hash; a duplicate aborts the build. Three, no family appears in more than one split, no held-out template has a family in training or validation, and no holdout hash appears in the training export; any of these aborts.

Four, template caps hold on both sides. Five, a diversity floor is met: enough distinct families and enough distinct templates in training that the model has something to generalize from rather than a handful of shapes to memorize. Record the distinct counts per split in the manifest. A build that passes all five produces a manifest with content hashes that the gate later verifies before it will run.

The process leaks are the subtle ones. A reviewer who has read the holdout cases will, without intending to, write corrections and choose training examples that resemble them. A teacher model whose outputs on holdout families are exported becomes a source of holdout answers in training. A judge prompt refined by looking at holdout disagreements is tuned on the holdout.

Close these by scope. Protected families are never opened in the review tool. Teacher runs on protected families are stored for the gate and never exported. Judge prompts, scoring weights, and thresholds are tuned on a development split that is disjoint from both training and holdout, and their hashes are frozen before any protected output is read. Part 12 covers the lineage that makes these scopes auditable.

Once a candidate has been evaluated on the holdout and a decision has been made from the result, that holdout has influenced the next candidate. Moving a failed template into training, regenerating cases the candidate found hard, or adjusting the margin after seeing the gap are all forms of training on the test set with extra steps. A single corrected rerun for a diagnosed execution defect, such as an adapter that was not served, is defensible. A third candidate against the same holdout after two content failures is not.

When the holdout is spent, the answer is more families: generate or collect new protected families under the same split rule, freeze a new contract, and evaluate the next candidate against it. Keep the old holdout's results as history. This costs data, which is why the sizing in part 17 should be done before the first freeze rather than discovered at the second.

Suppose a generator produces 240 supplier families across 14 templates, 60 of them seeded with an adverse finding. Before any agent output exists, the families are hashed and sorted into 120 training, 20 development, and 100 protected, and three templates are assigned wholesale to protected. The build checks find two families whose fixture documents hash identically after a name change; they are merged. It then finds one template supplying 18 percent of protected cases, over the cap, and aborts.

The team adds families under other templates rather than removing the over-represented ones, rebuilds, passes all five checks, and freezes the manifest. Reviewers see only training and development families. The gate later refuses to run until the manifest hash matches. The numbers are illustrative; the sequence is the point, and the abort is the feature.

Converra's fine-tuning workflow curates reviewed production runs into versioned datasets with protected splits and per-example admission decisions, so protected families are assigned before review and stay out of the review queue and the training export. It records the training job and its gate result against a specific dataset version and holds the production switch behind an explicit approval. Its synthetic personas and simulation scenarios are derived from production patterns for testing changes outside the live stack.

Converra does not run training and does not use customer data to train any model without a written, tenant-exclusive election. No open-weight production verdict for a customer agent has been published yet. A clean holdout makes the gate result trustworthy; whether the candidate holds parity on live traffic is unobserved until the model production test measures it.