How Do You Preserve the Source, Evidence, and Reviewer Lineage of Every Training Example?

Lineage exists to answer specific questions quickly. Which production run did this example come from, and does that run still exist? What evidence was in the trace when the reviewer approved the target? Who made the decision, on what date, against which version of the agent's instructions? Was this example reviewed directly or did it inherit a decision made on a representative case? Which dataset versions include it, and which trained models used those versions? Is the customer's consent for this data still in force?

If any of these takes more than a query to answer, the dataset is not auditable, and a fine-tuned model built on it cannot be defended when a customer, a reviewer, or a regulator asks. The failures this prevents are not hypothetical: a withdrawn consent that cannot be traced to the examples it covers, a corrected label that leaves stale copies in three dataset versions, and a model that was trained on a target nobody can now justify.

The chain has six kinds of artifact. A source run: the production conversation or trace, identified by its own stable id. A finding: what the evidence pipeline or a reviewer observed about that run, with a pointer to the exact evidence. A decision: the reviewer's judgment about intended behavior, scoped to a cluster or a case, bound to the instruction version in force. A training example: the input and target derived from a run under a decision. A dataset version: a frozen, enumerated set of examples with counts and split assignment. A training job: a run of the trainer against exactly one dataset version, with its provider receipt and gate result.

Each artifact gets an identity when it is created and keeps it forever. Content hashes work well for examples and dataset versions because they make tampering detectable; opaque ids work for decisions and jobs. What matters is that an identity is never reused for changed content. Changing content creates a new artifact with a new identity and a link back to the one it supersedes.

Reviewers do not look at every example. They look at a representative case for a cluster of similar findings, decide what the intended behavior is, and that decision is propagated to the cluster's other members after a per-member applicability check. This is the only way review scales, and it is also where lineage most often goes missing. An example admitted by propagation carries a different evidentiary weight from one a reviewer looked at directly, and the dataset must say which is which.

Store the flag on the example itself, along with the id of the representative case whose decision it inherited and the result of its own applicability check. Then the dataset manifest can report how many examples were directly reviewed and how many were propagated, and a random audit can target propagated examples specifically. When a propagated example fails its check, it is not silently dropped; it is recorded as rejected or unresolved with the reason, so the count of candidates always reconciles.

A decision about intended behavior is only meaningful relative to the agent's instructions at the time. If the instruction later changes so that the old behavior is now correct, or the new behavior is now required, the decision does not become wrong retroactively; it becomes scoped to a version that is no longer current. Record the instruction version on every decision, and when a customer changes the instruction, treat every decision bound to the old version as needing reassessment before its examples are reused.

This separates two things that are easy to conflate: a new preference and a historical correction. A customer deciding they now want a different tone is setting a target for future examples, not relabeling the old runs as violations. Lineage that carries the instruction version lets you build a new dataset for the new preference without rewriting history, and lets a reviewer see why an old example was approved under rules that no longer apply.

When a reviewer changes their mind, the correct operation is not an update. It is a new decision that supersedes the old one, followed by reassessment of every example that depended on the old decision. Some of those examples will be re-admitted under the new decision, some rejected, and some marked unresolved. Every dataset version that contains one of the affected examples is now known to contain a superseded decision, and any model trained on it carries that fact in its lineage.

Fail closed here. A dataset version whose examples include a superseded decision should not be used for a new training job until the affected examples are reassessed and a new version is frozen. The old version stays as a historical record; it is never modified. This is more work than editing a row, and it is the only way the training job's claim, that it was trained on reviewed examples under decisions in force, stays true.

Training data derived from a customer's production runs exists under a specific grant of rights, and that grant can end: a contract lapses, consent is withdrawn, a data-retention window closes, a run is deleted at the customer's request. Lineage is what makes the consequence tractable. From the source run, follow the links to every example derived from it, every dataset version containing those examples, and every training job that used those versions.

The examples become invalid, the affected dataset versions are marked as containing invalid examples, and the models trained on them are flagged. What happens to the models is a product and legal decision that varies by agreement, but it cannot be made at all without the chain. A team that stores training examples as a flat JSONL export with no back-links to source runs has no way to honor a withdrawal except by discarding everything.

Metadata copied onto rows, such as a family id or a template id stamped on each run, is convenient for filtering and dangerous as a record. Copies drift. Keep one manifest per dataset version that enumerates every example hash, its split, its family, its template, its source run, and its decision, and make the manifest's own content hash the dataset version's identity. Any row-level copy is verified against the manifest at curation time and rejected on mismatch.

The same discipline applies to the evaluation side. The protected holdout's manifest is frozen before any candidate exists and its hash is recorded in the gate's pre-registration. At evaluation time the harness recomputes the hash and refuses to run if it differs. A gate whose holdout could have changed since the freeze is not a gate.

Suppose a reviewer approves a representative case for a cluster of 18 findings about an agent citing an out-of-date policy. The decision is recorded with the cluster id, the reviewer, the date, and instruction version 14. Seventeen members pass the applicability check and are admitted as propagated examples; one fails because its trace shows the current policy was in the context, and it is recorded as rejected. Dataset version A freezes with those 17 examples among 200.

A month later the customer updates the instruction to version 15, which changes what the correct citation is. Every decision bound to version 14 is flagged for reassessment. The reviewer issues a new decision for the cluster under version 15; 12 examples are re-admitted with new targets and new hashes, 5 are rejected. Dataset version B freezes with the new examples. Version A remains on record as the version that trained the earlier model, with a note that 17 of its examples were superseded. Nothing was edited; everything is explainable.

Converra's fine-tuning workflow records the chain this part describes: production runs and their evidence findings, cluster-level review decisions bound to the agent's instruction version, per-example admission with direct versus propagated lineage, and frozen dataset versions with enumerated examples and split assignments. The training job and its gate result are recorded against exactly one dataset version. Training runs on your provider or infrastructure; Converra records the job rather than running it.

Converra does not use customer data to train any model without a written, tenant-exclusive election, and its trust page states the default: no training on customer data. No open-weight production verdict for a customer agent has been published yet, and lineage does not change that; it only makes whatever the verdict is auditable.