How Do You Turn Real Conversations Into Trustworthy Training Examples?

The tempting shortcut is to export the closed model's successful conversations and train on them. Two things are wrong with it. First, successful means nobody complained, which is not the same as correct; a confident wrong answer that the user accepted looks identical to a right one in the logs. Second, the closed model's output is its opinion under its instruction, and part of the point of the transfer is to fix behaviors the customer did not like.

So the pipeline has to insert a decision between the run and the dataset. Something has to say, for this run, this is what the agent should have produced, and here is why. That decision needs evidence, an owner, and a record. The rest of this article is the shape of that decision at scale.

Start with automated evaluation of each run against the agent's own instruction and the evidence the run carried: tool results, documents, prior turns. Deterministic checks catch schema violations, missing required fields, and tool calls with invalid arguments. Semantic judges propose findings like the answer contradicts the tool result, or the agent skipped a required confirmation. Each finding points at a step and quotes the evidence.

These are proposals. A judge's finding is a claim about the run that a person has not yet confirmed, and part of the pipeline's job is to keep that status visible. A finding that no one reviews should not become a training label by default, and a run with no findings should not be assumed clean, because judges have recall limits like any other measurement.

A reviewer cannot look at ten thousand runs. They can look at forty clusters. Group findings by the underlying behavior: the rule or instruction involved, the conditions under which it applied, and the version of the instruction that governed the run. Two runs that use different words but fail the same rule under the same conditions belong together. Two runs with similar wording that fail different rules do not.

Clustering by surface similarity, such as embeddings of the final answer, produces groups a reviewer cannot decide on with one judgment. Clustering by behavior produces groups where one decision covers the members, and where the exceptions are visible as members that do not quite fit. Show the reviewer a representative case, the cluster size, the meaningful variations inside it, and access to every member.

For each cluster the reviewer sees the representative run, the original evidence, the proposed finding, and a proposed better response. Their job is to decide, and their options are richer than accept or reject. The finding is correct. The behavior was actually acceptable. It depends on context the cluster does not capture, so split it. The instruction should change instead. There is not enough evidence to say.

The reviewer also edits the target. Approving a diagnosis is not the same as approving what the agent should have said; the training example is the desired output, and the reviewer owns that text. A decision also has to be bound to the instruction version it applies to, so a new preference can define a future target without relabeling past behavior as wrong. The table below lists the decision states and what each does to the members.

A cluster decision is not blanket approval. Every member example is checked on its own: does the reviewer's decision actually apply to this run, and is the complete proposed target correct for this run's specific inputs? A member that differs from the representative in a way that changes the answer is an exception. It splits out or goes back for review. It does not get admitted because its cluster was approved.

Record which examples the reviewer looked at directly and which were admitted by propagation from a cluster decision. Direct and propagated examples have different confidence, and later measurement, such as random inspection of propagated examples, needs to know which is which. A final checker, which may be automated, can accept, reject, or abstain on each propagated member; abstain is a valid answer and an abstained example stays out.

A dataset built only from corrected failures teaches the model that every input is a failure case. It also removes the examples of the agent doing its job well, which are most of production. Admit verified successes: runs the reviewer confirmed were correct, with their evidence. Admit counterexamples: cases where the right answer was to refuse, escalate, or say the evidence was insufficient, so the model learns those shapes are legitimate outputs.

Do not treat an unflagged run as a verified success. A run with no judge findings is a run the judges did not flag, and the difference matters when judges have imperfect recall. Verified means someone or something with an independent check confirmed the outcome. Part 11 takes up the balance between common cases and difficult ones once the admitted set exists.

The pipeline will produce cases nobody can decide. The evidence is incomplete, the instruction is silent, the reviewer disagrees with the judge and cannot say why. The correct handling is exclusion with a recorded reason, not a coin flip and not a default to the closed model's output. An excluded example costs nothing. An admitted wrong example teaches the wrong thing and is very hard to find later.

Excluded and unresolved counts should be reported alongside admitted counts. A dataset version that says 412 admitted, 96 rejected, 61 unresolved tells a reader something about the data's quality. A dataset that says 412 examples does not. Part 12 covers how to carry that accounting in the lineage of each example.

The evaluation set has to be decided before anyone reviews or trains, and it has to be kept out of everything that follows: representative selection, propagation, target editing, and model tuning. Split at the level of the entity or conversation family, not the individual run, so that two runs about the same customer or the same document cannot land on both sides. Part 19 covers leakage in detail.

Protected cases are not filtered for reviewer agreement or for whether the closed model got them right. If the closed model was wrong on a protected case, that case stays, because the gate is measuring the candidate against the task, not against the teacher. Reviewing a protected case burns it; it moves to a development split and a replacement is drawn.

The export is immutable and versioned. It carries every admitted example with its lineage, the split assignment, the instruction version the decisions were bound to, the accounting of admitted, rejected, and unresolved candidates, and the counts of distinct sources and behavior clusters in each split. A later dataset version adds to it; it never edits an example in place.

The counts of distinct sources matter more than the total. Two hundred examples drawn from ten templated scenarios are ten examples, and a training run on them produces a model that has memorized ten things, which part 13 and part 15 describe. The export should make that visible so the diversity floor is checked before training, not diagnosed after it.

This pipeline is the shape of Converra's fine-tuning workflow. Its judges propose step-level findings over the evidence each run carries. A reviews queue presents behavior clusters with a representative case, the evidence, and a proposed target for a customer reviewer to decide and edit. Datasets record per-example admission with direct versus propagated lineage, exclude unresolved cases, keep protected splits out of curation, and freeze into versioned exports with an accounting of admitted, rejected, and unresolved candidates.

Training runs on the customer's provider or infrastructure; Converra records the training job and its gate result against the dataset version and holds the production switch behind an approval. Converra does not use customer data to train any model without a written, tenant-exclusive election. No open-weight production verdict for a customer agent has been published yet, and the quality of any given team's reviewed data is unobserved until its candidate is measured on a protected gate and on real traffic.