Which Examples Matter More: Common Requests or Difficult Edge Cases?

A common request, answered well, teaches the model the shape of a normal turn: the tone, the structure, which tool to call first, how to cite what it found, how to close. Because these cases dominate traffic, they also set the model's prior about what a request usually wants. That prior is what makes the model fast and consistent on the bulk of traffic, and it is not optional.

A difficult edge case teaches the model the conditions under which the normal answer is wrong: the sanctioned entity, the account under review, the request that looks routine but must escalate, the conflicting evidence. These are the outputs that cost money when they are missed. They are rare in traffic, so a model trained on natural proportions sees them as noise around the mode and learns to ignore them. Training on them deliberately is what gives the model a boundary as well as a default.

Suppose 2 percent of an agent's runs involve a condition where the correct output is the rare one. Sampled at that rate, a 300-family training set contains 6 adverse families. Six examples of a behavior, spread across whatever templates happened to occur, are not enough for the model to learn the trigger. The model fits the other 294 and treats the 6 as outliers. On the gate it answers the adverse cases with the majority output, and on traffic it does the same.

Over-sampling the rare condition is the fix, but it has its own failure mode. Push adverse families to half the dataset and the model learns that the rare answer is common, which produces false positives on routine traffic. The ratio has to be high enough that each adverse behavior is learned and low enough that it is learned as a conditional, not a default. A working range for many agent workloads is adverse families making up somewhere between a fifth and a third of training, adjusted by the gate's precision and recall results.

The gate measures two things about rare behaviors: whether the model produces them when it should, and whether it withholds them when it should not. The first is recall, and it can only be measured on adverse cases the model did not train on. If every adverse family is in training, a perfect recall score is a memorization score. Split adverse families before curation, send a majority to training and keep the rest protected, and never move a family across that line.

The protected adverse set should be large enough that the recall bound means something. Zero misses on 10 held-out cases still permits a true miss rate near 26 percent under the rule of three; zero on 25 tightens that to around 11 percent; zero on 50 to around 6 percent. Decide the bound you can live with before you know how many adverse families you have, then generate or review enough to reach it.

For every adverse condition there is a family that resembles it and is not it: the supplier with a similar name to a sanctioned one, the refund request that looks like fraud and is a legitimate return, the urgent-sounding message that is routine. These near-misses teach the boundary from the other side. Without them the model learns a loose trigger, fires on surface features, and the gate shows a precision problem on routine traffic.

Near-misses should be reviewed with the same care as adverse cases, because the reviewer has to confirm that the correct output really is the routine one. They are also the families where a closed model's original output is most often wrong in the cautious direction, refusing or escalating something it should have answered, so they need targets written from the evidence rather than copied from the teacher.

Even with a good adverse ratio, a dataset can be dominated by one scenario template if that template happened to generate many families. A support agent might have 80 families about delivery delays and 4 about address changes. The model learns delivery delays well and treats address changes as an odd variant. The cap is a rule that no single template supplies more than a bounded share of training families, on the order of 15 percent, and no template supplies more than around 10 percent of the protected holdout.

Caps force the curation step to either trim the dominant template or add families to the thin ones, and the second is almost always the better choice. Trimming loses real variety; adding fills a cell that was under its floor. Record the per-template counts in the dataset manifest so the cap is checkable at export time and the build fails closed when it is violated, rather than someone noticing after the gate.

The gate result tells you which kind of example was short. Low recall on held-out adverse families with good precision on common families: the adverse side is thin, add adverse families. Good recall with false positives on common or near-miss families: the boundary is loose, add near-misses. Poor performance on held-out templates with good performance on seen ones: the template spread is narrow, add templates. Collapse to one output across all cases: the whole dataset is effectively small, as in part 13.

Each diagnosis names the count to increase. None of them is answered by more rows from the templates already present, and none is answered by more epochs. The gate is the instrument that tells you what the data was missing; treat its per-slice results as a curation plan, not a pass-fail bit.

Suppose a refunds agent. Its traffic is 90 percent routine refund requests, 6 percent requests that require a policy exception, 3 percent requests that must be escalated for suspected abuse, and 1 percent with no usable order evidence. Sampled naturally, a 250-family training set would hold about 15 exception families, 8 escalation families, and 2 no-evidence families.

The team instead builds 250 training families as 150 routine, 40 exception, 30 escalation, 15 no-evidence, and 15 near-miss families that resemble abuse but are legitimate. No template exceeds 30 families. It holds out 100 families including 25 escalation cases and 10 near-misses, plus every family from 2 of its 12 templates. On the gate the candidate misses 0 of 25 escalations and wrongly escalates 3 of 10 near-misses, which sends the team back to add near-miss families rather than to celebrate the recall number.

Converra's regression suite is built from scenarios the current agent handles well, which is the common-case side of this problem, and its fleet view aggregates failure patterns by business impact, which is how the rare, costly conditions are found and prioritized in production. For a fine-tuned candidate, its fine-tuning workflow admits reviewed examples into versioned datasets with protected splits and per-example decisions, records the training job and gate result against a dataset version, and holds the production switch behind an explicit approval. Training runs on your provider or infrastructure.

Converra does not use customer data to train any model without a written, tenant-exclusive election, and no open-weight production verdict for a customer agent has been published yet. Whether your ratio of common to rare families was right is unobserved until the gate's per-slice results and then real traffic say so.