How Many Reviewed Examples Does Customer-Specific Fine-Tuning Need?

As of 2026-09-23, OpenAI's supervised fine-tuning guide states a minimum of 10 examples and recommends starting with about 50 well-crafted demonstrations before evaluating. Fireworks' supervised fine-tuning documentation accepts a dataset of as few as 3 examples. These are the numbers below which the job will not run or the provider will not vouch for a result. They are not the numbers at which a customer agent's behavior will hold on traffic the examples did not cover.

The gap between the two comes from what a customer-specific fine-tune is asked to do. A general instruction-following tune benefits from every example because each one is a new task. A customer-specific tune is asked to hold one agent's behavior across the variety of that agent's traffic, and its examples are drawn from a narrow distribution where many rows say the same thing. Fifty rows that are five requests rephrased ten ways is not fifty examples.

A family is the unit of learning signal: a group of rows that share the same underlying situation and would, if you changed one, force the others to change too. For a supplier-risk agent the family is the supplier and its ownership graph. For a support agent it is the customer account or the order. For a routing agent it is the scenario template that generated the conversation. Rows within a family are near-duplicates from the model's point of view even when the wording differs.

When you count families instead of rows, most datasets shrink by an order of magnitude, and that shrunken count is the honest one. A dataset that reports 240 admitted rows should also report how many distinct families they came from and how many distinct templates those families were instantiated from. If either number is small, the model is going to learn the templates rather than the task, and it will show the collapse pattern described in part 13.

Rather than one dataset-wide number, write a floor for each cell you need the model to learn. A cell is a behavior crossed with an evidence condition: answering an order-status request with full evidence, answering it with a missing field, abstaining when the order does not exist. Each cell needs enough distinct families that the model sees the behavior in several situations, and no single template may supply more than a bounded share of them.

A practical starting floor for a bounded agent workload is on the order of a dozen distinct families per cell that matters, a few dozen templates across the training set, and a hard cap so no template supplies more than around 15 percent of training families. These are design defaults to be adjusted by the gate, not universal constants. A cell you cannot fill to its floor is a cell the candidate should not be certified on, and the parity claim in part 21 should exclude it explicitly.

The failure that makes the row count misleading is one-class collapse: the model learns to produce the majority output regardless of input. It happens when the training inputs are effectively few, because the model can fit them by memorizing surface features and defaulting on everything else. The loss curve looks fine during training. The holdout, if it comes from richer families, reveals a model that gives the same answer to every case.

Collapse is not fixed by adding rows from the same templates, and it is not fixed by a larger base model, because the cause is the data distribution rather than capacity. It is fixed by adding families and templates, and by seeding the rare behaviors so they are present in enough distinct situations to be learned rather than treated as noise. Part 13 covers the epoch side of the same mechanism.

For most agents the costly outputs are rare: the sanctioned supplier, the account with a fraud flag, the request that must escalate. Natural traffic under-represents them, and a model trained on natural proportions learns that the safe majority answer is nearly always right. Seed adverse families deliberately, drawn from real reviewed cases where they exist and from carefully built synthetic families where they do not, and record them as their own count in the dataset manifest.

Keep a share of adverse families out of training entirely, for the protected holdout. The gate's recall check has to run on adverse cases the model never saw, or it measures memorization. A dataset with 60 adverse families might send 35 to training and hold 25 for the gate. If 25 seems small, remember the bound: zero misses on 25 still leaves a true miss rate that could be around 11 percent. More holdout families tighten that bound; fewer training families do not.

Suppose a supplier-risk agent that reads a document, calls three tools, and writes a tiered report. The team has 2,400 production runs. Grouped by supplier ownership graph, those runs come from 310 families across 14 scenario templates, of which 48 families are adverse. The team assigns splits before reviewing anything: 190 families to training, 20 to development for sizing the gate, and 100 to the protected holdout including 25 adverse and all families from 3 templates held out entirely.

After review, 162 training families are admitted with verified targets and 28 are rejected or unresolved. The largest template supplies 27 admitted families, under the 15 percent cap against the original 190 but over it against 162, so the team trims it to 24 before export. The export is 159 rows, one reviewed target per family, plus a handful of verified variations for the partial and no-evidence cells. The training run costs a few dollars at the per-token training price listed on the Fireworks pricing page on 2026-09-23. Whether 162 families were enough is now a question the frozen gate answers.

A gate failure with the collapse signature says add families, not rows. A gate failure concentrated in one cell says that cell was under its floor, and the fix is more families in that cell. A gate failure on the held-out templates says the training templates did not span the task, and the fix is new templates, not more instances of the existing ones. A tie with the base model, covered in part 15, says the training changed nothing the gate could see, which usually means the targets did not differ from what the base already did.

None of these failures is answered by retraining with more epochs on the same data. The count you increase is always the one the gate identified as thin, and every added family goes through the same review and admission as the originals, with lineage preserved as described in part 12.

It is tempting to fold the development split into training once the gate design is settled. Do not. The development families were used to size the gate, estimate variance, and tune review templates; they have influenced decisions the protected holdout is supposed to be independent of. Adding them to training after the fact means the candidate has, indirectly, seen data that shaped its own acceptance test.

If you need more training families, generate or review new ones and assign them to training at admission time. If you need a larger holdout, generate new protected families before the freeze. The rule is that family assignment happens once, before any candidate exists, and is never revised to make a number look better.

Converra's fine-tuning workflow curates reviewed production runs into versioned datasets with protected splits and a per-example admission decision, so the counts that matter, families admitted, rejected, and unresolved, are recorded per dataset version rather than reconstructed afterward. It records the training job and its gate result against that version. Training itself runs on your provider or infrastructure; Converra does not run it. Its simulation testing uses synthetic personas derived from real production patterns, which is one way to build additional families for a cell that is under its floor.

Converra does not use customer data to train any model without a written, tenant-exclusive election, and no open-weight production verdict for a customer agent has been published yet. How many families your agent needs is unobserved until a frozen gate on unseen families says the count was enough.