Why Can More Training Epochs Make an Open Model Worse?

One epoch is one pass over every training example. On a dataset of a few hundred rows the whole set can fit in a handful of optimizer steps, so each epoch is a small number of large updates and each additional epoch repeats the same updates with a lower loss. The model is not seeing new situations; it is seeing the same situations with the residual error driven down further each time.

For a general-purpose instruction tune over hundreds of thousands of diverse examples, extra epochs mostly refine. For a customer-specific tune over a few hundred rows drawn from a few dozen families, extra epochs mostly memorize. Fireworks' supervised fine-tuning documentation, as of 2026-09-23, sets the default epoch count to 1 and suggests raising it by 1 or 2 only if the model is not adhering to the training format. That default is a good prior for this kind of data.

The trap is that the metric you watch during training is the one that cannot detect the problem. Cross-entropy loss on the training set measures how well the model reproduces the training targets. It falls with every epoch, and on a small dataset it falls fast. A validation loss on rows drawn from the same families falls too, because those rows share the surface features the model is memorizing. Neither curve shows what happens on a family the model has never seen.

What happens is that the model learns the cheapest features that separate the training examples and stops there. If the training families are templated, with a handful of shapes and a setup token that varies, the model learns the shapes and the token. On a holdout from richer families, none of the learned features fire, and the model falls back to the output it produced most often in training. That is the collapse: one answer for everything, with high confidence.

The collapsed model produces nearly identical outputs across cases that should differ, and its confidence is uniform, because it is not distinguishing inputs. Rare behaviors vanish first: abstention, escalation, the adverse tier. Where the training data had a majority label, the model produces that label on every case, which gives a recall of zero on the minority behaviors and a suspiciously exact accuracy equal to the majority share of the holdout.

The format usually survives collapse, and that is the misleading part. Every output parses, every field is present, and a reviewer skimming a few examples sees well-formed answers. Part 14 covers the related failure where format is learned and substance is not. Here the tell is the distribution: count distinct outputs across the holdout, measure the share of cases that received the majority answer, and compare both to the incumbent. A candidate whose distinct-output ratio is a fraction of the incumbent's has collapsed regardless of its loss curve.

When a candidate collapses, the instinct is to reach for a larger base model on the theory that capacity was the limit. It usually is not. Collapse driven by thin data reproduces across model sizes, because a larger model memorizes a small dataset at least as easily as a smaller one. The extra capacity has nothing new to learn from, so it learns the same shapes with more parameters. Spending on a larger base attacks the one cause the evidence has already ruled out.

The same logic applies to LoRA rank, learning rate schedules, and batch size. These change how efficiently the model fits the data it has. None of them adds variety to that data. When the holdout shows the collapse signature, the first question is how many distinct families and templates the training set contained, and the answer is usually the whole explanation.

The fix order is fixed. First, count families and templates per behavior and evidence condition, as in part 10, and fill the cells that are thin. Second, check the ratio of common to rare families and the per-template caps, as in part 11. Third, verify the targets actually differ from what the base model already does, because a dataset of targets the base would have produced anyway teaches nothing, and more epochs on it teach nothing harder; part 15 covers that case. Only then, if the model is still not adhering to the format or the behavior on a holdout from unseen families, consider raising epochs by one.

Every step of this order is a data change that requires a new dataset version and a new candidate identity. The candidate that collapsed is spent. Retraining it with more epochs under the same identity, or tuning epochs by watching the protected holdout, turns the holdout into a development set and invalidates the gate.

One epoch is not always enough, and the failure on the other side is distinguishable. An under-trained model on a small dataset often half-learns the output format: it produces the right shape with a missing key, an unclosed structure, or a target field it never fills. Training loss at the end of the run is still high and visibly mid-descent. The holdout shows format failures spread across cases rather than one answer repeated.

This is the case where the provider guidance to add an epoch or two applies, and the loss curve is informative here in a way it is not for collapse. If loss is still falling steeply at the end of the run and outputs are structurally broken, another pass is reasonable. If loss has flattened near zero and outputs are structurally perfect but identical, another pass will make the collapse worse. The two failures need opposite responses, so identify which one you have before touching the setting.

Suppose a team fine-tunes an 8B open model to triage support conversations into three tiers, from 240 rows drawn from 30 families across 6 templates. At 1 epoch the outputs are malformed on a third of a 60-case holdout and the training loss ends high. The team raises the setting to 10 epochs. Now every output parses, training loss is near zero, and the holdout accuracy is 62 percent, which is exactly the share of the holdout in the majority tier. Every case got the majority tier.

The loss curve for the second run looked better than the first at every point. The holdout is what showed that the first candidate was under-trained and the second had collapsed. The team's next candidate is not an 8-epoch compromise; it is a new dataset version with families from 20 templates and a seeded minority tier, trained at 2 epochs, gated on a holdout that includes templates the training never saw.

Converra's benchmarks run every candidate model against the same scenarios three times with median scoring and report quality, cost, and latency side by side with the incumbent, which is the comparison that exposes a collapsed candidate's narrow output distribution rather than its loss curve. For a fine-tuned candidate, its fine-tuning workflow records each training job and its gate result against a specific dataset version, so a second candidate trained on more epochs is a new job with its own record, not an overwrite. Training itself runs on your provider or infrastructure.

Converra does not use customer data to train any model without a written, tenant-exclusive election, and no open-weight production verdict for a customer agent has been published yet. Whether a candidate's behavior on unseen families holds at parity is unobserved until the protected gate and then real traffic measure it.