What Does It Mean When a Fine-Tune Ties Its Untuned Base Model?

Two models tie when the evaluation assigns them the same scores within noise. That can happen because they behave the same, or because the evaluation cannot tell their behaviors apart. A gate that scores only schema validity will tie any two models that both format well. A gate built from cases the base already handles will tie a candidate that learned a rare behavior the cases never demand.

So before reading a tie as a training failure, ask what the evaluation is capable of seeing. If the cases, the scorer, and the pairing were designed around the specific behavior the fine-tune was meant to add, a tie is informative. If they were borrowed from a general benchmark or from the incumbent's regression suite, a tie mostly says the suite was not built for this question.

Ties come from five places, and the cheap checks come first. One, the adapter is not loaded at serving time and you evaluated the base twice. Two, the evaluation does not exercise the trained behavior. Three, the training examples look like what the base already does, so there was nothing to learn. Four, the training signal was too weak to move the weights: too few steps, a learning rate near zero, or a rank too small for the change. Five, the base is genuinely good enough on this workload.

Checking in this order matters because the later causes are expensive to investigate and the earlier ones are embarrassing to discover after a week of data work. The table gives the signature and the test for each.

The cheapest cause is also the most common in a first attempt. Managed fine-tuning platforms return a model identifier for the adapter, and serving that identifier requires a deployment that has the adapter attached. It is easy to deploy the base, point the evaluation at it, and never notice, because everything works and the scores look plausible.

Prove the adapter is loaded before believing any result. At temperature zero, send a few training inputs to both the base and the candidate endpoint and compare completions; a served adapter should reproduce its training targets closely, and the base should not. Then record the deployment receipt: model id, adapter id, deployment id, and a hash of the request configuration. Every gate result should carry that receipt, so a tie can be checked against the serving path months later.

Count the cases in the evaluation that a model could only get right by doing the thing the fine-tune was supposed to teach. If the fine-tune taught abstention on missing evidence and only three of a hundred cases have missing evidence, the maximum measurable effect is three points, well inside noise. If it taught a domain-specific finding and the cases were written from a general template, none of them demand it.

The remedy is not to rewrite the gate after seeing the result, which would burn the holdout. It is to write down, before freezing the evaluation set, what behavior the fine-tune targets and how many cases exercise it. A development set, disjoint from the protected holdout, is the place to confirm the evaluation can see the behavior at all. Part 17 covers what the frozen set should contain.

Run the untuned base over the training inputs and compare its outputs to the training targets. If the base already produces the target decision and something close to the target format on most examples, the fine-tune had little to learn, and a tie is the expected outcome. This happens when training data is collected by taking the closed model's outputs on easy cases, which are exactly the cases a competent open base also handles.

The informative examples are the ones where the base is wrong and the target is verified right. Measure how many of those the training set contains. A set with two hundred rows and fifteen base-wrong rows is teaching from fifteen examples. Part 11 takes up the balance between common requests and hard cases, and this measurement is the fastest way to see that balance in an existing dataset.

Weak training signal is real but it is the last thing to blame, because the fix, more steps or a larger adapter rank, is also what turns a narrow dataset into an overfit one. Look at the loss curve first. A curve that never moved suggests a learning rate or step-count problem. A curve that dropped fast and flattened suggests the format converged and there was nothing else to learn, which is a data problem wearing a training costume.

Provider defaults are conservative on purpose. Fireworks documents a default of one epoch and recommends raising it by one or two only if the model does not follow the training adherence you expect, as listed in its supervised fine-tuning documentation on 2026-09-23. Raising epochs to chase a tie without first checking the four earlier causes is how a tie becomes a collapse, which part 13 explains.

If the candidate and the untuned base both pass the frozen gate against the closed incumbent, the fine-tune was unnecessary. That is a good outcome. The base costs nothing to train, has no adapter to version, carries no training-data rights questions, and is not tied to a dataset that will need to be rebuilt when the task changes. The transfer succeeds with less machinery.

Write that conclusion down as a decision with evidence: the gate result for all three arms, the deployment receipts, the dataset version that was tried, and the reason fine-tuning was set aside. A team that keeps this record can return to fine-tuning when the base stops being good enough, with the dataset work already done. A team that quietly ships the base and forgets why will re-run the same experiment a year later.

Most gates compare the candidate to the incumbent and stop. That design cannot distinguish a fine-tune that added something from a base that was already sufficient, because both look like a candidate that matched the incumbent. Adding the base as a third arm costs one more set of runs and turns every result into three questions with separate answers: did the base match the incumbent, did the candidate match the incumbent, and did the candidate beat the base.

The three-arm layout also catches the serving failure automatically. A candidate that is byte-identical to the base across the holdout is a deployment bug, and it shows up as a suspiciously exact tie rather than a plausible near-tie. Run the base, keep its outputs, and compare per case.

Converra's model benchmarks run each candidate model and the incumbent on the same scenarios generated from the agent's instruction, three runs per scenario with median scoring, and report quality, cost, and latency per model. A winning switch ships as a reviewed pull request with the comparison table. For a fine-tuned candidate, its workflow records the training job and its gate result against a specific dataset version and keeps the production switch behind an explicit approval; the model production test then decides on non-inferiority plus strictly lower cost on live traffic.

Converra does not run training, and it does not use customer data to train any model without a written, tenant-exclusive election. No open-weight production verdict for a customer agent has been published yet, so a tie or a win on a gate says nothing about a particular agent's traffic until the verdict is measured there as parity verified, regressed, confounded, or insufficient data.