Can a Fine-Tuned Open-Weight Model Replace the Closed Model Powering a Production AI Agent?

Teams ask this question as if it were about model size. It is about four things that have nothing to do with parameter counts. First, the workload: a narrow, repetitive, well-evidenced task transfers; open-ended judgment across a long tail does not. Second, the training data: reviewed examples of the behavior you actually want, not a dump of whatever the closed model produced. Third, the gate: a frozen set of cases the candidate has to match the incumbent on, scored the same way for both. Fourth, production evidence: a measured result on real traffic after the switch, not a projection from the gate.

When all four hold, a fine-tuned 8B to 30B open-weight model can carry a production agent at a fraction of the per-token cost. When any of them is missing, the transfer is a bet, and the usual outcome is a model that looks right in a demo and fails on the requests the demo did not include.

A replacement keeps everything about the agent fixed except the model. Same instruction, same tools, same retrieval, same output schema, same escalation rules. Only the weights change. That framing matters because it turns a vague question into a testable one: does the agent, with the new model in the same seat, behave at least as well as it did with the old one on the requests it actually receives?

Notice the phrase at least as well. Replacement is a non-inferiority claim, not a superiority claim. You are not trying to beat the closed model. You are trying to show the open model is not worse than the closed model by more than a margin you agreed on in advance, on a set of cases you froze before you looked at any candidate output. Part 21 of this series works through what that margin should cover.

An untuned open-weight model in the 8B to 30B range is usually a competent generalist with weaker instruction following, less consistent formatting, thinner domain knowledge, and shallower multi-step reasoning than a frontier closed model. It is also cheaper by an order of magnitude per token, runs where you choose, and does not change under you when a provider ships a new version.

Fine-tuning narrows the gap on the dimensions that live in the model's behavior, and leaves the dimensions that live in its knowledge and capacity roughly where they were. The table below is the honest starting map for a typical transfer. Every row is a dimension you should measure on your own workload rather than assume.

Supervised fine-tuning teaches a model to produce the kind of output your examples contain, under the kind of input your examples contain. It is very good at format, tone, conventions, the order of operations in a multi-step task, which tool to call in which situation, and when to say the evidence is insufficient. Those are behaviors, and behaviors transfer.

It is bad at adding knowledge the base model does not have, at extending reasoning to problem shapes the examples never showed, and at generalizing from a handful of templated cases to a genuinely varied production stream. If your agent's value comes from the closed model knowing things, fine-tuning a smaller model will not replace that. If its value comes from the closed model behaving a certain way over evidence the agent already has in hand, it usually can. Part 3 goes through this distinction request type by request type.

Most failed transfers fail the same three ways, and each is visible before the model goes anywhere near traffic. The first is collapse: the model learns to give the majority answer on everything, because the training data had too little real variety. A dataset of two hundred rows that come from ten templates is ten examples, not two hundred. The candidate scores well on cases that resemble the templates and answers every hard case with the safe default.

The second is format without judgment: every output parses, every field is present, and the content is worse than the base model's. The model learned the shape of a good answer faster than the substance of one. The third is a tie: the candidate performs no differently from its untuned base, which means the training run changed nothing that the evaluation can see. Parts 13 through 16 take each of these apart, because a team that can name the failure it is looking at knows which fix to try next.

A gate is a fixed set of cases, frozen before any candidate is evaluated, that the incumbent and the candidate both run under identical conditions. The same inputs, the same pre-fetched tool results, the same scoring. Each case is scored for both models and the comparison is paired, case by case, so the decision rests on where the two models differ rather than on two averages that hide it.

A serviceable gate has five parts. A non-inferiority rule with a pre-registered margin. A recall check on seeded hard cases the model must not miss. Mode checks that catch collapse: how often the candidate gives the default answer, how many distinct outputs it produces, how long its answers run relative to the incumbent's. A competence check on the incumbent itself, because parity with a model that is failing the task proves nothing. And a symmetric rule for missing data, so an infrastructure failure on one side does not become a quality win for the other. Part 17 covers what the evaluation set needs to contain for that gate to mean anything.

Full replacement is the strongest version of the claim, and it is not the only one. Many agents have a large share of requests that a fine-tuned open model handles as well as the closed model, and a smaller share it cannot. Routing sends the first group to the open model and the second to the closed model, and the transfer succeeds on the share it can prove rather than on the whole.

Routing also changes what you measure. Instead of asking whether the open model matches the closed model everywhere, you ask which requests it matches on, how reliably a router can identify them before the answer is produced, and what the blended cost and quality look like once escalations are counted. Parts 22 through 25 treat routing as its own design problem, because a routing policy that only works on the customer it was built for is not a product.

The training run is the cheap part. Managed supervised fine-tuning on a model up to about 16B parameters is priced per million training tokens at a rate that makes a customer-sized dataset cost a few dollars, and a hosted GPU endpoint for evaluation costs a few more per hour. Neither number says anything about what serving costs at production volume with production latency requirements.

The real comparison is the closed model's per-token bill against the open model's serving bill including idle capacity, autoscaling headroom, the closed-model fallback for escalations and outages, and the engineering time to operate it. At low volume, the open model can cost more, because a dedicated endpoint sits idle. At high volume it can cost dramatically less. Parts 26 through 29 work through where the break-even sits and how to keep endpoint startup from eating an experiment's budget.

Passing the gate earns the candidate a place in production, not a conclusion. The proof is a measured result on live traffic after the switch, on the same outcome the gate was measuring, with the deployment boundary preserved so post-switch runs can be compared to pre-switch runs without mixing in other changes. The result should come back in one of a small number of states: parity verified, regressed, confounded by a concurrent change, or insufficient data to say.

That verdict is also what makes the economics honest. Savings are a per-customer measurement of what the same traffic cost before and after, at the quality the customer agreed to, not a theoretical difference between two token prices. Part 32 and part 47 cover the production verdict and the savings proof in detail.

Converra benchmarks candidate models against scenarios built from your agent's own instruction, measures quality, cost, and latency for each model on the same cases, and ships a winning switch as a reviewed pull request. For a fine-tuned candidate, its fine-tuning workflow curates reviewed production runs into versioned datasets with protected splits, records the training job and its gate result against that dataset version, holds the production switch behind an explicit approval, and reports a parity verdict on real traffic once the switch is live.

Two boundaries are worth stating plainly. Converra does not use customer data to train any model without a written, tenant-exclusive election; its default terms forbid it. And no open-weight production verdict has been published yet for a customer agent. The workflow exists; the outcome on your traffic is unobserved until it is measured there. The rest of this series is written so that a team can run this transfer well with or without Converra in the loop.