Why Isn't a Cheap Training Run Proof of Cheap Production Inference?

Managed LoRA supervised fine-tuning on models up to 16B parameters is listed at $0.50 per million training tokens on the Fireworks pricing page as of 2026-09-23. A dataset of 2,000 examples averaging 1,500 tokens is 3 million tokens per epoch, so three epochs cost about $4.50. That number is real, and it is almost irrelevant to what the model costs to run.

Serving bills in a different unit. A dedicated replica is priced per hour, listed at $8.00 for an on-demand H100 on the same page and date, and it serves however many requests it can within the latency target. The cost per request is the hourly rate divided by requests actually served in that hour. Nothing in the training run determines that quotient.

A small training bill confirms three things. The dataset was small in tokens. The adapter was trained with a parameter-efficient method rather than a full fine-tune. The provider's training path is inexpensive at this scale. Those are useful facts for the data and governance parts of the transfer, and they have no bearing on serving.

It can also mislead. Teams that see a cheap training run tend to assume the model is cheap in every respect and skip the serving measurement. The base model's size, not the adapter's, determines memory and throughput. A LoRA adapter on a 30B base costs about the same to train per token as one on an 8B base, and several times more to serve.

Throughput per GPU is how many requests a replica completes per second at the latency target, and it depends on base model size, quantization, serving engine, batch scheduling, and the token mix of real requests. Latency at concurrency is how response time degrades as simultaneous requests rise; a model that answers one request in 400 milliseconds may take three seconds under a queue of fifty. Memory for long contexts determines how many concurrent long conversations fit before the replica rejects or evicts requests. Fallback share is the fraction of traffic the open model cannot serve and hands back to the closed model.

Each of these moves the cost per request, and none appears anywhere in a training receipt. The table below pairs each property with the measurement that reveals it and the cost consequence of guessing wrong.

A single-request benchmark reports the best case. Production is many requests at once, and a serving engine schedules them into batches whose latency rises with size. Suppose a replica answers one request in 500 milliseconds and the agent's target is 2 seconds. A ramp test might show the replica sustaining 30 concurrent requests inside the target and breaking it at 45. Throughput inside the target, not peak throughput, is what sets the replica count.

Memory compounds the problem. Each concurrent request holds a key-value cache proportional to its context length. A support agent with 6,000-token conversations fits far fewer concurrent requests on one GPU than a classifier with 400-token inputs, on the same base model. Run the load test with your own context lengths, not the serving engine's demo settings.

Replay a representative sample of production requests, with real token counts and context lengths, against one replica of the exact serving configuration you intend to run: base model, adapter, quantization, engine version, and hardware. Ramp concurrency from one to past the expected peak. Record throughput, latency percentiles, error and timeout counts, and output length at each level.

Read off the highest concurrency at which the latency percentile you care about stays inside the target. Requests per hour at that level, divided into the hourly rate, is the variable cost per request for one replica. Peak concurrency in production divided by that level, rounded up, is the replica count, and replica count times hourly rate times hours up is the fixed cost. Those two numbers feed the break-even in part 27.

Suppose two candidates train on the same dataset: an adapter on an 8B base and an adapter on a 30B base. Both training runs cost about $5. The 30B candidate scores slightly higher on the gate. The load test shows the 8B replica sustaining roughly three times the requests per hour of the 30B replica inside the latency target. At equal traffic, the 30B deployment needs three times the replicas and costs about three times as much per request.

Whether the gate margin justifies that is a real decision, but it is only visible once serving is measured. From the training invoices alone, the two candidates look identical. Part 37 covers choosing a base model size against the actual workload; this part exists so the choice is made with serving cost in view.

Quantizing weights to fewer bits raises throughput and lowers memory, and can change outputs. A serving engine's batching and caching strategy can double throughput on the same hardware. Both are legitimate ways to lower the serving bill, and both have to be present in the configuration the gate evaluated and the load test measured. A cost quoted on a quantized engine build and a quality quoted on an unquantized one describe two different models.

Part 44 of this series covers what changes under quantization. For cost purposes the rule is simple: the configuration you load-test is the configuration you gate, and the configuration you gate is the configuration you deploy.

Present the training cost as what it is: a small, one-time line that will recur at each retraining. Present the serving cost as a measured cost per request at the latency target, with the replica count and utilization assumptions stated, and with the fallback share applied. Present the difference between that and the closed-model bill on the same traffic as the projected saving, labelled as a projection until production confirms it.

A customer who hears that training cost $4.50 and concludes that inference will be nearly free has been given a number without its unit. The number that matters is the one from the load test, and it is the one that belongs on the slide.

Converra measures cost and latency from the real calls when it benchmarks candidate models against scenarios built from the agent's own instruction, running every model on every scenario three times and reporting the median, so the per-request numbers in a comparison come from observed inference rather than from a training receipt. When a fine-tuned candidate is promoted, the model-swap verdict on real traffic reports the cost delta per arm together with parity verified, regressed, confounded, or insufficient data.

The load test at production concurrency, the replica sizing, and the choice of serving engine and quantization are yours; Converra does not run the training and does not size your GPUs. No open-weight production verdict for a customer agent has been published yet, so a serving cost is a measured estimate until the cost delta is confirmed on your traffic.