How Much Does Open-Weight Inference Save After Hosting, Idle Capacity, and Fallbacks?

A closed model bills per token. A dedicated open endpoint bills per hour of capacity, whether or not a request arrives. Comparing a per-token rate to a per-token rate hides the conversion between the two, and the conversion is where the money goes. An hour of an H100 at $8.00, the on-demand rate listed on the Fireworks pricing page on 2026-09-23, costs the same at 3 a.m. with zero requests as it does at peak.

The correct comparison starts from a period of real traffic, prices it on the closed model, prices the same period on the open model including every fixed and variable cost below, and subtracts. Anything that is not in the second number is not a saving; it is a cost you have not counted yet.

The open-model bill has three layers. The capacity layer is GPU-hours for the replicas that must be up to meet the latency target at peak concurrency, including headroom the autoscaler holds so a traffic spike does not queue. The traffic layer is the share of requests that still go to the closed model: routed escalations, fallbacks during incidents, retries after timeouts, and any request the open model refuses. The operations layer is observability, load testing, adapter versioning, security patching, and the engineering hours those consume.

The table below lists what belongs in the bill and where the number comes from. If a line has no measured source yet, put a placeholder in and mark it unobserved rather than leaving it out, because a missing line reads as zero when someone adds up the column.

Capacity cost is fixed per hour. Traffic is not. A replica that handles its full throughput all day spreads its hourly cost over many requests and comes out far cheaper per request than the closed model. The same replica handling a fraction of that throughput spreads the same cost over fewer requests, and the per-request cost climbs until it crosses the closed model's rate.

Utilization is also uneven across the day. An agent that serves business hours in one time zone leaves its replicas idle for most of the night. Unless the endpoint scales to zero and the cold start is acceptable, those hours are pure cost. Measure the hourly traffic profile before estimating savings, because the average utilization number hides the shape that determines the bill.

Suppose an agent handles requests averaging 1,500 input tokens and 300 output tokens. Suppose the closed model bills a blended $4.00 per million tokens on that mix, so each request costs about $0.0072. Suppose one open-model replica costs $8.00 per hour and sustains 4 requests per second at the latency target, which is 14,400 requests per hour at full load. These numbers are illustrative; substitute your own.

At 60 percent average utilization, the replica serves about 8,640 requests per hour for $8.00, or about $0.0009 per request. Against $0.0072 on the closed model, that is a saving of roughly 87 percent before fallbacks and operations. Now suppose the same agent receives 400 requests per hour, which is under 3 percent utilization of one replica that has to stay up for latency. The replica still costs $8.00, so each request costs $0.02, nearly three times the closed model's rate. Same model, same adapter, opposite sign.

Almost every transfer keeps the closed model reachable. A router sends hard cases to it, a timeout falls back to it, an invalid output retries on it, and an incident fails everything over to it. Each of those requests is billed at the closed rate and often carries the open-model attempt's cost as well, because the open replica already spent capacity producing the output that got discarded.

Count fallback as a share of traffic multiplied by the closed per-token cost, plus the wasted open attempt. Suppose 15 percent of requests escalate. The closed bill does not fall by 100 percent; it falls by at most 85 percent, and the open bill grows by the capacity used on those 15 percent before they escalated. A router that escalates too eagerly can erase the saving entirely while the token-price chart still looks like a win.

An autoscaler that adds a replica when load rises has to start it before the queue grows past the latency budget. Loading weights and warming caches on a fresh replica takes long enough that most teams hold a replica of headroom permanently. That replica is a fixed cost that exists only so spikes are absorbed, and it belongs in the bill as headroom, not as serving.

Scale-to-zero removes idle cost at the price of a cold start on the first request after quiet. Whether that is acceptable depends on the latency the agent promises. A background document-processing agent can absorb a cold start; a live chat agent usually cannot. Decide the policy from the latency requirement, then price the capacity the policy implies.

Running an open endpoint means someone owns the deployment, the adapter registry, the serving-engine version, GPU driver updates, security patches, load tests after each change, and the on-call rotation when it degrades. Those hours have a fully loaded cost. For a small team the engineering line can exceed the GPU line at moderate traffic.

Observability also costs money that the closed model's bill already included. Sampling outputs, running judge calls for drift detection, storing traces, and re-running the gate after each adapter version all show up as inference and storage spend. Part 45 of this series covers what that drift-detection sampling has to look like; here it only needs to be in the bill.

A hosted open-weight endpoint billed per token removes the idle-capacity problem and most of the operations line. It does not remove fallback traffic, retries, or evaluation cost, and it gives up control over serving-engine version, quantization, and residency. The per-token rate is also higher than a fully utilized dedicated replica, so the advantage flips back at sustained high volume.

Price both deployment shapes on the same traffic period. At low and bursty volume, per-token hosting usually wins. At steady high volume, dedicated capacity usually wins. In between, the answer depends on the latency requirement and how much headroom that requirement forces. Part 27 turns this into a break-even volume.

A savings claim needs the same discipline as a quality claim: the traffic period it was measured on, the closed-model bill for that period, the complete open-model bill for that period, the fallback share, and the quality result over the same window. A number without those five pieces is a projection, not a measurement.

Report it per customer and per agent. Two customers on the same adapter can have opposite results because their traffic shapes differ. Part 47 covers how to prove savings per customer instead of from a token-price comparison; this part exists so that the bill you compare against is complete.

Converra benchmarks candidate models against scenarios generated from the agent's own instruction and measures cost and latency from the real calls, so the comparison starts from observed per-request cost on your agent rather than a rate card. When a fine-tuned candidate reaches production, the model production test decides on non-inferiority plus strictly lower cost, and the model-swap verdict on real traffic reports parity verified, regressed, confounded, or insufficient data with the cost delta per arm.

The capacity, headroom, and operations lines above are yours to measure; Converra reports what your traffic cost on each model, not what your GPUs cost to keep up. No open-weight production verdict for a customer agent has been published yet, so treat any projected saving as unobserved until it has been measured on your own traffic.