How Do You Prove Savings per Customer Rather Than From a Theoretical Token-Price Comparison?

The price comparison assumes the two models consume the same tokens per request, answer the same share of requests, and produce the same quality. None of those hold by default. A fine-tuned open model may need a longer prompt or produce longer outputs. A share of requests fall back to the closed model and cost both. Hosting is billed by the hour whether or not requests arrive. And if quality dropped, the cheaper request bought a worse outcome, which is not a saving.

The list-price view has one legitimate use: sizing the opportunity before the work starts. A cost view that estimates what current traffic would cost on a cheaper model, labeled as an estimate at list prices, is a fair way to decide whether to try. It stops being fair the moment it appears in a report as a result.

Tokens are the input to a cost, not the outcome anyone buys. The unit that matters is the cost of a request that achieved the agent's outcome: a resolved conversation, a correct report, a completed booking. Cost per successful request combines price, token usage, fallback share, and quality into one number that moves in the right direction when any of them gets worse.

Define success the way the gate and the production verdict define it, using the same outcome version. If a request was unscored, it does not enter the numerator or the denominator. If it failed, its cost counts and its success does not. The result is a number the customer can reconcile against their own bill and their own outcome, which is the test a savings claim has to pass.

A production split test is the cleanest source of paired cost data. During the test, each arm serves a share of comparable traffic and every run records the model that served it, the input and output tokens the provider reported, and the outcome. Cost per run is the reported usage multiplied by the price in force for that model on that date. Cost per arm is the sum over runs the arm served, divided by the arm's successful requests.

Use reported usage, not an estimate from a tokenizer, because providers bill on their own count and the two can differ. Use the price in force on the day, recorded with the run, because prices change and a savings claim computed at today's price for last quarter's traffic is wrong in both directions. Record the price with the run, not in a spreadsheet.

The open arm's cost is more than its per-token bill. Dedicated capacity costs money while idle. Autoscaling headroom is capacity you pay for and rarely use. Every fallback costs the closed model's tokens plus the wasted open attempt. Retries cost tokens. Observability, storage of traces, and the engineers who operate the endpoint cost money that the closed arm does not incur. A savings claim that leaves any of these out is comparing a full bill to a partial one.

The table below lists the cost components, where each one shows up, and how to attribute it to a customer. The attribution rules are the part teams skip, and they are where a plausible-looking savings figure most often falls apart under a customer's finance review.

A shared endpoint that serves several customers has a single hourly bill and no natural per-customer split. Allocate it by each customer's share of served tokens in the period, and report the idle share separately rather than folding it into one customer's rate. If one customer generates most of the traffic during business hours and the endpoint idles overnight, the overnight cost belongs to the fixed line, not to that customer's per-request cost.

State the allocation rule in the report. A customer who sees an allocation they disagree with can argue about the rule; a customer who discovers an undisclosed allocation stops trusting the number. When a customer runs the endpoint in their own account, the bill is theirs and the attribution problem disappears, which is one of the reasons part 43 lists control over hosting as a benefit beyond price.

A cheaper arm that delivers worse outcomes has not saved money; it has shifted cost to the customer's users and to whoever cleans up the failures. So the savings figure is conditional on the quality result. If the production verdict is parity verified, the cost delta is a saving. If it is regressed, the cost delta is the price of a worse product and should be reported as such. If it is confounded or insufficient data, the savings are unproven.

Attach the verdict state to the savings figure wherever the figure appears. Cost per successful request already encodes quality in its denominator, but a reader skimming a percentage does not see that. A single line, savings measured at parity verified over this period and population, keeps the claim honest in every slide it gets copied into.

Compare the arms over the same window and the same traffic mix. A before-and-after comparison across different months mixes in seasonality, product changes, and traffic growth, all of which move tokens per request and success rates independent of the model. The split test avoids most of this by serving both arms concurrently; a sequential comparison needs the mix to be checked explicitly.

Record concurrent changes: a prompt edit that shortened outputs, a new caching layer, a provider price change, a tool that started returning shorter results. Any of these moves cost without the model doing anything. If one landed in the window, the comparison is confounded for cost just as it would be for quality, and the report should say so rather than crediting the model.

A savings claim should read like a measurement, not a headline. State the cost per successful request on each arm, the number of successful requests and total requests per arm, the period, the fallback share, the hosting allocation rule, the verdict state, and the confidence interval on the difference. If the interval crosses zero, say the saving is not yet distinguishable from noise and what volume would resolve it.

Give the customer the projection separately. The measured difference per successful request multiplied by their expected volume is a forecast, useful for planning, and it should be labeled as one next to the measured figure it is built from. Keeping the two apart is what lets the measured number stay credible when the forecast turns out to be off.

Suppose a split test serves twenty thousand comparable requests per arm over three weeks. The closed arm reports an average of six thousand tokens per request at its list price and a success rate of ninety-one percent. The open arm reports seven thousand tokens per request on the requests it served, an eight percent fallback share that costs the closed price plus the wasted attempt, a hosting allocation for the period, and a success rate of ninety percent, inside the gate's margin. The production verdict is parity verified.

Cost per successful request on the closed arm is its total spend divided by roughly eighteen thousand two hundred successes. Cost per successful request on the open arm is open-model spend plus fallback spend plus wasted attempts plus the hosting allocation, divided by roughly eighteen thousand successes. The saving is the difference, reported with both denominators, the fallback share, and the verdict. Whatever percentage results, it is a measurement of these three weeks, not a property of the model, and every number in this paragraph is hypothetical.

Converra's model production test decides on non-inferiority plus strictly lower cost, with cost and latency measured per arm from reported usage during the split, and the model-swap verdict on real traffic reports parity verified, regressed, confounded, or insufficient data together with the cost delta per arm. That is the measured, verdict-attached figure this part asks for. Converra's cost view separately shows what current traffic would cost on a cheaper model at list prices, and labels it as an estimate.

Hosting allocation, operations cost, and the customer's finance rules for what counts remain the customer's to define, and the open model itself runs on the customer's provider or infrastructure. No open-weight production verdict for a customer agent has been published, so no savings figure from this workflow exists to cite; the saving on a given agent is unobserved until the split has run and the verdict has come back.