Does the Open Model Meet the Customer's Latency and Throughput Requirements?

A user-facing agent's response time is built from four pieces. Time to first token: how long until the model starts producing. Generation speed: tokens per second once it starts. Queueing: how long a request waits when the endpoint is busy. Tool round trips: the calls the agent makes between model turns, each of which adds its own latency and often a second model call to interpret the result. A multi-step agent may pay the first three costs several times per user turn.

The closed provider handled the first three invisibly at scale. On an open endpoint, each becomes a decision. Time to first token depends on how fast the prompt is processed, which depends on prompt length and hardware. Generation speed depends on model size, quantization, and batch size. Queueing depends on how many concurrent requests the endpoint can hold, which depends on memory. Write the budget as those parts, then measure each.

Every request in flight holds its context in the GPU's key-value cache for the duration of the request. That cache is finite and shared. A model that fits forty short requests in the cache at once fits perhaps eight requests at the agent's worst-case prompt length. The ninth queues. This is the mechanism that connects part 38's long-context question to this one: an agent with long prompts gets lower throughput from the same hardware, and its p95 latency rises with concurrency faster than a short-prompt agent's does.

Batch size is the knob. Larger batches raise tokens per second across all requests and raise per-request latency, because each request shares the GPU with more neighbors. Serving stacks such as vLLM manage this dynamically, but the ceiling is set by memory. Quantizing the weights frees memory for the cache, which is why part 44 treats quantization as a throughput decision as much as a quality one.

A dedicated endpoint gives fixed capacity: predictable latency up to a concurrency ceiling, then queueing, then failure. It also idles when traffic is low, which is a cost problem rather than a latency one and belongs to part 26. A serverless open endpoint shares capacity with other tenants: no idle cost, but latency depends on their load, and a cold start can add seconds to time to first token when the model has to be loaded.

The test plan has to match the endpoint the agent will run on. Load-testing a dedicated endpoint and then deploying serverless, or the reverse, produces numbers that describe a system the customer does not have. If the plan is to start serverless and move to dedicated at volume, test both and record the switchover point in the plan.

Take the agent's real concurrency from its traffic: requests in flight at peak, not average requests per minute. Take the prompt-length distribution from the same traffic, including system instruction, retrieved context, tool results, and conversation history, and sample prompts from it rather than using one representative prompt. Replay that mix against the candidate endpoint at peak concurrency for long enough that queues reach steady state, and record time to first token, total latency, and failures for every request.

Then raise concurrency past peak until the p95 breaches the budget. That breach point is the endpoint's real capacity for this agent, and the gap between it and peak is the headroom. Headroom is what absorbs a traffic spike, a slow tool, or a retry storm. An endpoint whose capacity equals peak has none, and its p95 in production will not match the test.

The closed model's latency is also a distribution, and it is often wider than teams remember. Measure its p50 and p95 from the agent's own traces over the same period the load test represents. The candidate's target is that measured p95, plus whatever margin the customer accepts, not a number from a provider status page and not the team's impression of how fast it feels.

This comparison sometimes favors the open model. A dedicated endpoint near the application, with a smaller model, can beat a shared frontier API on time to first token and on tail latency, because it is not waiting behind other tenants. Whether it does for this agent is an empirical question, which is the point of measuring both sides the same way.

An agent that makes three model calls and two tool calls per user turn pays time to first token three times and generation latency three times, with tool latency in between. A small per-call regression becomes a large per-turn regression. Budget at the turn level, measure at the turn level, and report the per-call breakdown so a regression can be attributed to the model, the tools, or the number of steps.

This is also where a model swap can change the step count. A smaller model that makes an extra tool call to confirm what the frontier model inferred is slower for a reason that per-call latency will not show. Part 39 covers scoring the whole trajectory; the same replay produces the per-turn latency this section needs.

State the budget in the customer's terms: a user-facing turn completes within a stated time at p95, at peak concurrency, with a stated headroom multiple, at a stated failure rate. State which endpoint type it applies to. Then run the test and record the raw per-request results, the endpoint configuration, the model and quantization identity, the serving stack version, and the prompt sample used. A pass without that record cannot be reproduced when traffic changes.

If the candidate misses, the options are ordered by cost: a serving change such as quantization or a larger batch, a capacity change such as more replicas, a context change such as tighter retrieval, or a different base model. Each is a new configuration and needs the test rerun. The one option that is not available is deploying and hoping the tail behaves.

Converra measures real latency and cost from the actual calls when it benchmarks candidate models against scenarios generated from the agent's own instruction, and reports both alongside quality in the comparison table that ships with the model-switch pull request. That gives a per-scenario latency picture across models on the same cases, which is the starting point for the budget in this article.

A production load test at peak concurrency on the customer's endpoint is the customer's to run; Converra does not host or train models, and no open-weight production verdict for a customer agent has been published. After an approved switch, Converra's model production test and model-swap verdict report on real traffic whether the change held parity at strictly lower cost, regressed, was confounded, or has insufficient data.