What Changes When You Quantize or Otherwise Optimize the Open Model for Serving?

Post-training quantization stores weights in fewer bits, usually 8 or 4 instead of 16. Methods such as GPTQ and AWQ pick the rounding so that the model's outputs on a calibration set stay close to the original. Close is not identical. Every quantized model is an approximation of the one you evaluated, and the approximation error is not spread evenly across requests.

The practical consequence is that a candidate can pass the gate at full precision and fail it at 4 bits, without any change to the prompt, the data, or the adapter. Teams that treat quantization as a deployment detail rather than a model change discover this on production traffic. Teams that treat the quantized artifact as a new candidate discover it in the gate, where it is cheap.

Quantization error shows up first where the model was already uncertain. Rare tool choices, long-tail vocabulary, numbers and identifiers copied from evidence, multi-step reasoning that depends on a narrow margin between two continuations, and abstention decisions near a threshold. These are exactly the cases a transfer gate is built to protect, which is why an average score can look unchanged while adverse-case recall drops.

Fine-tuned behavior is more exposed than base behavior. A LoRA adapter encodes a small, specific change on top of the base weights. Quantizing after merging can round away part of that change, and the model drifts back toward the base model's defaults on precisely the behaviors you trained. Measure the trained behaviors separately from general quality after any quantization step.

The key-value cache holds the attention state for every token in the context. Quantizing it saves memory in proportion to context length, which is why serving stacks offer it. The error it introduces also grows with context length, because each generated token attends over more approximated state. A model that answers short requests perfectly can lose track of an instruction buried early in a long transcript.

Evaluate at the agent's real worst case, not its median. If production requests run to forty thousand tokens with a long document and a tool history, the gate needs cases of that shape with KV-cache quantization enabled. A clean result on two-thousand-token cases says nothing about the long ones, and long ones are usually the ones where the customer's evidence lives.

Serving a LoRA adapter separately keeps the base weights untouched and applies the adapter at inference. Merging folds the adapter into the base weights, which is simpler to deploy and often faster. Merging is exact in floating point and approximate once the merged weights are quantized. The evaluated artifact was base plus adapter; the served artifact is a rounded merge. They can disagree.

Treat the merged, quantized model as a new candidate identity with its own receipt: base model version, adapter version, merge method, quantization method and calibration set, and the serving stack version. Run the gate on it. If it passes, the receipt is what production verification later joins to. If it fails, you have learned that the deployment path, not the training, is where the quality went.

Speculative decoding uses a small draft model to propose several tokens and the target model to verify them in one pass. Done correctly, the output distribution is unchanged, and the gain is latency. The acceptance rate of drafted tokens determines how much latency you gain, and that rate depends on how predictable your agent's outputs are to the draft model.

Two checks matter. First, sample outputs with and without speculative decoding at temperature zero and confirm they match; a mismatch means the implementation is not the exact variant or a numerical detail differs. Second, measure latency under production concurrency. Speculative decoding helps most at low batch sizes and can help less, or not at all, once the server is saturated. The latency table in part 40 is the right place to record this.

Continuous batching lets a server interleave many requests and keeps the GPU busy, which is where most of the cost advantage of self-hosting comes from. It also means one request's latency depends on what else is in flight. A long prompt arriving while the batch is full waits, and a burst of long-context requests can push p95 latency past the budget while the median looks fine.

Load-test with the real distribution of prompt lengths, output lengths, and arrival rates, and read the p95 and the error rate, not the mean. Memory is the coupling: each in-flight request holds KV-cache proportional to its context, so the batch size the server can sustain falls as contexts grow. A configuration that works at the median context can fail under a run of long ones, which is when fallbacks in part 46 take over.

Serving stacks impose a maximum context, a maximum output length, and often a request timeout. When an input exceeds the context limit the server may truncate silently, reject the request, or drop the earliest tokens depending on configuration. When output hits the length cap, the answer ends mid-sentence or mid-JSON. Each of these looks like a model failure in the transcript and is a configuration failure in fact.

Set the limits from measured production lengths with headroom, then verify the failure path. Send a request over the limit and confirm the agent sees an explicit error it can route to the fallback rather than a truncated context it answers from. Record the truncation rate as a served metric; a rising rate after launch usually means the traffic mix moved, not that the model regressed.

Every optimization in this part moves at least two of quality, latency, and cost, and the direction is not always the one you expect. Four-bit quantization halves memory, which can double the batch size, which raises throughput and lowers cost per token, and it can also raise p95 latency at high load and lower adverse-case recall. You cannot pick the configuration from one number.

Run the paired gate for quality, the load test for latency at the required concurrency, and the cost model from part 26 with the measured throughput. Record all three against the served artifact's receipt. The same set of cases, scored the same way, on every configuration you consider. Then choose, and keep the alternatives' receipts so a rollback to a less aggressive configuration is a known quantity rather than a guess.

The rule that ties this part together: the artifact that passes the gate is the artifact you serve. If anything between the gate and the endpoint changes the weights, the cache precision, or the decoding path, the gate runs again on the changed artifact. The earlier result stays valid for the earlier artifact and nothing else.

This is cheap relative to the alternative. A paired gate run on a hosted endpoint costs an hour or two of GPU time. A quantization regression found on production traffic costs a rollback, a confounded verdict, and the trust of the customer who approved the switch. Part 32 explains how production verification joins a verdict to an exact served identity; that join only means something if the identity is precise.

Converra records candidate identity per gate result and per production switch, so a quantized or merged artifact enters the workflow as its own candidate with its own gate record rather than inheriting the full-precision result. Converra's model benchmarks measure quality, cost, and latency from real calls on the same scenarios for every model under test, which is the comparison this part asks for across serving configurations. The production A/B test then decides on non-inferiority plus strictly lower cost, and the model-swap verdict on real traffic reports parity verified, regressed, confounded, or insufficient data with the cost delta per arm.

Converra does not run training or quantization itself. Those steps happen on the customer's provider or infrastructure, and the resulting artifact identity, gate result, and serving receipt are what Converra records and verifies against. No open-weight production verdict for a customer agent has been published yet, and the effect of a given quantization on a given agent stays unobserved until the paired gate and the production measurement have both run.