How Do You Evaluate the Entire Agent When Only Its Underlying Model Changes?

An agent is an instruction, a model, a set of tools, a retrieval layer, some state handling, and an output contract, wired together and tuned as a unit. The instruction was written against the old model's habits. The tool descriptions were phrased so the old model would pick them. The retrieval budget was sized to what the old model could use. None of that is documented as a dependency on the model, but all of it is one.

This is why a chat playground comparison says so little. It shows how two models answer a prompt in isolation. It does not show whether the new model, given the same instruction and the same tools, follows the same route through the task, calls the same tools with valid arguments, uses what retrieval returns, and produces an output the downstream parser accepts. Those are the properties the agent's users experience, and they can only be observed by running the agent.

Write down the exact agent bundle: the instruction text and its version, the tool schemas and descriptions, the retrieval configuration and the index snapshot, the sampling parameters, the output schema, the maximum turns, and the timeout and retry policy. Every one of these stays fixed across both arms. If any of them changes between the incumbent run and the candidate run, the comparison measures two changes and attributes the result to one.

The hardest item to freeze is the environment. Tools that hit live systems return different data on different days. Retrieval over a changing index returns different documents. Pre-fetch tool results and pin the retrieval snapshot for the evaluation set, so both models see the same world. Where a tool result depends on what the model asked, serve it from a fixture keyed on the call and record any call the fixture cannot answer as off-script.

The scenarios should be the ones the agent actually faces: reviewed production cases, seeded hard cases, and clean controls for behavior the incumbent already handles well. Each scenario has a defined starting state and an expected outcome, and each is run through the full agent loop rather than through a single model call. A scenario that takes four tool calls in production takes four tool calls in evaluation.

Run each scenario more than once per model, because sampling noise is real even at low temperature, and score the median. Then pair by scenario: the incumbent's median on case 17 against the candidate's median on case 17. Aggregates come last. Two averages that look similar can hide a candidate that improved on easy cases and collapsed on the hard ones, and the paired view shows exactly where the two diverged.

A trajectory has several checkpoints and each can fail independently. Did the agent route to the right specialist or branch? Did it call the tools the case required, with arguments drawn from the conversation rather than invented? Did it use the retrieved evidence, or answer past it? Did it carry state from early turns to late ones? Did the final output satisfy the schema and the task? A final answer that happens to be right after a wrong route is a failure waiting to recur.

Score each checkpoint with the cheapest trustworthy method. Routing, tool choice, argument validity, and schema conformance are deterministic checks. Whether the answer used the evidence and completed the task is a semantic judgment, and it needs a calibrated grader or a reviewer. Keep every checkpoint score separate. Collapsing them into one number is how a model that formats beautifully and reasons worse passes.

A regression suite is the set of scenarios the incumbent passes today. It exists to make sure the candidate does not trade a gain on the target behavior for a loss somewhere the team stopped watching. Run it in full, pair it in full, and treat any material regression as a blocker regardless of the aggregate. The suite should include the boring cases, because boring cases are most of production.

The regression suite is also where output repair has to be turned off. Production stacks often quietly fix malformed outputs before the parser sees them. In evaluation that hides exactly the failures a smaller model produces most. Score the raw output and count repairs separately; part 20 of this series covers why.

When the candidate loses a paired case, find the checkpoint where the trajectories diverged. That is the diagnosis, and it decides the next move. A divergence at routing on a conditional instruction says the instruction reads differently to the new model. A divergence at tool arguments says the new model fills gaps by inventing. A divergence at the final answer with identical trajectories says the models differ in generation quality on this case class.

Group the diverging cases by checkpoint and by case class. A candidate with ten losses all at the same checkpoint has one problem, possibly fixable. A candidate with ten losses scattered across checkpoints and classes is behaving differently in general, and that is a capability gap rather than an integration gap. Step-level attribution is what separates those two conclusions, and they lead to opposite decisions.

Instructions written against a frontier model lean on habits that model had. It inferred which tool to call from a terse description. It tolerated an ambiguous conditional. It formatted the output correctly from a one-line schema hint. A smaller model may need the description expanded, the conditional made explicit, or the schema shown in full. These are small edits, and they are legitimate, as long as they are tested as their own treatment.

The discipline is sequencing. First, evaluate the candidate with the frozen prompt and record the result. Then, make the smallest prompt edit that targets the diagnosed divergence, and evaluate again, on both models, with the edit as the only change. If the edit helps the candidate without hurting the incumbent, the transfer now includes a prompt change, and the production switch has to ship both. If the edit only helps because it re-teaches the candidate the test cases, the protected holdout will show it. Part 17 covers keeping that holdout honest.

Record per-scenario latency and cost for both models from the actual calls, not from list prices. A candidate that matches quality while taking twice as many tool round trips has a different cost and latency profile than the per-token price suggests. Record execution failures too: timeouts, malformed outputs that the parser rejected, tool calls the fixture could not answer. Exclude infrastructure failures symmetrically and count them per arm; if too many cases are excluded on either side, the run is invalid rather than a pass or a fail.

The final artifact is a per-scenario table with both models' checkpoint scores, latency, cost, and any exclusion reason, plus the aggregate and the list of diverging cases with their attributed step. That is the evidence a reviewer needs to approve a switch. A single overall score is not.

Converra benchmarks candidate models by running every model on every scenario generated from the agent's own instruction, three times with median scoring, and measures real cost and latency from the calls. Its regression testing replays the scenarios the current agent handles well against any change, and its step-level diagnosis locates the exact step and turn where a conversation diverged, which is the attribution this article's fifth step depends on. A winning switch opens as a reviewed pull request with the comparison table.

Converra does not train models, and no open-weight production verdict for a customer agent has been published. After a switch is approved, Converra's model production test decides on non-inferiority plus strictly lower cost, and the model-swap verdict on real traffic reports parity verified, regressed, confounded, or insufficient data. The outcome for any given agent is unobserved until it is measured there.