How Do You Test Supported, Partial-Evidence, and No-Evidence Requests Separately?

A supported request is one where the input, the retrieved documents, or the tool results contain what the agent needs to answer. The correct behavior is to answer, and to cite what supported it. A partial-evidence request has some of what is needed and not all: a policy that applies but a threshold that is missing, a record found but a status unconfirmed. The correct behavior is a scoped answer that says what is known, what is not, and what would resolve it.

A no-evidence request has nothing in the available material that bears on the question. The correct behavior is to say so and to escalate or ask, not to produce a plausible answer from general knowledge. These are three distinct skills. A model can be excellent at one and dangerous at another, and a closed frontier model with a good prompt is often reasonable at all three while a fresh fine-tune is strong on the first and weak on the third.

Suppose the evaluation set is 70 percent supported, 20 percent partial, and 10 percent no-evidence, which is close to many production distributions. A candidate that answers every supported case correctly, handles partial cases adequately, and invents an answer on every no-evidence case scores around 90 percent overall. So does an incumbent that abstains correctly on every no-evidence case and misses a few supported ones. The blended number says they tied.

They did not tie. One of them will tell a customer something false ten percent of the time. The blend cannot see that, because the cases where the two models differ most are the smallest slice, and the metric weights slices by frequency rather than by consequence. The remedy is not to reweight the blend; it is to stop blending.

The evidence condition is a property of the input, so it can and should be assigned when the case is built, before any model runs on it. For synthetic cases, the generator knows what it put in the fixture and can label the case directly. For cases drawn from production, a reviewer labels the condition from the input and the available tool results, without seeing any model's answer, so the label is not biased by what a model happened to say.

Record the label on the case with the rest of its lineage. A case whose condition is disputed goes to a second reviewer or is set aside; it does not go into the holdout with a guessed label. The label is what the scorer keys on, and a wrong label produces a wrong score for every model in the comparison.

For supported cases, score the answer against the answer key or reviewed target: correct decision, required findings present, citations that exist in the input, no unsupported entities. For partial cases, score two things: whether the answered portion is correct and whether the model marked the missing portion as missing rather than filling it. A partial case answered completely and confidently is wrong even if the completed part happens to be right.

For no-evidence cases, score abstention. Did the model say it could not answer from the available material, and did it take the right next action: escalate, ask, or return an explicit insufficient-evidence result. A fluent answer scores zero regardless of whether it is coincidentally true, because the behavior being tested is the refusal to invent, not the accuracy of the invention.

Abstention is a decision, and decisions have two error types. Abstention recall is the share of no-evidence cases where the model abstained; a low number means the model invents. Abstention precision is the share of abstentions that were on no-evidence cases; a low number means the model refuses supported requests it should have answered. A fine-tune trained heavily on refusals can push recall to one and precision to the floor, which is a model that says it cannot help to customers it could have helped.

Report both numbers per model. The incumbent's values are the target; a candidate that matches recall and loses precision has traded one failure for another, and the blended accuracy will not show it. Part 9 goes into what the correct no-evidence response looks like and how to put it in training data without teaching blanket refusal.

Lay the results out with one row per evidence condition per model, and the abstention rates beside them. The table below is a hypothetical layout with illustrative values for an incumbent and one candidate. Read down the no-evidence rows first; that is where transfers fail.

A pass rule that requires parity on the blend is the wrong rule. Require non-inferiority on each row separately, with a margin that reflects the consequence of failure in that row. The supported row can carry the widest margin, because a wrong answer there is usually caught downstream. The no-evidence row should carry the tightest, often zero tolerance for a drop in abstention recall, because an invented answer is the failure customers remember.

Sizing matters here. The no-evidence slice is small in a frequency-weighted sample, and a small slice cannot support a tight margin. Oversample it. Build the evaluation set so that each condition has enough cases to support its own rule, even if that means the set's condition mix no longer matches production. The set is for detecting differences, not for estimating production frequency; production frequency is used later to weight the cost and quality projection.

Partial cases resist a single correct answer, which is why they are often dropped from evaluation sets and why models are then never tested on the thing they do worst. Make the scoring concrete. The answer key for a partial case lists what can be concluded from the available evidence, what cannot, and what the model should say about the gap. The scorer checks each part: concluded items present and correct, unconcludable items not asserted, gap named.

Two failure modes to distinguish. Overclaiming asserts the unconcludable item; that is the hallucination pattern and should be weighted heavily. Underclaiming withholds a concludable item; that is a usefulness loss and should be weighted lightly. A candidate that shifts from overclaiming to underclaiming is safer but less useful, and the customer should see that trade-off as two numbers rather than one.

The same three rows should survive into the production test. Live traffic does not arrive labeled, but a judge or a deterministic check can assign evidence condition after the fact from the retrieved material, and the production verdict can then report parity per condition rather than in aggregate. A model that verified parity on the blend and regressed on no-evidence requests has regressed.

Carrying the row structure from gate to production also makes the two comparable. If the gate said the candidate abstains as well as the incumbent and production says it does not, the gap is diagnostic: either the production distribution of no-evidence requests differs from the set, or the retrieval that determines evidence condition behaves differently live. Both are worth knowing, and a blended metric would have shown neither.

Converra's step-level diagnosis identifies where in a conversation an agent's behavior went wrong and what evidence it had at that point, which is the raw material for labeling evidence condition on production runs. Its fine-tuning workflow curates reviewed runs into versioned datasets with protected splits and per-example admission decisions, records the training job and gate result against that dataset version, and holds the production switch behind an explicit approval. The model production test decides on non-inferiority plus strictly lower cost on live traffic.

Converra does not run training and does not use customer data to train any model without a written, tenant-exclusive election. No open-weight production verdict for a customer agent has been published yet, so per-condition parity on a given agent's traffic is unobserved until the verdict returns parity verified, regressed, confounded, or insufficient data.