Can a Smaller Model Learn When the Evidence Is Incomplete?

An agent run carries its evidence in the trace: the user turn, retrieved documents, tool results, and any structured state. Evidence is complete when every claim the agent needs to make is supported by something in that trace. It is incomplete when a tool timed out, a lookup returned nothing, a document covers two of the three questions asked, or a record exists but a required field is blank. The agent still has to produce a turn.

Frontier closed models tend to paper over these gaps with fluent, plausible text. Their training makes a confident answer the default. A smaller open model trained on those outputs inherits the same default with less knowledge to back it up, which is a worse combination. So the question is not whether the smaller model can be as fluent when evidence is missing. It is whether it can be taught a different behavior for that condition.

The cleanest way to reason about this is to split every training and evaluation case into one of three conditions before you write a single label. Supported: the trace contains everything needed and the target is a full answer. Partial: the trace supports some claims and not others, and the target is a scoped answer that names its boundary. None: the trace supports nothing relevant, and the target is an explicit abstention or escalation. Part 9 covers the third condition in detail.

The partial condition is where most transfers go wrong, because it is the one teams forget to label. Rows land in the dataset as either good answers or bad answers, and a scoped answer that says less than the teacher said looks like a bad answer to a reviewer who is grading fluency. Labeling the condition first, then the target, changes what a correct answer is for that row.

Recognizing that a field is blank, that a tool result is empty, or that two numbers disagree is a pattern-matching task over the context window, not a knowledge task. Smaller models are reasonably good at pattern matching over text they can see. What they lack is the world knowledge to fill a gap correctly, which is exactly the capability you do not want them to exercise here. The behavior you are training is, in effect, to notice the gap and stop.

This is why partial-evidence handling often transfers better than open-ended reasoning. The target output is bounded by the trace. A model that says what is present and names what is absent has done the job. A model that goes further is the one you need to catch, and the failure is visible in the output because the extra claim has no anchor in the input.

Production traffic will contain partial-evidence runs, but not in the proportion you need and not with the variety you need. If ten percent of runs are partial and they all come from the same tool timing out, the model learns one pattern and one tool. Seed the training set with partial cases across every evidence source the agent uses: each tool, each document type, each structured field that can be empty, and each pair of sources that can conflict.

Balance by family, not by row. A family is the unit that shares a learning signal: one customer account, one scenario template, one document. If forty partial rows come from six families, the model has seen six partial situations. Count families per condition and set a floor for each. A partial condition with fewer than a dozen distinct families is not represented; it is mentioned. Part 10 goes further into how to count.

A prompt instruction such as only answer what the evidence supports helps a frontier model and does little for a small fine-tuned one, because the small model learns from what the targets do, not from what the instruction says. The scoped answer has to be present in the training targets in a consistent shape: the supported claims, then an explicit statement of what could not be determined, then what would resolve it. If the agent emits structured output, put the boundary in a field so it can be checked by code.

Consistency matters more than eloquence. If half the partial targets say the record was unavailable and the other half silently omit the topic, the model learns that omission is acceptable. A reviewer creating targets for partial cases should follow one template for the boundary statement, and the review step in part 6 should reject targets that skip it.

The closed model that produced the original runs almost certainly filled some of these gaps. Its fluent answer to a partial case is the worst possible training target, because it teaches the exact behavior you are trying to remove while looking like a high-quality example. Every partial-condition target needs a reviewer to compare the answer against the trace and strip any claim that has no anchor in it.

The practical check is claim by claim: for each factual statement in the target, point to the tool result, document span, or state field that supports it. A statement with no pointer is either removed or converted into the boundary statement. Part 7 covers this process for the whole dataset; here the point is that partial cases need it most, and they are the cases where a reviewer grading on fluency will most often wave the guess through.

If partial cases are pooled with supported cases in the evaluation, a model that fills gaps confidently can score well on average, because most rows are supported and the fill looks fluent. Hold the partial cases out as their own slice and score them on two things. Coverage: did the answer include the claims the evidence supports? Containment: did it exclude every claim the evidence does not support, and did it state the boundary?

Both directions matter. A model tuned only against invention will learn to say less and less, which is its own failure. Suppose a protected slice holds 30 partial cases. A candidate that names the boundary in 29 and invents in 1 has a measured invention rate of about 3 percent with a wide interval; a candidate that names the boundary in all 30 but answers only half the supported claims has a containment win and a coverage loss. Report both numbers, never one.

Suppose a support agent answers order questions from an order-lookup tool and a policy document. Across a sample of 2,000 production runs, a reviewer finds 1,600 supported, 300 partial, and 100 with no usable evidence. The partial runs split into 180 where the lookup returned an order with an empty delivery-estimate field, 90 where the policy document did not cover the customer's region, and 30 where two lookups disagreed on order status.

Trained on the original closed-model outputs, the candidate learns to state a delivery estimate in the 180 empty-field cases, because the teacher did. Trained on reviewed targets that answer the order status and state that the estimate is not yet available, it learns the boundary. On a protected partial slice of 60 cases held out by order family, the first candidate shows a high unsupported-claim rate concentrated on the empty field; the second shows near-zero invention with a small coverage cost. Only the second is a candidate for the parity gate in part 21.

Converra's evidence pipeline scores each production run on whether the agent's claims are backed by the tool results and documents in the trace, and its tool-calling analysis records per-call verdicts, so partial-evidence runs surface as findings rather than as invisible fluent answers. For a fine-tuned candidate, its fine-tuning workflow curates reviewed runs into versioned datasets with protected splits and a per-example admission decision, which is where the claim-by-claim boundary check for partial cases belongs. Training itself runs on your provider or infrastructure; Converra records the job and its gate result against the dataset version.

Converra does not use customer data to train any model without a written, tenant-exclusive election, and no open-weight production verdict for a customer agent has been published yet. Whether your candidate handles partial evidence at parity is unobserved until the protected slice and then real traffic say so.