What Behavior Can You Transfer From a Closed-Model Agent, and What Gets Lost?

An agent's instruction asks a single model to do many things at once: follow rules, produce a schema, pick tools, read evidence, recall facts, reason through ambiguity, and decide when to stop. When the agent works, all of those jobs are being done well enough. When the model changes, each job transfers or fails on its own terms.

Supervised fine-tuning adjusts weights so that, given inputs like the training inputs, the model produces outputs like the training outputs. That description contains the whole answer. Jobs that show up as regularities in the input-output pairs transfer. Jobs that depend on what the base model brought with it from pretraining do not change, and jobs that depend on inputs the examples never covered are unaffected by training at all.

Output format transfers first and most reliably. A model that has seen several hundred examples of a schema stops producing near-misses. Rule adherence transfers when the rule fires often enough in the examples: refuse requests of type X, always confirm before calling tool Y, cite the document section. Domain conventions transfer: the customer's vocabulary, the order fields appear in, the tone of a reply, the length of a summary.

Tool selection transfers when the preconditions for each tool are visible in the examples and the tool set is stable. Abstention transfers when the examples include cases where the correct response was to say the evidence is insufficient. That last one is easy to forget and expensive to omit, because a model trained only on confident answers learns that confidence is always the right shape.

A closed frontier model answers questions about products, regulations, geography, and history because those facts were in its pretraining corpus. A 9B open model has seen a smaller corpus and retains less of it. Fine-tuning on a few hundred conversations does not put the missing facts back; the examples teach the model to sound like it knows them, which is worse than not knowing.

The fix is architectural, not a training change. Any fact the agent needs has to arrive in the context: retrieved from a knowledge base, fetched from a tool, or written into the instruction. If a team cannot enumerate where each fact comes from, the closed model is quietly supplying it, and that part of the workload will fail after transfer. Part 8 and part 9 cover what the open model should do when the evidence in context is incomplete or absent.

Frontier models handle a request they have never seen by decomposing it. Smaller models decompose less reliably, and fine-tuning helps only for decompositions that appeared in the examples. A training set full of three-step lookups teaches three-step lookups. It does not teach the model what to do with a five-step case that combines two lookups in an order the examples never showed.

The practical consequence is that the transferable workload is the set of problem shapes represented in the reviewed data, not the set of problem types. Two requests of the same type can differ in shape. The evaluation set has to include shapes the training set did not, or the gate will report a generalization the model does not have. Part 16 covers how to tell learning the task from learning the test set.

Production traffic has a head of common requests and a long tail of rare ones. The closed model handles the tail acceptably because it has general competence. The fine-tuned open model handles the head well and the tail badly, because there were never enough examples of any single tail request to learn from. This is the most common way a transfer that passes its gate still disappoints: the gate was drawn from the head.

Two responses are honest. Either measure the tail explicitly, accept the loss, and route tail requests to the closed model, or accept that the transfer covers the head and design the escalation rule accordingly. Training on the tail does not work; twenty examples spread over twenty rare types is one example each. Part 11 takes up the head-versus-edge-case question directly.

Closed models are often better at expressing uncertainty than open ones, and fine-tuning on confident outputs removes what uncertainty the base model had. The transferred agent then produces the same tone for a well-supported answer and an unsupported guess. Downstream systems that relied on hedging language as a signal lose that signal without anyone noticing.

Restore it deliberately. Include examples where the correct output says the evidence supports only part of the request. Add a structured confidence or support field to the schema so the signal is machine-readable rather than tonal. Then evaluate no-evidence and partial-evidence requests as separate populations, which part 18 explains, so a model that has learned to abstain gets credit for it.

Pull a sample of successful production runs and annotate each with the jobs the model performed: which rules fired, which tools were chosen, which facts were stated, and where each fact came from. Count how often the answer required knowledge outside the context. Count how often the request shape was unusual. Count how often the model hedged and whether anything downstream depended on it.

That annotation is the transfer map. It shows the share of traffic the open model can plausibly inherit, the facts that must be moved into retrieval or tools first, and the request shapes the evaluation set must contain. It also produces a first estimate of how much of the agent stays on the closed model under a routing design, which is the number the economics in part 26 depend on.

Suppose an agent reads a supplier document, calls a sanctions screen, a registry lookup, and an adverse-media search, and writes a structured risk report. The schema, the risk-tier rule, the requirement to cite each tool result, and the choice of which lookups to run all transfer, because they are regularities over evidence the agent already has. The report's narrative judgment about how two conflicting sources should be weighed transfers weakly, and any claim about an entity that the lookups did not return would be knowledge the open model does not have.

The transfer plan follows. Train on the structured report over pre-fetched tool results. Score the tier, the required findings, and the citations deterministically. Keep narrative quality as secondary evidence. Route any case where the lookups return conflicting or empty results to the closed model, and seed the evaluation set with adverse cases so a model that has learned to write reassuring reports gets caught.

Converra's step-level diagnosis identifies the exact step in a run where behavior went wrong and classifies the cause, which is the same annotation this article asks for when it says to map which jobs an agent depends on. Its model benchmarks then run candidate models against scenarios built from the agent's own instruction and open a pull request with the winning switch, so a team can see where a smaller model already competes before training anything.

For a fine-tuned candidate, Converra curates reviewed production runs into versioned datasets with protected splits, records the training job and gate result against a dataset version, holds the switch behind an approval, and reports a parity verdict on real traffic. Training runs on the customer's provider, not inside Converra. Converra does not use customer data to train any model without a written, tenant-exclusive election, and no open-weight production verdict for a customer agent has been published yet. Which jobs transfer for a given agent is unobserved until measured on that agent's traffic.