Which Agent Workloads Are Good Candidates for Closed-to-Open Model Transfer?

Teams usually start by comparing models. The more useful first step is to describe the workload the agent is actually doing and ask which of its properties a smaller model can inherit from examples. A closed frontier model wins on a broad benchmark because it knows more and reasons deeper. A fine-tuned open model wins on a narrow production task because it has seen hundreds of reviewed examples of exactly that task. Those are different games.

The screening question is therefore not whether the open model is as good as the closed one, but whether this specific workload sits inside the region where behavior learned from examples is enough. That region is easy to describe once you know what to look for, and most agents have parts inside it and parts outside it.

Six traits show up in every transfer that holds. Bounded scope: the agent does a small number of things and the boundaries are written down. Evidence in hand: the answer is derivable from the prompt, the tool results, or retrieved documents rather than from what the model remembers. Checkable outcomes: a correct answer can be verified by a rule, a reference, or a reviewer in minutes. Volume: enough traffic that serving cost matters and enough examples to train on. Stable contract: the task has not changed materially in months. Tolerance for escalation: a wrong answer can be caught and handed off rather than silently acted on.

A workload with all six is a strong candidate. A workload missing one is worth a closer look. A workload missing three is a research project. The table below turns the traits into questions with a pass, caution, or stop reading for each.

Classification and routing over a fixed set of outcomes. Extraction of structured fields from documents the agent is given. Summaries of supplied material into a fixed format. Grounded question answering over retrieved passages where the answer is quoted or derived, not composed from memory. First-line support replies that follow a written policy with a handful of branches. Tool-call selection over a stable tool set with clear preconditions. Report generation from a document and a set of pre-fetched lookups.

What these have in common is that the closed model was never really being used for what it knows. It was being used for how reliably it follows the instruction over evidence the agent already collected. That reliability is exactly what supervised examples teach a smaller model, and it is why formatting and rule adherence are usually the first things that improve after fine-tuning.

Open-domain research where the model must know things the prompt does not contain. Multi-step planning with a long horizon and many branch points. Code generation across unfamiliar libraries. Persuasion, negotiation, and creative writing where quality is a matter of taste. Anything where the long tail of request types is wide and each type is rare. Safety-critical judgment calls where the cost of a confident wrong answer is high and no escalation route exists.

These workloads fail transfer for a structural reason, not a tuning reason. Fine-tuning changes how a model behaves over inputs that resemble its examples. It does not add knowledge and it does not deepen reasoning on shapes the examples never showed. A team that pushes a workload like this through a transfer will produce a model that passes a narrow gate and disappoints on live traffic, and part 3 explains why in detail.

A real production agent rarely sits entirely on one side. A support agent answers policy questions from a knowledge base (transfers), files tickets with structured fields (transfers), and occasionally reasons through an unusual billing dispute (does not). A research agent classifies a document's type (transfers), extracts named entities (transfers), and writes a risk narrative that weighs conflicting sources (mostly does not).

The right move is to classify the agent's request types individually and treat the transferable share as the candidate. That reframes full replacement as routing, which part 22 covers, and it lets the project deliver a measured saving on the share it can prove instead of stalling on the share it cannot. A workload split also tells you what the training data has to cover and what the evaluation set must exclude.

Suppose an agent handles 300 requests a day. Even if the open model is free per token, a dedicated endpoint that has to stay warm costs more than the closed model's bill at that volume. Now suppose it handles 30,000 requests a day with a steady daily curve. The same endpoint is busy most of the time and the per-request cost falls well below the closed model's. Traffic shape decides the economics before quality is even measured.

Volume also decides the data. A workload with thousands of requests a day accumulates reviewed examples quickly and produces a holdout that looks like production. A workload with dozens per day may never reach the count and variety a fine-tune needs, and the gate will be too small to say anything. Part 27 works through break-even in detail; the screening rule is that low-volume workloads are candidates for routing to a shared open endpoint, not for a dedicated fine-tune.

A fine-tuned model encodes the task as it was when the examples were reviewed. If the agent's rules change every week, the model is stale before the gate closes, and every rule change forces a retraining decision. The closed model absorbs rule changes through the prompt; the fine-tuned model only partially does, because it has learned the old rule from hundreds of examples and the new rule from one sentence.

This is why the screening rubric asks when the task last changed. A task that has been stable for months is a candidate. A task whose owner is still rewriting the policy is not, and the honest recommendation is to keep it on the closed model until the contract settles. Part 33 covers when to retrain once a transfer is live.

Take three hypothetical agents. An order-status agent answers where-is-my-order questions from a tool lookup, handles a dozen request types, sees 20,000 requests a day, and can hand off to a person. It passes every row of the rubric and is the strongest candidate. A supplier-risk agent reads a document, calls three lookups, and writes a structured report: it passes on scope, evidence, and checkability, but its narrative section leans on judgment, so the candidate is the structured part of the report and the narrative stays on the closed model until it can be measured.

A general assistant that answers anything an employee asks fails on scope, evidence, and stable contract. It is not a candidate, and no benchmark result changes that. The useful output of screening is a sentence like that for each agent, written before anyone opens a training console.

Passing the rubric means a transfer is worth attempting, not that it will succeed. It says nothing about whether the reviewed examples exist yet, whether the open model you pick can handle the context length and tool calls involved, or whether the candidate will clear a non-inferiority gate. Those are the questions the rest of this series takes one at a time.

Screening also does not replace measurement. A workload that looks bounded on paper may have a long tail that only appears in production logs. Pull the request-type distribution from real traffic before trusting the written scope, and treat the tail as part of the workload rather than as noise.

Converra benchmarks candidate models against scenarios built from an agent's own instruction, runs every model on every scenario three times with median scoring, measures real cost and latency from the calls, and opens a pull request with the winning switch and a comparison table. That is the cheapest way to find out whether a workload is even in the region where a smaller model competes, before any fine-tuning is considered.

For a fine-tuned candidate, Converra's workflow curates reviewed production runs into versioned datasets with protected splits, records the training job and its gate result against a dataset version, holds the production switch behind an approval, and reports a parity verdict on real traffic. Training runs on the customer's provider or infrastructure, not inside Converra. Converra does not use customer data to train any model without a written, tenant-exclusive election, and no open-weight production verdict for a customer agent has been published yet. Whether a given workload transfers is unobserved until it is measured on that agent's traffic.