How Do You Choose the Right Open-Weight Model for an Agent's Actual Workload?

A public leaderboard measures a fixed set of tasks under a fixed harness. Your agent's workload is a specific distribution of requests, with specific tools returning specific shapes, under a specific instruction, at a specific latency budget. A model that leads on general reasoning can be the wrong choice for a support agent that needs reliable tool calls at low latency, and a mid-table model can be exactly right.

So the first artifact of selection is a workload profile, written down: the request types and their share, the median and worst-case context length, which tools are called and how often, the output schema, the languages in the traffic, and the p95 latency the customer actually holds you to. Every constraint below is checked against this profile, not against a general impression of what the agent does.

Measure the rendered prompt length across a sample of production runs: instruction plus history plus retrieved documents plus tool results. Take the p99, not the mean, because the long cases are the ones that fail. Then check that the candidate's context window covers it with headroom for growth, and that the model was trained to use that window rather than merely accept it. Many models accept long inputs and quietly degrade past a fraction of the advertised length.

Long context also costs throughput. A model that fits your p99 in principle may not serve it at your latency target on the hardware you plan to use. Part 38 goes into how to test long context, tool calls, and multi-turn behavior on candidates; the selection step only needs the number and a candidate that clears it on paper.

Agents call tools, and the model has to emit a call the runtime can parse. Check whether the candidate's chat template has native support for tool definitions and tool-call outputs, or whether tool use would have to be taught entirely through fine-tuning. Native support is a large head start. Check the same for structured output: does the model reliably produce valid JSON against a schema, or does it need a constrained decoder at serving time?

These are testable before any fine-tuning. Send the candidate the agent's real tool definitions and a dozen real requests and count parse failures. A model that cannot emit a well-formed tool call untuned will need it taught from examples, and the examples will have to be far more numerous than a model that already knows the form.

Look at the actual language distribution in production traffic. Agents that were scoped as English-only routinely receive a few percent of requests in other languages, and the closed model handled them without anyone noticing. A candidate that is weak in those languages fails a slice of traffic that the workload profile might not have listed.

If the non-English share is small, decide whether it routes to the closed model rather than requiring the candidate to handle it. That is a routing decision, covered in parts 22 through 25, and it is cheaper than eliminating candidates over a slice you could escalate.

Licenses differ in what they require of a derived model. Qwen3's dense models are released under Apache 2.0, which permits commercial use and redistribution with attribution and notice requirements and no naming rule. The Llama 4 Community License permits commercial use but requires that a model built from the materials include Llama at the beginning of its name and display Built with Llama where the model is offered. Both are workable; they are not the same, and the obligation belongs to whoever owns the adapter, which is the customer.

Read the current license text for the exact base you intend to use, record the version you read, and put the obligations in the model manifest. Part 35 explains why ownership language has to reference them. A license constraint should eliminate a candidate early, before anyone has spent evaluation budget on it.

A candidate is only usable if you can fine-tune it and serve it where you need to. Check whether your chosen provider offers managed LoRA fine-tuning for the exact base and revision, and whether the fine-tuned result can be served on the deployment type you plan to use. Fireworks, for example, documents managed supervised fine-tuning to LoRA adapters with deployment to dedicated on-demand endpoints, and its model catalog lists which bases support it; the list changes, so it is a per-engagement check, not a stored fact.

Capture a provider receipt before training: the exact model identifier, its readiness state, its fine-tuning support flag, its context length, and the price. A base that is available today and withdrawn next quarter strands the adapter. If the customer's data cannot leave their cloud account, the same check applies to the in-account training path, which is a different product with different support.

Latency is a function of model size, hardware, concurrency, and output length. A 30B model may clear quality easily and miss the p95 bound at peak load, while a 9B model meets it with margin. Run the candidate on the intended serving hardware at the intended concurrency with realistic output lengths and measure the distribution, not the average. Part 40 covers the full measurement.

Size-class economics follow directly. Larger models need more or larger GPUs, which means more idle capacity at low volume and a higher break-even. The selection step needs a rough serving cost at expected volume for each size class, including idle time, to see whether the savings that motivated the transfer survive. Part 26 does that arithmetic.

Apply the constraints to the current catalog and keep what clears all of them. That is usually two to four models, often one small and one mid-sized from the same family. Benchmark those untuned on scenarios built from the agent's own instruction and traffic, scoring the same way the eventual gate will score, with real cost and latency measured from the calls. The untuned result tells you where each candidate starts, which is a much better predictor of where fine-tuning will land than any public number.

Then choose one candidate to fine-tune and name one sequential fallback, typically the next size up in the same family. Train and gate the first. Only if it fails its frozen development gate does the fallback get its own run. This is deliberately not a bake-off. Fine-tuning and gating ten models spends the budget on comparisons between candidates that were never going to be chosen, and it tempts the team to pick whichever happened to score best on a holdout that has now been looked at ten times.

When a candidate fails the gate, the reflex is to reach for a larger model. Check the failure mechanism first. A candidate that collapsed to a default answer because the training data had ten effectively distinct inputs will collapse at twice the size, because the cause was the data. A candidate that learned the output format but not the judgment behind it needs different examples, not more parameters. Parts 13 through 16 describe these failures.

A larger model fixes capacity failures: the task needs reasoning or knowledge the small model does not have, and the untuned benchmark already showed it lagging on exactly the hard cases. That is what the sequential fallback is for. Reaching for it on any other failure buys a more expensive model with the same problem.

Converra's model benchmarks are the untuned comparison step of this procedure: candidate models run against scenarios generated from the agent's own instruction across difficulty levels, every model runs every scenario three times with median scoring, and real cost and latency are measured from the calls. The winner ships as a GitHub pull request with the comparison table. For a fine-tuned candidate, the fine-tuning workflow records the training job and its gate result against a versioned dataset, and the production switch waits for an explicit approval.

Training itself runs on the customer's provider or infrastructure; Converra does not run it. The provider receipt, the license review, and the throughput measurement on the customer's hardware are engagement steps rather than product output, and no open-weight production verdict for a customer agent has been published. Which base is right for a specific agent is unobserved until it is benchmarked on that agent's scenarios.