Can the Open Model Handle Tool Calls, Long Context, and Multi-Turn Conversations?

Teams tend to treat these as one question, because a closed frontier model handles all three well enough that they never had to think about them separately. An open model in the 8B to 30B range forces the separation. Tool calling is a formatting and decision problem: emit the right structure, pick the right tool, pass valid arguments, stop when done. Long context is a capacity problem: quality degrades before the token limit, and memory cost rises with every token. Multi-turn is a state problem: what the model carries forward, what it forgets, and what it repeats.

Each one is testable in isolation, and each one has to be, because a failure in one looks like a failure in another from the outside. An agent that re-asks a question the user already answered might have lost multi-turn state, or it might have run out of usable context, or it might have failed to read the tool result that contained the answer. The transcript alone will not tell you which. The test plan below is built to tell them apart.

A closed provider exposes tool calling as a clean API: you pass schemas, you get back structured calls. With an open model, that structure is produced by a chat template that the serving stack applies, and the model has to have been trained on that template's tool format. A model served through a stack whose template does not match its training will emit tool calls as plain text, wrap them in the wrong tags, or ignore the schemas entirely. This looks like a model capability failure and is actually a configuration failure.

Before evaluating anything about tool choice, confirm the plumbing: the same request through the same serving stack should produce a parseable tool call for a trivially obvious case. Managed fine-tuning providers accept function-calling datasets in the OpenAI-compatible chat format, and Fireworks documents this on its supervised fine-tuning page as reviewed on 2026-09-23. That means the tool format can be taught. It also means a mismatch between the training template and the serving template will undo the teaching.

Once tool calls parse, four behaviors remain. Tool choice: given the request, does the model call the tool that the incumbent calls, or a tool that would also work? Arguments: are required fields present, typed correctly, and drawn from the conversation rather than invented? Error handling: when a tool returns an error, an empty result, or a result that contradicts the request, does the model recover, retry sensibly, or report the gap instead of fabricating a value? Stopping: does the model stop calling tools once it has what it needs, or loop?

Replaying tool-using conversations requires the tool results to be fixed in advance. If the open model calls a live tool during evaluation, it sees different data than the incumbent saw and the comparison is no longer paired. Pre-fetch the tool results for every evaluation case and serve them from a fixture keyed on the call the model makes. When the model makes a call the fixture does not cover, record that as an off-script call and score it separately. That count is itself a useful signal about tool choice.

A model card lists a context window. That number is the point at which the model refuses input. Quality declines well before it, and the decline is uneven: facts placed in the middle of a long prompt are recovered less reliably than facts at the start or the end, and the effect grows with length. An agent whose system instruction, retrieved documents, tool results, and conversation history together reach a large share of the window is operating in the region where this matters.

So the test is not whether the model accepts the agent's longest prompt. It is whether the model's quality at the agent's 95th-percentile prompt length matches its quality at the median. Bucket the evaluation cases by total context length, score each bucket separately, and look for the length at which the open model's score diverges from the incumbent's. If that length is inside the agent's real distribution, either the context has to shrink through retrieval or summarization, or the model is not a candidate for this workload.

Every token in the context occupies key-value cache memory on the serving GPU for the duration of the request. That memory is shared across every concurrent request. A model that serves forty short requests at once may serve eight long ones, and the ninth waits. The closed provider absorbed this; a self-hosted or dedicated open endpoint does not. Long context therefore couples directly to throughput, which is why part 40 treats latency and throughput together.

Load-test at the agent's worst-case length and its production concurrency, not one or the other. The failure to look for is not a wrong answer but a slow one: time to first token that climbs as the queue fills, or a request that fails outright when the cache is exhausted. Both are invisible in a single-request evaluation and both are routine in production.

A multi-turn agent has to use information from early turns in late turns, honor constraints the user stated once, and avoid asking again for what it already has. Smaller models fail this in characteristic ways. They re-ask a question after the answer scrolled out of the region they attend to well. They drop a constraint stated in turn two when producing turn nine. They repeat a clarifying question because the earlier answer was in a tool result rather than in a user message.

Test with full-length conversations that plant a fact early and require it late, and score the late turn on whether the fact was used. Include conversations where the fact arrives through a tool result, and conversations where the user changes a constraint mid-way, because a model that holds state well can still hold the wrong version of it. Measure per-turn latency as well; a conversation that gets slower every turn because the whole history is re-encoded is a serving decision, not a model limit, and it has a fix.

Of the three capabilities, tool calling responds best to fine-tuning. The format is learnable, tool choice on the agent's own tools is learnable, and the habit of reporting an empty result rather than inventing one is learnable if the training examples show it. Multi-turn habits such as not re-asking answered questions also improve when the examples are full conversations rather than isolated turns.

Long-context quality is the exception. Fine-tuning on a modest number of long examples does not extend the region of the context the model uses well; that is a property of pretraining and architecture. If the agent needs a longer usable context than the base model provides, choose a different base, not a longer training run. Part 37 covers choosing the base against the workload's real shape.

Write the rule before running the test. For tool calling: parse rate at or near the incumbent's, tool-choice agreement within an agreed margin on paired cases, zero fabricated argument values on seeded error cases. For long context: no score divergence from the incumbent inside the agent's real length distribution, and p95 latency within budget at production concurrency. For multi-turn: state-retention checks pass at the same rate as the incumbent on full-length conversations.

These are floors, and they are agent-specific. An agent that never exceeds a few thousand tokens does not need the long-context test at all, and an agent with no tools skips the first block entirely. The point of separating the capabilities is that each agent tests only the ones it depends on, and tests those at the shape it actually depends on them in.

Converra benchmarks candidate models against scenarios generated from the agent's own instruction, runs every model on every scenario three times, scores by median, and measures cost and latency from the real calls. Its tool-calling analysis scores how each agent uses its tools from actual call evidence: whether the right tool was chosen, whether the arguments were valid, and whether a needed tool was missed. A winning model switch ships as a reviewed pull request with the comparison table attached.

Converra does not run model training, and no open-weight production verdict for a customer agent has been published. The capability tests in this article are the customer's to run at their agent's real shape; Converra's model production test and model-swap verdict then report, on live traffic, whether the switch held parity at lower cost, regressed, was confounded, or has insufficient data to say.