When Should a Product Recommend Fine-Tuning, Routing, Retrieval, or a Prompt Change?

The four fixes look interchangeable from a distance because each can change what an agent says. They are not interchangeable, because each acts on a different layer. A prompt change alters the instruction the model reads. Retrieval alters the evidence in the context. Routing alters which model reads the context. Fine-tuning alters how a model responds to a context. A diagnosis has to say which layer the failure lives in before a recommendation makes sense.

The cost of getting this wrong is not just wasted effort. Applying the wrong intervention can hide the failure. A fine-tune trained on outputs that were wrong for lack of evidence produces the same wrong outputs with more confidence. A prompt change that tries to compensate for a knowledge gap grows a paragraph of caveats and still fails on the next unseen fact.

Start from a failing run and ask one question at a time. Did the model have the evidence it needed in context? If not, the cause is a knowledge gap. Did the instruction say what to do in this situation, unambiguously? If not, the cause is an instruction gap. Did the model have the evidence and the instruction and still produce the wrong output? Then the cause is capability, and the question becomes whether this request is harder than most.

A team that skips this step recommends by habit. Prompt engineers recommend prompts, ML engineers recommend training, platform engineers recommend routing. The table below maps observable signals in a run to the cause and to the intervention that addresses it, so the recommendation follows from the evidence rather than from who is looking.

A prompt change is the right recommendation when the run shows the model lacked a clear instruction for the situation. It is also the cheapest intervention to test: generate the variant, run it against the same scenarios as the baseline, compare pairwise, ship the winner. Because it is cheap and reversible, it is always the first thing to try when the diagnosis is ambiguous between an instruction gap and something else.

Its limit is that it cannot add facts and cannot make a model more capable. A prompt that grows every time a new failure appears is a signal that the team is using instruction to paper over knowledge or difficulty gaps. When the instruction is already clear and the model still fails, stop adding rules and move down the list.

If the correct answer depended on a fact the context did not contain, no amount of instruction or training will reliably produce it. The fix is to make the fact arrive: a retrieval step over the knowledge base, a lookup tool, a structured field in the request. Recommend this whenever the failing runs show the model asserting something it could not have known from context, whether the assertion was right by luck or wrong.

Retrieval has its own failure modes, which is why it is a separate recommendation and not a default. The wrong passage retrieved is a new kind of failure. A knowledge base that is stale produces confident stale answers. Evaluate retrieval on its own, with recall over a labeled set of question-to-passage pairs, before evaluating the agent's answers over it. And note that adding retrieval to an agent changes its context length and cost, which may change the routing and fine-tuning calculus later.

Routing is the right recommendation when the evidence and instruction are present and the difference between success and failure is which model is answering. The signal is a paired comparison: the same request succeeds on the strong model and fails on the weak one. If that pattern is concentrated in an identifiable subset of requests, a router can send that subset to the strong model and the rest to the cheap one.

Routing is a partial answer that is honest about its partiality. It does not claim the cheap model matches the expensive one everywhere; it claims the router can tell the difference before the answer is produced. Its cost is the router's own error rate and the operational surface of running two models. Parts 22 through 25 treat routing design in full, including when full replacement is better and whether a policy generalizes.

Fine-tuning is the right recommendation when the task is stable, the correct behavior is well represented in reviewed examples, the volume is high enough that per-token cost matters, and the failures are about consistency rather than knowledge. The signal is a task the strong model already does well, repeated thousands of times a day, where a smaller model could do the same if it behaved more reliably.

It is last on the list because it is the slowest to change and the easiest to get quietly wrong. A rule change that takes one sentence in a prompt takes a new dataset version, a training run, a gate, and an approved switch in a fine-tune. And a fine-tune inherits whatever the training examples contained, including the closed model's mistakes, which part 7 addresses. Recommend it when the other three have been ruled out or already applied, not as a first resort.

The four interventions are ordered by how fast they can be reversed and how much they change. A prompt variant can be rolled back in a minute. A retrieval index can be rebuilt in an hour. A routing policy can be tightened in a day. A fine-tuned model has to be retrained. Trying the reversible fixes first means the team learns what the failure actually is before committing to the least reversible one.

The order also protects the fine-tuning data. Fixing instruction and knowledge gaps first means the production runs that later become training examples were produced by an agent that already had the right rules and the right evidence. Training on runs from before those fixes teaches the model the old failures. The reviewed-example pipeline in part 6 depends on this ordering.

Most real transfers combine at least two. A knowledge gap fixed with retrieval often exposes an instruction gap about how to use the retrieved passage. A routing design almost always needs a prompt change that tells the cheap model when to escalate. A fine-tune for a stable task frequently sits behind a router that sends the long tail elsewhere. The recommendation should name the combination and the order in which to apply it.

What a product should not do is recommend all four at once. Combining interventions in one release makes the result impossible to attribute: if quality improved, which change did it? Apply one, measure, apply the next. Part 39 covers evaluating the whole agent when only one component changes, and that discipline applies to every step here.

Given a diagnosed failure cluster, ask in order. Is the correct answer derivable from the context? If not, recommend retrieval or a tool. Is the instruction clear for this case? If not, recommend a prompt change and test it against the baseline. Does the strong model succeed where the cheap one fails on this cluster? Recommend routing with escalation. Is the task stable, high volume, and already handled well by the strong model? Recommend a fine-tune, on reviewed examples, behind a gate.

Each recommendation should carry its evidence: the runs that show the cause, the intervention's expected effect on that cluster, and the measurement that would confirm it. A recommendation without a confirming measurement is a guess with better formatting.

Converra's loop is built around the first of these interventions: it diagnoses the step where a run failed, generates a targeted prompt variant, tests it against the baseline in paired simulation, deploys the winner, and reports whether the fix held on real traffic as verified, not fixed, or confounded. Its model benchmarks cover the routing and replacement question at the catalog level, running candidate models on scenarios from the agent's own instruction and opening a pull request with the winning switch.

For fine-tuning, Converra's workflow curates reviewed production runs into versioned datasets with protected splits, records the training job and gate result against a dataset version, holds the switch behind an approval, and reports a parity verdict on real traffic. Training runs on the customer's provider. Converra does not use customer data to train any model without a written, tenant-exclusive election, and no open-weight production verdict for a customer agent has been published yet. Which intervention a given failure needs is decided by that agent's evidence, not by this article.