BlogSeries

Closed-to-open model transfer for production agents

48 partsUpdated September 23, 2026

Every team running an agent on a closed model eventually asks whether a smaller open-weight model could do the same job for less. The honest answer depends on the workload, the training data, the evaluation, and the production evidence, not on a benchmark score.

This series answers the questions in the order a team actually meets them: which workloads qualify, how to build training examples from real conversations, why fine-tunes fail in predictable ways, what parity means, how routing and economics work, and what has to be true on live traffic before anyone calls the transfer done.

Start with part 1

Parts 1 to 5

Should you do this at all

Whether a fine-tuned open model can replace the closed model, which workloads qualify, what transfers, and when a different fix is cheaper.

  1. 1Can a Fine-Tuned Open-Weight Model Replace the Closed Model Powering a Production AI Agent?Whether a fine-tuned open-weight model can replace the closed model behind a production agent depends on the workload, the training data, and the parity gate.
  2. 2Which Agent Workloads Are Good Candidates for Closed-to-Open Model Transfer?The workload traits that predict a successful move from a closed model to a fine-tuned open model: bounded scope, evidence in hand, volume, checkable outcomes.
  3. 3What Behavior Can You Transfer From a Closed-Model Agent, and What Gets Lost?Fine-tuning transfers behavior: format, rules, conventions, tool choice, and abstention. It does not transfer knowledge, reasoning depth, or the long tail.
  4. 4When Should a Product Recommend Fine-Tuning, Routing, Retrieval, or a Prompt Change?A decision rule for agent fixes: prompt change for rule gaps, retrieval for missing facts, routing for uneven difficulty, fine-tuning for stable work at volume.
  5. 5What Does a Customer Need to Provide Before Model Transfer Can Begin?Before a model transfer starts, a customer supplies the exact agent contract, evidence-carrying runs, a named reviewer, a parity definition, and data rights.

Parts 6 to 12

Training data

Turning real conversations into reviewed examples, handling incomplete or missing evidence, and preserving lineage.

  1. 6How Do You Turn Real Conversations Into Trustworthy Training Examples?From production runs to admitted training examples: judges propose findings, a customer reviewer decides on clusters, and each example is admitted with lineage.
  2. 7How Do You Teach an Open Model Without Teaching It the Closed Model's Hallucinations?Separate what the teacher asserted from what the evidence supports, admit only supported outputs, train abstention on purpose, and seed adverse recall cases.
  3. 8Can a Smaller Model Learn When the Evidence Is Incomplete?A smaller model can answer what partial evidence supports and mark the rest, if partial cases are labeled, balanced by family, and gated on their own slice.
  4. 9How Should an Open Model Respond When There Is No Evidence at All?With no evidence, an open model should abstain in a fixed shape: name the gap, what would resolve it, and hand off. Train the contract; measure both directions.
  5. 10How Many Reviewed Examples Does Customer-Specific Fine-Tuning Need?There is no universal minimum. Count distinct families, not rows, set a diversity floor per behavior and evidence condition, and let a frozen gate decide.
  6. 11Which Examples Matter More: Common Requests or Difficult Edge Cases?Both, in a deliberate ratio. Common requests anchor the model's default; seeded edge cases are the recall guard. Cap templates so no scenario dominates.
  7. 12How Do You Preserve the Source, Evidence, and Reviewer Lineage of Every Training Example?Give every artifact an immutable identity, from source run to decision to example to dataset version, and make revisions reassess dependents, never overwrite.

Parts 13 to 16

Why fine-tunes fail

Epochs, format learning, ties with the base model, and telling task learning from test-set memorization.

  1. 13Why Can More Training Epochs Make an Open Model Worse?More epochs on a small, templated dataset memorize the examples. Loss keeps falling while holdout behavior collapses to a default. Fix data first, then epochs.
  2. 14Why Can a Fine-Tuned Model Produce Perfect JSON but Worse Answers?A fine-tuned model learns output shape long before it learns judgment. Why schema validity is the wrong pass signal and how to score content separately.
  3. 15What Does It Mean When a Fine-Tune Ties Its Untuned Base Model?A fine-tune that ties its untuned base means the evaluation cannot see what training changed. Five causes, how to separate them, and when a tie is good news.
  4. 16How Do You Know Whether a Fine-Tune Learned the Task Rather Than the Test Set?Tell task learning from memorization with template holdouts, family-level splits, paraphrase probes, and a seen-versus-unseen gap frozen before scoring.

Parts 17 to 21

Evaluation and parity

What the evaluation set needs, how to separate evidence conditions, contamination, output repair, and the definition of parity.

  1. 17What Should an Evaluation Set Contain Before You Claim Closed-Model Parity?A parity claim needs a frozen evaluation set with all three evidence conditions, seeded adverse cases, held-out templates, a margin, and enough paired cases.
  2. 18How Do You Test Supported, Partial-Evidence, and No-Evidence Requests Separately?Label each evaluation case by evidence condition, define the correct behavior for each, score each as its own row per model, and gate on all three, not a blend.
  3. 19How Do You Prevent Training Data From Contaminating the Holdout?Split by family key before generation or review, dedupe by content hash, withhold whole templates, abort the build on any leak, and never reopen a holdout.
  4. 20How Do You Compare Two Models Without Output Repair Hiding Their Failures?Score raw outputs before repaired ones, record every repair as structured metadata, and apply one symmetric missing-data rule so neither model gets an edge.
  5. 21What Counts as Parity: Correct Answers, Useful Follow-Ups, Safe Behavior, or All Three?Parity is a bundle of pre-registered dimensions: correct answers, useful next actions, safe behavior on thin evidence, format, latency, cost, plus a veto.

Parts 22 to 25

Routing and escalation

Full replacement versus routing, when to escalate, how much traffic can move, and whether a policy generalizes.

  1. 22When Is Full Replacement Better Than Routing Between Open and Closed Models?Replace fully when the workload is narrow and the gate passes on every evidence condition. Route when a measurable slice fails. The escalation rate decides.
  2. 23When Should the Open Model Escalate a Request to the Closed Model?Escalate on signals available before inference: evidence condition, request class, context size, policy category. Escalate after inference only with a verifier.
  3. 24What Percentage of Calls Can Safely Move to the Open Model?The safe share is measured, not chosen: per-class gate results weighted by real traffic mix, capped by risk tolerance, ramped with a rollback rule.
  4. 25Can a Routing Policy Generalize Beyond the Customer Used to Develop It?Parts of a routing policy transfer: evidence detection, tool-failure signals, context size. Taxonomy and thresholds do not. Validate on a second customer first.

Parts 26 to 29

Economics

Real savings after hosting and fallbacks, break-even volume, and why training cost says nothing about serving cost.

  1. 26How Much Does Open-Weight Inference Save After Hosting, Idle Capacity, and Fallbacks?Open-weight savings are the closed-model bill minus dedicated capacity, idle time, autoscaling headroom, fallback traffic, and operations on the same traffic.
  2. 27At What Traffic Volume Does Fine-Tuning an Open Model Break Even?Fine-tuning an open model breaks even when cumulative per-request savings exceed the one-time setup cost plus the recurring fixed cost of dedicated capacity.
  3. 28Why Isn't a Cheap Training Run Proof of Cheap Production Inference?Training bills once per training token; inference bills per served token and per hour of capacity at production concurrency, for as long as the agent runs.
  4. 29How Do You Keep Endpoint Startup From Consuming an Experiment's Budget?On-demand endpoints bill from creation, including warmup and idle. Pre-flight everything, batch calls into one window, cap time and spend, and verify deletion.

Parts 30 to 36

Approval, monitoring, and ownership

What customers should see, how to approve and roll back, what proves it on production traffic, retraining, and who owns the model.

  1. 30What Should Customers See in a Closed-Versus-Open Model Comparison?A closed-versus-open comparison must show paired per-case results, evidence splits, seeded-case recall, measured cost and latency, and exact model identities.
  2. 31How Should Customers Approve, Monitor, and Roll Back a Model Replacement?Approve one named candidate against its gate result, monitor the same outcome across a preserved boundary, and rehearse rollback with the old model kept live.
  3. 32What Would Prove the Open Model Works on Real Production Traffic?Proof is the same outcome measured on comparable traffic before and after the switch, keyed on the model that served each run, ending in one of four verdicts.
  4. 33When Should You Retrain the Student, and When Should You Keep the Current Model?Retrain a fine-tuned student when reviewed evidence shows a new failure class, a changed contract, or measured drift. Never on a calendar or unreviewed traffic.
  5. 34What Happens When the Closed Teacher Model Changes?When the closed teacher changes, your labels change with it. Freeze the teacher version per dataset, re-check the gate baseline, never retrain automatically.
  6. 35Who Owns a Customer-Specific Fine-Tuned Model and Its Training Data?The customer should own the adapter and the training data from their runs, tenant-exclusive, under a written election covering export, deletion, and licenses.
  7. 36Can Closed-to-Open Model Transfer Become a Repeatable Product Rather Than a Custom ML Project?Model transfer becomes a product when every step is a versioned artifact under a standard contract and tenants differ only in configuration, never in code.

Parts 37 to 40

Model choice and capability

Picking the open model for the actual workload, tool calls and long context, whole-agent evaluation, and latency requirements.

  1. 37How Do You Choose the Right Open-Weight Model for an Agent's Actual Workload?Choose an open-weight model by the agent's hard constraints, benchmark a short list untuned on its own scenarios, fine-tune one candidate, keep one fallback.
  2. 38Can the Open Model Handle Tool Calls, Long Context, and Multi-Turn Conversations?Open-weight models can handle tool calls, long context, and multi-turn agents, but each capability must be tested at the agent's real shape, not the model card.
  3. 39How Do You Evaluate the Entire Agent When Only Its Underlying Model Changes?Freeze the prompt, tools, retrieval, and scoring, change only the model, replay the same scenarios through the full agent, and pair the results case by case.
  4. 40Does the Open Model Meet the Customer's Latency and Throughput Requirements?Decompose the latency budget, load-test at production concurrency with real prompt lengths, and compare p95 against the closed model's measured numbers.

Parts 41 to 44

Data cost, licensing, and serving

Teacher data cost, provider terms and distillation, self-hosting benefits, and quantization trade-offs.

  1. 41What Is the Cost of Producing Teacher-Generated Training Data?Teacher-generated training data costs frontier inference, verification calls, and reviewer time per accepted example, divided by yield. Review time dominates.
  2. 42How Do Licensing and Closed-Provider Terms Affect Model Distillation?Distillation runs under four sets of terms: the closed provider's output rules, the open base license, the fine-tuning host's terms, and customer data rights.
  3. 43Does Self-Hosting Offer Benefits Beyond Lower Inference Cost?Self-hosting an open model buys version pinning, data residency, latency control, dedicated capacity, adapter ownership, and auditability at an operations cost.
  4. 44What Changes When You Quantize or Otherwise Optimize the Open Model for Serving?Quantization and serving optimizations change the model you evaluated. What moves, what to re-measure, and why the served artifact needs its own gate result.

Parts 45 to 48

Running it in production

Drift detection, automatic fallback, proving savings per customer, and when the whole move is not worth it.

  1. 45How Do You Detect Quality Drift After Moving Traffic to the Open Model?Detect drift after moving traffic to an open model by measuring the gate's outcome on fresh traffic, keyed on the served model and split by request class.
  2. 46What Should Happen Automatically When the Open Model Fails or Times Out?When the open model times out or returns invalid output, the request fails over to the closed model inside the latency budget and is recorded as a fallback.
  3. 47How Do You Prove Savings per Customer Rather Than From a Theoretical Token-Price Comparison?Prove open-model savings by measuring what the same traffic cost on both models, per customer, at verified parity, with fallbacks and hosting included.
  4. 48When Is Moving to Open Weights Not Worth the Operational Complexity?Moving to open weights is not worth it at low volume, on workloads that need the closed model's knowledge, without a named owner, or without a trustworthy gate.

See what a model swap does to your own agent.

Converra benchmarks candidate models on scenarios built from your agent, ships the change as a reviewed pull request, and reports whether quality held on real traffic.

Start for freeHow model benchmarks work