Can Closed-to-Open Model Transfer Become a Repeatable Product Rather Than a Custom ML Project?

Someone exports conversations to a spreadsheet. Someone else labels a few hundred rows by hand, with rules that live in their head. A notebook trains a model. An evaluation is invented for this customer, run once, and the numbers go into a slide. The model is deployed by editing a config. Six weeks later nobody can say which examples produced the model, which version of the evaluation passed it, or whether it is still doing what the slide said.

Every one of those steps can be done well by a good team, and none of them repeats. The second customer starts from zero. The third customer gets a slightly different evaluation because the person who built the first one left. That is the signature of a project: the knowledge is in people and notebooks, not in artifacts and contracts.

A product pipeline produces a fixed chain of artifacts with immutable identities: a dataset version, a candidate model identifier, a gate result bound to that dataset version and candidate, an approval that names the candidate, a deployment marker, and a production verdict keyed to the deployment. Each one references the one before it. Any question about a model in production can be answered by walking the chain.

The chain is what makes the second customer cheap. The pipeline does not care whose runs it curates; it cares that the runs have provenance, the examples have admission decisions, and the gate has a frozen contract. When those are data rather than habits, the pipeline is the same for every tenant and the work per tenant is the data.

A project relies on judgment at every step because nothing else exists. A product moves as many decisions as possible into checks that produce the same answer every time: schema validity, required fields, leakage between splits, minimum family diversity, template caps, missing-data handling, and the arithmetic of the non-inferiority rule. Human and model judgment are reserved for the decisions that need them, which are the ones about what correct behavior is.

Deterministic checks are also what let a pipeline refuse. A dataset build that fails closed when a protected family appears in training, or a gate that returns invalid when the incumbent misses its competence bar, is doing the product's job. A project has no way to refuse; it has a person who notices, or does not.

The step that cannot be automated is the customer establishing what the agent should have done. A project handles this with unbounded labeling. A product handles it by clustering findings by mechanism, showing a representative with its evidence and a proposed correction, taking one scoped decision per cluster, and propagating that decision to members with a per-example check. The customer reviews dozens of clusters, not thousands of transcripts.

Then the product measures the step: reviewer time per decision, accepted-example yield per cluster, exception and reversal rates, and the share of examples admitted directly versus by propagation. Those numbers are how you know whether the review step scales. A project has no such numbers because the labeling was done once, by whoever was available.

Every agent handles three kinds of request: ones where the evidence supports an answer, ones where it partly does, and ones where there is none. A product tests all three separately for every tenant, because the failure modes differ and because the no-evidence condition is where fine-tuned models most often learn to invent. Part 18 covers the split. What is standard is the shape of the evaluation. What varies is the content.

The gate contract works the same way. The rule, the sizing method, the symmetric handling of missing data, the recall check on seeded hard cases, the mode checks for collapse: those are fixed. The margin, the outcome definition, the hard cases, and the approval belong to the customer, because they encode what the customer will accept. A product that hard-codes the margin is guessing on the customer's behalf. A project that reinvents the rule per customer is not a product.

The test for whether transfer is a product is what changes when a new customer arrives. If the answer is a configuration record naming the agent, the outcome definition, the margin, the base model, the provider, the review roles, and the approval policy, the pipeline is a product. If the answer includes a fork of the evaluation script, a new labeling guideline document, or a custom training notebook, it is a project with a shared template.

Per-tenant code accumulates. Three customers in, there are three gates that disagree in small ways, and a fix to one does not reach the others. Per-tenant configuration composes. A fix to the pipeline reaches every tenant at once, and every tenant's chain of artifacts records which pipeline version produced them.

A project ends when the model is deployed and the offline numbers look good. A product ends when the production measurement says parity verified, or says regressed and the switch rolls back, or says confounded and the window reruns. The verdict is the deliverable, because it is the only artifact that says whether the transfer worked for the customer rather than for the holdout.

The verdict also feeds the next iteration in a controlled way. Regressions become reviewed failures that enter the next dataset version through admission, not through a bulk import. Successes do not silently become training data. The loop closes through artifacts, which is what keeps it a loop rather than a series of one-off projects with the same name.

The first is the customer's intended behavior. No pipeline can know what correct means for a customer's agent; only the customer can. Bound it by making the decision cheap: representative cases, evidence attached, a proposed target, one decision per cluster. The second is the base model choice. Bound it by choosing from a short list on hard constraints and gating one candidate with one fallback, rather than running a bake-off per customer. Part 37 is that procedure.

The third is provider and legal terms: output-use restrictions, training input warranties, retention, base licenses. These do not productize; they are re-read per engagement. Bound them by making the review a checklist with a recorded outcome, so the engagement does not start until the checklist is complete and the record shows who reviewed which version of which terms.

Ask five questions of the last transfer. Can you name the dataset version the deployed model was trained on and list every example in it with its source and reviewer? Can you re-run the gate today against the same frozen holdout and get the same contract? Did the customer's review take a bounded, measured amount of time? Did anything in the pipeline change for this customer other than configuration and data? Is there a production verdict, and is it keyed to the deployment?

Five yeses is a product. Any no is the place the project is hiding. That is not a criticism of the team; it is where the next engineering effort belongs, because it is the step that will not repeat for the next customer.

Converra's model benchmarks are already a repeatable version of the first step: candidate models run against scenarios generated from the agent's own instruction, three runs per scenario with median scoring, real cost and latency measured from the calls, and the winning switch shipped as a pull request with a comparison table. Its fine-tuning workflow follows the same artifact discipline: reviewed production runs are curated into versioned datasets with protected splits and per-example admission decisions, each training job and gate result is recorded against a dataset version, the production switch waits for an explicit approval, and the model production test reports parity verified, regressed, confounded, or insufficient data with the measured cost delta.

The boundaries are the ones this article argues for. Training runs on the customer's provider or infrastructure, not on Converra. Customer data is never used for training without a written, tenant-exclusive election. And no open-weight production verdict for a customer agent has been published, which means the repeatability claim is a claim about the pipeline's shape, not yet about a series of customer outcomes.