How Do You Keep Endpoint Startup From Consuming an Experiment's Budget?

Fine-tuning a small candidate is cheap. Evaluating it is where the bill is. To score a candidate on a few hundred cases you need it running on a GPU, and an on-demand deployment bills per hour from the moment it is created, through weight loading and warmup, through every minute of idle, until it is deleted. At the $8.00 per hour listed for an on-demand H100 on the Fireworks pricing page as of 2026-09-23, an evaluation that should take ten minutes of inference can cost an hour of billing if the surrounding script is careless.

The failures are mundane. A missing API key discovered after the endpoint is up. A malformed evaluation file. A script that creates the deployment, crashes, and leaves it running overnight. Two evaluation passes that each create their own endpoint instead of sharing one. None of these is a modeling problem, and together they can cost more than training.

Run a pre-flight stage that touches nothing billable and fails on anything that would fail later. Check that the credentials authenticate, that the candidate model identity exists and is in a ready state, that the evaluation manifest parses, that every case has the inputs and pre-fetched tool results it needs, that the output directory is writable, and that the comparison arms are reachable. Only when pre-flight passes does the script create the deployment.

Make pre-flight a separate command with its own exit status so a person can run it alone. A pre-flight that is buried inside the deployment step is not a pre-flight; it is a warmup you pay for.

Stand the endpoint up once, run every evaluation call the experiment needs, then tear it down. If the candidate must run three times per case for a median, all three runs belong in the same window. If a diagnostic probe is needed, it runs in the same window as the main evaluation. Each additional window pays the startup and warmup cost again.

Plan the window before creating it: the list of calls, the expected count, the concurrency the endpoint can take, and the expected duration. A window with a known call list is one whose cost can be estimated in advance and whose overrun is visible as soon as it begins.

Set two hard caps per window: a maximum wall-clock duration and a maximum dollar amount computed from the hourly rate. The script checks both between batches of calls and stops the run when either is reached, records the partial results, and proceeds to teardown. A cap that only warns is a suggestion. A cap that stops the run and deletes the endpoint is a control.

Size the caps from the planned window plus a margin, not from the total budget. If the plan says fifteen minutes, a thirty-minute cap catches a runaway; a four-hour cap does not. Record the cap values in the experiment's manifest so an overrun can be compared to what was authorized.

Tearing down the endpoint is the step most likely to be skipped by an error path. Wrap the entire evaluation in a handler that runs teardown on success, on failure, and on interruption. After issuing the delete, poll the provider until the deployment is reported gone and record that confirmation in the ledger. A delete request that was sent is not a delete that happened.

Where the provider supports scale-to-zero, use it as a second line of defense so an endpoint that escapes teardown stops billing when idle. Do not rely on it as the first line; some providers keep a minimum replica up unless deletion is explicit, and cold-start settings can keep capacity warm longer than expected.

A ledger records, for each window, when the endpoint was created, how many calls ran, when the delete was confirmed, the computed cost from the hourly rate, and the provider-reported cost once the invoice exists. Reconcile the two. A gap between computed and billed cost is either a leaked endpoint or a pricing assumption that was wrong, and both are worth knowing before the next experiment.

The ledger also makes authorization concrete. An experiment approved for a fixed amount has a line to compare against. Without it, the total is discovered on the monthly invoice, after the money is gone and the endpoints that spent it are forgotten.

A candidate is usually evaluated more than once: on a development split while the team iterates, and on a protected holdout once the gate contract is frozen. Run those in separate windows with separate ledgers. A development window can be relaxed, rerun, and probed. A protected window runs exactly the frozen call list once, under the frozen contract, and its ledger is part of the acceptance record.

Mixing the two invites the wrong economy. A team that opens protected cases during a development window to save a second startup has spent the holdout to save a few dollars of GPU time. Part 19 of this series covers why protected data has to stay protected; the budgeting consequence is that it costs one more window, and that window is worth paying for.

Where the provider offers per-token serverless inference for the candidate, evaluation can run without any deployment at all, and the startup problem disappears. The trade is that the serving configuration may differ from the one you will deploy, so the numbers may not transfer to production cost or latency. Use serverless for development iteration and a dedicated window that matches production for the final gate, or make sure the gate contract states which one it used.

Batch inference endpoints, where available, price queued work below on-demand rates and remove the idle problem for evaluations that can wait. They are a good fit for scoring a few hundred cases whose results are not needed within minutes.

Converra's fine-tuning workflow records the training job and its gate result against a specific dataset version and holds the production switch behind an explicit approval. The training and evaluation endpoints themselves run on the customer's provider or infrastructure; Converra does not create, run, or bill for them, so the controls above belong in the customer's evaluation scripts and ledgers.

What Converra does measure is cost from real calls when it benchmarks candidate models on scenarios built from the agent's own instruction, and, after a switch, the cost delta per arm on real traffic alongside the parity verdict. No open-weight production verdict for a customer agent has been published yet; an evaluation window that stayed within its cap is a well-run experiment, not evidence about production.