How Many Simulated Conversations Do You Need? Stop Looking for a Magic Number.

A sample size only means something relative to a question. Twenty conversations might reproduce a broken tool path. They cannot establish that a one-percent failure is absent. One thousand near-duplicate conversations can add less evidence than a smaller set that covers the actual failure mechanisms.

The mistake is asking whether 20, 100, or 1,000 is enough before defining the decision. A useful plan starts with what the evidence will authorize, how costly a wrong decision would be, and which assumptions make the calculation credible.

One evaluation episode is one complete, decision-relevant attempt under a declared starting state and outcome contract. For a conversational agent, that is usually a full multi-turn conversation. Messages, tool calls, turns, and complete conversations are not interchangeable observations.

Keep the episode definition identical in planning, execution, scoring, and reporting. Otherwise a claim about 100 tests can quietly become a mixture of 100 messages, a few complete workflows, and several repeated attempts sharing the same state.

Exploration can stop after it produces a useful hypothesis. A baseline-versus-challenger comparison needs enough matched evidence to separate a worthwhile lift from noise. Regression protection needs deliberate coverage of important behaviors, including failures that random sampling may never encounter. Production verification requires fresh production evidence after deployment.

A larger simulation run does not cross the boundary into production proof. It can select a candidate under the tested conditions. It cannot establish that production traffic, tool state, model behavior, latency, or user response will match the simulator.

For a binary outcome under a simple independent-sampling approximation, a planning formula is n = z² × p × (1 − p) ÷ E². Here p is the expected failure rate, E is the desired half-width in absolute percentage points, and z represents the confidence level.

At 95 percent confidence with a five-point margin, the approximation gives 139 observations when the planning failure rate is 10 percent. With no credible planning rate, using p = 0.5 gives the conservative value 385. These are worked examples, not recommended AI-agent thresholds.

The calculation assumes an appropriate sample and independent observations. Repeated synthetic conversations often share personas, scenarios, seeds, model state, or upstream dependencies. That correlation reduces the effective information in the nominal count.

If a failure independently occurs with probability p, the chance of seeing it at least once in n trials is 1 − (1 − p)ⁿ. The arithmetic explains why random volume is a poor safety strategy.

Even these counts can be optimistic when the simulator does not reproduce the failure mechanism or the episodes are correlated. If an event is important but rare, construct a targeted scenario and make it a required coverage stratum. Zero observed failures means none were captured in that batch—not that the true rate is zero.

When possible, run both versions against the same persona, scenario, initial state, and tool conditions. Each baseline result then has a natural challenger partner. Analyze the paired differences instead of comparing two unrelated piles of conversations.

A normal-approximation planning formula for a mean paired difference is n = ((z₁₋α⁄₂ + z₁₋β) × s_d ÷ δ)². If the standard deviation of paired differences is 10 points, the smallest worthwhile lift is five points, alpha is 0.05, and power is 0.80, the approximation gives 32 pairs.

Change the variance, target lift, or error tolerance and the answer changes immediately. Binary paired outcomes, small samples, repeated scenarios, and clustered observations may require a different method or statistical review. Raw conversation totals and unmatched runs do not repair a weak comparison.

Twenty complete episodes can validate the harness, expose an obvious failure, estimate variance for a later plan, or show whether a scenario actually elicits the intended behavior. The number becomes misleading only when an exploratory batch is relabeled as decisive after the result looks attractive.

A single confirmed hard-safety regression can still stop a candidate. That does not mean one observation estimates prevalence. It means the release policy declares some failures unacceptable regardless of average lift.

1. Complete the planned number of valid matched pairs. 2. Complete every required coverage stratum. 3. Reject pairs that violate the declared validity contract, not pairs with inconvenient outcomes. 4. Require the effect to clear the practical threshold. 5. Apply regression and safety gates separately. 6. Report unresolved evidence as inconclusive.

Do not stop at the first favorable checkpoint unless the testing method and error budget were designed for sequential decisions. Repeatedly checking and stopping when the chart looks good increases false positives.

More simulations add little when they repeat the same easy scenario, share the same state, or omit the slices that drive uncertainty. Improve the design before multiplying the count.

Converra does not treat a fixed number of simulations as proof that an agent is ready. Each test should be sized around the decision, matched personas and scenarios, risk slices, target effect, and stop rule. If the evidence remains sparse or unstable, the run expands. If a critical slice has no coverage, aggregate volume cannot rescue it.

Converra compares candidates and baselines on the same personas and scenarios, then checks regressions before deployment. After deployment, production evidence must still determine whether the targeted failure improved. Simulation selects a candidate; production closes the change.