Prompt CI/CD That Ends in a Production Verdict, Not a Green Check

A CI job that runs your eval suite against a prompt change answers one question: does the candidate pass the cases in the suite? That is necessary. It catches regressions on behaviors you have already written down, and it stops the obvious mistakes before they ship.

It is silent on two other questions. Is the candidate better than the prompt that is running now, on the same inputs? And after it ships, did the failure it was meant to fix stop happening for real users? Both need a different kind of evidence than a pass count. The guide to why passing LLM evals fails in production covers the first gap; this guide covers the pipeline that closes both.

Put the system prompt, tool descriptions, and the configuration that shapes behavior (model, temperature, guardrail thresholds, routing rules) under version control, with one change per pull request and a description that names the failure it targets. A change that touches five behaviors cannot be attributed to any one of them when the measurement comes back.

Record the version that is actually serving traffic, not only the version in the repository. Slow rollouts and cached prompts put conversations on the wrong side of a release unless the serving version is logged with each conversation.

The regression suite is the CI stage most teams already have. Make sure it is built from the right material: scenarios derived from production conversations that failed, generalized into families, and kept forever once added. A suite written from imagination before launch covers what you anticipated; the failures that cost you are the ones you did not. The guide to building a regression suite from production traces covers construction and maintenance.

Run it on every candidate, and fail the pipeline on any regression against a scenario the suite has previously passed. That is the one place a hard fail is right.

Passing the suite is not the same as being better. Add a stage that runs the current prompt and the candidate on the same simulated multi-turn conversations, scores both with the same rules, and compares them per persona. The candidate advances only if its lift over the baseline is positive on the conversations both ran; averages across unrelated runs do not count.

Converra runs this as paired simulation with synthetic personas derived from production patterns, and only paired comparisons count as evidence. How many conversations the stage needs depends on the effect size; see how many simulated conversations agent testing needs.

Review first. In Converra, deployment is review-first by default: the winning change ships as a GitHub pull request or as a direct update that a person applies. Teams can turn on automatic deployment; it stays off until they do, and when it is on a change must pass the regression suite before it ships.

Whichever path you use, write down exactly what shipped and when. That deployment marker is what lets the next stage separate conversations before the change from conversations after it. Keep the previous version deployable so a regressed result has a fast remedy.

This is the stage a green check cannot replace. After the merge, count the targeted failure in production conversations with the same detector you used for the baseline, wait for enough conversations on each side, and check whether anything else changed the prompt in the window.

Converra measures pull requests your team merges by comparing seven days before the merge with seven days after, with at least 10 diagnosed conversations per side, and treats a difference smaller than about two standard errors as flat. The table shows the outcomes it reports as of October 2026.

A verdict is only useful if it changes what happens next. Improving or verified: the scenario that motivated the change stays in the regression suite as protection. Flat or not fixed: reopen the diagnosis, because the cause was probably in a layer the change did not touch. Confounded: ship the next change on its own so it can be measured. Regressed: revert to the previous version and treat the new failure as an incident; the incident response runbook covers that path.

Report the verdict where the team already looks: on the pull request, next to the test results. A merged PR with 'improving on 420 conversations' or 'confounded by PR #214' under it is a different artifact from one with a green check and silence.

The table contrasts what a typical CI gate proves at each stage with what the verdict-ending pipeline adds.

The numbers are illustrative, not customer data. A team opens a pull request that changes one instruction so the agent confirms the order number before quoting a delivery date. The regression suite of 160 scenarios passes. In the head-to-head stage the candidate removes the targeted behavior on the 12 scenarios in that family and matches the current prompt elsewhere. A reviewer approves; the merge is marked at 14:10 UTC and the serving version is logged.

Seven days before the merge, the detector found the pattern in 44 of 610 diagnosed conversations (7.2%). Seven days after, 11 of 590 (1.9%). The drop is 5.3 points, well over two standard errors, and no other merged change touched the prompt. The pipeline reports improving. Two weeks later a second PR edits the same prompt for an unrelated reason inside a new measurement window; that later result is reported confounded, which is the correct answer, not a failure of the method.

The one published Converra example is the Salespeak case study, where unsupported pricing, VAT, and infrastructure claims did not recur on live traffic after a deployed fix and routing failures fell 74% after a second deploy. Limits: one customer, one agent, and percentages rather than counts.