Production A/B Testing

Run a live split test between your current prompt (baseline) and the simulation-winning variant — on real traffic, before fully committing.

When to use it

After an optimization completes and selects a winner, you have two options:

Option	When
Direct deploy	High-confidence simulation result, low-risk change
Production A/B test	You want real-traffic confirmation before 100% rollout

A production test routes a slice of real traffic to the variant, scores both arms automatically, and tells you when it has enough evidence to decide.

Prerequisites

• Optimization is completed with a winning variant selected
• SDK is installed and configured in proxy mode — production tests require proxy mode to intercept and swap prompts per session
• The prompt has been matched through the SDK at least once with the current content — Converra validates a content hash on start and rejects stale matches
• Each conversation must supply an opaque session ID (a hashed token, not raw user-identifying text) — the assignment service throws SESSION_ID_REQUIRED for user traffic without one

Starting a test

POST /api/v1/optimizations/{optimizationId}/start-production-test

{
  "variantId": "variant_abc123",
  "trafficAllocation": 10,
  "variantPercentage": 50
}

All fields are optional. Defaults are trafficAllocation: 10, variantPercentage: 50, and the optimization's winning variant.

Traffic parameters

Parameter	Range	Default	What it controls
trafficAllocation	1–100	10	% of total traffic included in the test
variantPercentage	1–99	50	Of included traffic, % routed to the variant

A trafficAllocation: 10 with variantPercentage: 50 means 5% of all traffic sees the variant, 5% sees the baseline, and 90% is unaffected.

Response

{
  "success": true,
  "data": {
    "productionTest": {
      "_id": "683404a1c2e8f1d3a4b5c6d7",
      "status": "running",
      "config": {
        "trafficAllocation": 10,
        "variantPercentage": 50
      },
      "startedAt": "2026-05-14T10:00:00Z"
    }
  },
  "metadata": { "processingTime": 84 }
}

Monitoring the test

GET /api/v1/production-tests/{testId}

While running, Converra scores each arm as conversations come in and watches for statistical confidence. When it has enough data, it sets status to decided (variant won) or inconclusive (no clear winner). Both states require you to take action.

Status lifecycle

running → decided      → promoted
        → inconclusive → rolled_back
        → rolled_back  (manual abort at any point)

Status	Meaning	Next action
running	Test is live, data accumulating	Wait
decided	Variant won — ready to promote	Call promote
inconclusive	No clear winner found	Call rollback to restore baseline
promoted	Variant deployed to 100%	Done
rolled_back	Baseline restored	Done

When a test becomes inconclusive, traffic automatically returns to baseline — the assignment service only routes to tests with status: running. You can still call /rollback to formally close the record.

Acting on the decision

Promote the variant

When status is decided:

POST /api/v1/production-tests/{testId}/promote

Deploys the variant to 100% of traffic and closes the test.

Roll back

To abort at any point — including when the test is running, decided, or inconclusive:

POST /api/v1/production-tests/{testId}/rollback

Restores baseline immediately and closes the test. You cannot roll back a test that has already been promoted.

Attention flags

When attentionReason is set on a running test, something needs your review:

Reason	What it means
prompt_drift	Your baseline prompt changed after the test started — results may not be comparable
assignment_disabled	Session assignment is not working; traffic split is not running correctly
low_session_cardinality	Not enough unique sessions to get reliable results
scoring_backlog	Scoring is behind — decision may take longer than expected

Webhooks

Subscribe to production test events via webhooks:

Event	Fired when
production_test.started	Test goes live
production_test.decision_ready	Decision is ready (decided or inconclusive)
production_test.promoted	Variant promoted to 100%
production_test.rolled_back	Test rolled back

SDK: how traffic assignment works

The SDK reads the active variant for each session via:

GET /api/v1/sdk/prompts/{promptId}/active-variant

It returns either baseline or variant based on the session assignment. Your prompt is transparently swapped before the LLM call — no code changes needed.

If the SDK isn't installed, production testing is unavailable. See the SDK integration guide.

Related

• Understanding Results — how the simulation winner is selected
• Regression Testing — pre-ship safety checks
• Deployments API — verify impact after promotion