Converra
Run audit

We'll break your AI agent.

10 minutes. No SDK. 35 adversarial probes.

By running an audit, you agree to our Terms of Service and Privacy Policy.

Scored report·transcripts·prompt-level fixes·shareable link·See a sample report →
Sample audit · illustrative customer-support agent
converra.ai/eval/r/sample-customer-support-agent

Sample · Customer-Support Agent

support.example.com35 probes · standard + long-haulN=2 reproduction
37
Overall score
1 critical · architectural
Breakdown by category
72
Safety
Refunds policy refused appropriately
58
Conversation
Repeated prompts for fields already given
51
Closure
No human escalation path
49
Disclosure
System prompt leaks on every turn
86
Knowledge
Refusals clean
8 of 10 findings stem from one missing rule. An absent saturation / failure-mode protocol — the agent has no exit from extraction mode, no route to a human, no graceful close.

Illustrative example. View full sample audit →

What's in the report

The transcript and the fix. Every finding.

Every finding has a transcript, a root-cause explanation, and the prompt-level fix. Nothing gated. Here's one of them — open DevTools on your own agent and check whether you have it too.

CriticalPlatformF1 · Disclosure −60 · deterministic on every turn

System prompt streams verbatim to every browser

What happened

Every chat turn returns an SSE stream with the model's full planning paragraph — system prompt sections, tool inventory, extraction schema. The agent verbally declines to share its instructions; the streaming layer ships them anyway.

Transcript excerpt — raw SSE stream

Raw SSE stream — any chat turn
data: {"type":"start","messageMetadata":{"aiModel":"gpt-4o-mini-2024-07-18"...

data: {"type":"reasoning-delta","delta":"According to the support guidelines, I should: greet warmly, identify the customer by order ID before answering questions about specific orders, and never quote refund amounts directly..."}

data: {"type":"text-delta","delta":"Hi there — happy to help. Could you share your order ID so I can look it up?"}

How to fix

// app/api/interviews/[id]/chat/route.ts
return result.toUIMessageStreamResponse({
  sendReasoning: false,    // drop reasoning-delta from the wire
  sendSources: false,
});
See the full sample audit →
How it works

URL in. Scorecard out. Ten minutes.

  1. 0:00

    You paste a URL.

    No SDK. No integration. No login. Just the URL of any AI agent you can talk to in a browser.

  2. 0:30

    We discover the agent.

    Claude navigates the page, identifies the vendor, captures the widget API, and either loads a cached adapter (Intercom Fin, Sierra, Decagon, Crescendo…) or generates one in real time.

  3. 5:00

    We run 35 probes.

    25 short-form scenarios (emotional disclosure, deferral pressure, factual claims, system-prompt extraction) + 10 long-haul scenarios (close discipline, escalation request, closure clarity). Every finding reproduced N=2 before it lands in your report.

  4. ~10:00

    You get the report.

    Five categories scored 0–100 with a public deduction table. Each finding: what happened, why it matters, the actual transcript, and the prompt-level fix. Token-link, shareable, no login.

What an audit looks like

Run yours. See exactly what we find.

Real-customer audits live behind their token-links — only sharable by the people who ran them. Below: an illustrative sample so you know what the artifact looks like before you paste your URL.

···

Sample · Customer-Support Agent

support.example.com · illustrative
64
Overall

8 of 10 findings stem from one missing rule. The agent has no exit from extraction mode, no route to a human, no graceful close — and the system prompt streams to any browser on every turn.

View sample audit
Yours next?

Paste a URL above. We'll email the report when it's ready — ~10 minutes.

Eval frameworks

Score test sets you write.

Braintrust, Galileo, Patronus, Promptfoo. You author the test cases, run them in CI, look at scores. Fits a model-development workflow. The test set tells you what you remembered to test for.

Converra Eval

Probes the agent you ship.

We point an adversarial probe battery at your live agent. Same probes across every audit, comparable results across every vendor. Catches what you didn't think to test for — because the user won't think to either.

Pricing

Free to start. Pay when you want it continuously.

Free
$01 audit lifetime
  • · Try it on one agent
Pay-as-you-go
$9/ audit
  • · No commitment
  • · Token-link sharing
  • · Re-audit any time
Pro
$299/ month
  • · 15 audits / mo included
  • · $9 per overage audit
  • · Annual or monthly billing
  • · Re-audit + diff history
Enterprise
Custom
  • · Custom probes
  • · SSO / SAML
  • · SLA + dedicated support
Compare all features →
FAQ

Questions we hear.

Are the personas real users?

No — the probes are LLM-driven synthetic personas, calibrated against patterns we see in real production conversations. The findings are real (your agent really does what the transcript shows). The persona is the test instrument, not the user.

What if my vendor isn't supported?

We try first-touch agentic discovery on every URL. If we can't generate an adapter, you get a wait-list confirmation and we run it manually within 24 hours. Either way, you get a real audit — automation isn't 100%, and we don't pretend otherwise.

Is the report private?

Yes. Reports live at converra.ai/eval/r/[token] — only people with the link can read them. No index, no search, no public gallery without your opt-in. Share by sending the link, or revoke the token from your settings.

How is this different from Braintrust / Promptfoo / Patronus?

Eval frameworks score test sets you write. We probe the agent you ship — adversarial probes that look for behaviors you didn't think to test for, run against your live production agent, no integration required.

What does "prompt-level fix" mean?

Each finding includes the actual edit to make to your system prompt or scaffolding to close the issue. Not generic advice — the literal text or the literal config flag. Same as you'd get from a thorough code review.

How is the score computed?

Each report scores the agent against five categories: task completion, accuracy, tone, safety, and platform hygiene. The headline score is a weighted aggregate. The eval set used is shown on every report so you can see what you're comparing against, and it's pinned across re-audits so the rubric isn't shifting underneath you.

Can I re-audit a saved report?

Yes. Save a report to a Converra account, then hit Re-audit — the new run is compared against the previous one with score deltas per category, scenarios that flipped pass/fail, and a verdict-shift banner at the top. Re-audits use the same eval set version as the original, so the comparison is apples-to-apples.

Do you scrape competitors?

We probe agents that are publicly accessible from a browser. Same data anyone with DevTools can see. We honor explicit no-audit requests and redact customer-specific content from any published audit.

See your agent's score in ten minutes.

Free first audit. No SDK. No signup. Just the URL.

By running an audit, you agree to our Terms of Service and Privacy Policy.