How to Fix Voice Agent Failures That Transcript Evals Miss

Start by preserving the caller audio or an approved recording reference, speech-to-text output, agent text, text-to-speech output, timestamps, turn boundaries, tool calls, tool results, transfer events, agent version, and a caller outcome. A transcript alone cannot show whether speech recognition lost an order number, whether the agent began talking before the caller finished, or how long the caller waited in silence.

LiveKit's audio simulation guidance makes this distinction explicit: audio-mode tests can expose endpointing, interruption handling, transcription, speech quality, and latency that text-mode tests cannot. ElevenLabs likewise separates full conversation simulation, next-reply checks, and tool-call tests. Use the narrowest evidence that can actually observe the reported failure.

Use a pattern that a reviewer could identify again. For example: 'Among calls where a caller supplies a ten-digit account number after a verification request, the agent sends a different value to the lookup tool.' The eligible denominator is calls with that request and a usable recording or transcript. The numerator is calls where the captured value differs from the caller's spoken value under the stated review rule.

Keep incomplete recordings, dropped calls, and uncertain transcriptions visible rather than silently calling them passes. A generic rate such as 'bad calls' mixes speech recognition, turn-taking, policy, tool, and customer-side causes into a number that cannot guide one fix.

A voice-agent candidate might adjust endpointing, an interruption threshold, a pronunciation dictionary, a confirmation step, a tool contract, or a prompt instruction. Pick one diagnosed layer and keep the model, voice, tools, routing, and policy fixed unless the evidence says one of those is the cause. Changing the voice, prompt, and transfer workflow together can make a later improvement impossible to attribute.

For an account-number failure, a bounded candidate might add an explicit repeat-back and confirmation before the lookup tool call. For a false interruption, it might adjust only the turn-detection setting. Those are different candidates with different primary measures, even if callers describe both as a frustrating conversation.

Run the current agent and the candidate on the same failure-family and known-good scenarios. Include varied speaking pace, pauses, interruptions, names, amounts, confirmation codes, accents or noise only where those conditions are relevant and permitted. Preserve the same tools and test data so a tool response cannot masquerade as a speech improvement.

Score the candidate on the named failure, task completion, required transfer or escalation behavior, and the regressions that matter for that workflow. A voice agent that stops interrupting but becomes slow enough for callers to abandon has not produced a clean improvement. Repeat probabilistic scenarios when the behavior varies, but do not turn a single successful call into a reliability claim.

Record the approved version, release time, changed layer, and overlapping changes such as a model, speech provider, routing, or telephony update. On new eligible calls, apply the same failure definition and review rule. Compare a comparable pre-change and post-change window, and inspect the known-good behavior the candidate was required to retain.

If the pattern persists, the result is not fixed. If a speech-provider or routing change overlaps the comparison, the result is confounded. If there are too few eligible calls or too much missing audio to interpret, retain insufficient evidence. These are useful outcomes because they preserve the next diagnosis instead of crediting a release with a result it did not establish.

A coding agent can summarize the saved transcript, tool sequence, and release evidence for one pattern, identify missing fields, and prepare a bounded configuration candidate. It should preserve the failure definition, denominator, version, and unresolved cases. It should not declare a new voice or prompt verified because a transcript looks better.

Keep raw audio and timing evidence in the voice stack: Converra's current integrations accept text conversations. With Converra MCP, start from the saved account and agent inventory, inspect the diagnosed text and tool failure, run the authorized candidate comparison, then inspect deployment-impact and verification evidence after the team applies an approved change. State-changing actions remain subject to the connected account's scopes and the user's authority.