How to Tell Whether Your AI Voice Agent Is Actually Working

Research date: 23 September 2026. Last updated: 23 September 2026.

No vendor in this market publishes a containment rate, so the only way to know whether your deployment works is to measure it yourself. The complication is that the measurement the platform hands you is produced by the same kind of model that handled the call, scoring itself against objectives it inferred from your prompt.

This page sets out what is worth measuring, which numbers are safe to trust, and the one metric that catches the failure every other metric hides.

Brilo AI publishes this page and sells a voice agent. The section on what Brilo does not give you is held to the same standard as everything else here.

How we compiled this

Documented research, read from vendor technical documentation on 23 September 2026. No product was instrumented and no call data was analysed, so nothing here reports observed performance. What it does report is what the platforms say they measure, and what follows from that.

The built-in score is the system marking its own homework

Platforms do give you post-call analysis, and it is genuinely useful. It is just not independent.

Vapi produces three things after a call: a summary in call.analysis.summary, structured data extraction in call.analysis.structuredData against a JSON schema you define, and a success evaluation in call.analysis.successEvaluation which "determine[s] if the call was successful based on the objectives inferred from the system prompt", scored against a rubric you choose — NumericScale, Checklist, PassFail and several others. Source: docs.vapi.ai/assistants/call-analysis, read 23 September 2026.

Retell AI offers post-call extraction in four category types — text, selector, boolean and number — delivered to the dashboard, a webhook fired when the call ends, or the Get Call API. Its documentation describes the categories but does not define how a call is judged successful. Source: docs.retellai.com/features/post-call-analysis, read 23 September 2026.

Read that Vapi phrasing carefully, because it is doing a lot of work. The objectives are inferred from the system prompt — not stated by you, and not checked against what the caller wanted. A language model reads a transcript produced by a language model and decides whether the goal it guessed was met.

That is a reasonable triage tool and a poor evidence base. Use it to sort thousands of calls into a queue. Do not use it to tell your finance director the deployment works.

Containment is the most gamed number in the category

Containment usually means "resolved without a human". In practice it is almost always implemented as "not transferred to a human", and those are not the same thing.

A caller who gets a wrong answer and hangs up was not transferred. A caller who gives up in frustration was not transferred. A caller who is told the thing they need cannot be done, when it could, was not transferred. All three are counted as successes by a containment metric, and all three are the failures you most need to find.

This is why a containment rate can climb while the business gets worse, and it is the single most important thing to understand before you set a target on it.

The four things actually worth measuring

1. Repeat-caller rate — the honest containment proxy.

The number of callers who ring back within 24 to 72 hours about the same thing. If the agent genuinely resolved it, they do not call again.

This is the most useful metric in this page for three reasons. It catches exactly the failure containment hides: the caller who was not transferred and also was not helped. It needs no cooperation from your vendor, because it comes from your own telephony records — caller ID and timestamp, which you already have. And it cannot be gamed by prompt changes, because it measures what the caller did next rather than what the system thought it achieved.

Measure it before you deploy, so you have a baseline. A repeat rate that is flat or falling after deployment is real evidence. A containment rate of 80% with a rising repeat rate means the agent is efficiently failing.

2. Escalation quality, in both directions.

Two failures, and they need separate counts. Escalating what it should have handled wastes the investment. Not escalating what it should have is the one that causes harm — the distressed caller, the compliance-relevant question, the edge case answered confidently and wrongly.

Also measure whether the human receives context. An escalation where the agent hands over a summary and the caller does not repeat themselves is a different product from one where the call is simply transferred. No vendor publishes a metric for this; you can only get it by listening.

3. Fabrication rate.

How often the agent states something untrue. This cannot be automated away, because detecting a wrong answer requires knowing the right one. It needs a person reading transcripts against your actual policies, prices and availability.

It is also the risk that scales worst. A human agent who invents an answer does it on their own calls; a misconfigured voice agent does it identically on every call, which is the double edge of the consistency benefit.

4. Turn-taking failures.

Count interruptions, dead air, and calls where the caller and agent talk over each other. These are configuration problems — endpointing thresholds, turn detection — rather than model problems, and they are fixable once you know they are happening. They also correlate with abandonment, because a caller cut off twice tends to hang up.

How to sample when you have every transcript

Traditional call centre QA samples a handful of calls per agent because listening is expensive. A voice agent hands you 100% of transcripts, which sounds like it solves sampling and does not: nobody is reading ten thousand transcripts either.

The change is that you can now sample deliberately rather than randomly. Random sampling of mostly-successful calls mostly finds successes. Stratify instead, and weight toward where failures hide:

• Calls that ended early — under 30 seconds usually means something went wrong immediately

• Calls from repeat callers, which by definition contain an unresolved problem

• Calls that escalated, read for whether the escalation was right and whether context carried

• Calls the built-in evaluation marked successful but short, where the model's self-assessment is most likely to be wrong

• A small random control from everything else, so you can tell whether your strata are missing something

Twenty calls a week read properly against those strata will teach you more than a thousand read at random, and far more than any dashboard.

What to alert on, and what to review

Alert immediately on things that mean the system is broken: call failure or drop rate above baseline, latency above your threshold, and concurrency at its ceiling — because that one means callers are getting a busy signal and is invisible in the call records of calls that never connected.

Review weekly on things that mean the system is drifting: repeat-caller rate, escalation rate in both directions, and the sampled transcripts. These move slowly and a daily alert on them is noise.

The distinction matters because the common failure is inverting it — alerting on quality metrics that fluctuate, and reviewing capacity metrics monthly, by which time you have lost a month of calls.

Where tooling helps, and where it does not

A category of observability and evaluation tooling now exists for this, and it splits into things that work and things that are oversold.

Genuinely useful: transcript storage and search, so you can find every call that mentioned a term. Automated flagging against rules you define. Aggregate dashboards for latency and failure rates. Regression testing, where you replay scenarios after a prompt change to check you have not broken something that previously worked — this is the most underrated of the group, because prompt changes have non-local effects.

Oversold: automated quality scoring presented as ground truth. It is a model's opinion of a model's output. Useful for ranking a queue, not for proving a number.

Two practical cautions. Check whether analysis is metered separately from call volume — several products sell scoring on its own allowance, so "analyse 100% of calls" can mean 100% recorded and a capped number actually analysed. And check retention, because you cannot investigate a complaint against a transcript that has been deleted; Vapi's free tier, for instance, publishes 14-day retention. Source: vapi.ai/pricing, read 22 September 2026.

This page deliberately does not rank named observability vendors. Brilo does not sell in that category and reviewing it would be marking a market we are not in.

The part that still needs a person

Somebody has to own this — a few hours a month, indefinitely, reading transcripts and adjusting configuration. Not a project with an end date.

Without that owner, containment decays quietly: your business changes, your prices change, your opening hours change, and the agent keeps confidently giving the old answer. Nothing alerts you, because from the system's point of view nothing is failing. Deployments that go wrong slowly almost always go wrong this way, and the cost of the owner is small against the cost of not having one.

Brilo, and what it does not give you

Brilo provides call transcripts and records, and its free tier includes community support rather than a private channel. Source: brilo.ai/pricing, read 22 September 2026.

What Brilo does not publish, on the same standard applied above: no containment rate, no accuracy or fabrication rate, no recognition accuracy by accent, and no published detail on retention periods or transcript export on its pricing page. If a compliance review or an audit requires guarantees on retention and export, ask before you buy rather than assuming, and that applies to Brilo exactly as it applies to everyone else on this page.

Brilo is also a voice agent rather than a contact centre platform, so none of this covers monitoring a floor of human agents.