Voice AI Latency: What to Measure and How to Cut It
Latency decides whether callers talk to your agent or talk over it. What to measure, a methodology you can reproduce, where the milliseconds actually go, and nine changes that move the number.
On this page
Latency is the difference between a voice agent people talk to and a voice agent people talk over. It is also the metric most often quoted without a methodology, benchmarked in whichever configuration flatters the vendor, and optimised in the wrong place.
This article is about measuring it properly on your own deployment. We are deliberately not publishing a scoreboard of competitor numbers: any figure we produced for someone else’s platform would be unreproducible by you, and unreproducible benchmarks are marketing rather than engineering. What follows is the method, the anatomy, and the changes that actually move the number.
Why 300ms is the number
Conversation analysts have measured turn-taking gaps across many unrelated languages and found the same thing: the typical gap between one person finishing and the next starting is around 200 milliseconds. That is faster than it takes to plan a sentence, which means people begin formulating their reply before you have finished — and it means our tolerance for silence in conversation is calibrated extremely tightly.
| Response gap | How it reads to a caller |
|---|---|
| Under 300ms | Natural. Nobody notices the technology. |
| 300 – 500ms | Slightly considered. Reads as thoughtful rather than slow. |
| 500ms – 1s | Noticeably slow. Callers stay with it but the illusion is gone. |
| 1 – 2s | Callers start repeating themselves, assuming they were not heard. |
| Over 2s | Callers talk over the agent, or hang up. |
A fast agent giving a merely adequate answer is consistently rated better than a slow agent giving an excellent one. This is why the default model on Vociply agents is a fast one — on a phone call, the latency cost of a more capable model is usually a worse trade than the capability gain. See choosing a model.
Most people measure the wrong thing
The instinct is to measure model inference time — how long the language model takes to produce a reply. It is the easiest thing to measure and often a minority of what the caller experiences.
What the caller experiences is this, end to end:
Measure from caller stops speaking to caller hears the first syllable. Anything narrower measures a component rather than an experience.
Where the milliseconds go
| Stage | What it is | Typical contribution | Who controls it |
|---|---|---|---|
| Endpointing wait | The pause before the agent accepts that the caller has finished. | Frequently the largest single item | You. It is a setting. |
| Network to the caller | Telephony carriage between the caller and the platform. | Varies enormously by geography | Partly — through carrier and region choice. |
| Speech-to-text finalisation | Resolving provisional words into a final transcript. | Small, and overlapped with speech | Engine choice. |
| Model inference | Producing the reply. | Moderate, and grows with prompt and conversation length | Model choice and prompt length. |
| Text-to-speech first audio | Time until the first audio chunk exists. | Small when streaming, large when not | Voice engine. |
| Tool calls | Round trip to a calendar, CRM or your own webhook. | Whatever the other system takes | Mostly the other system. |
Prompt length and conversation length both increase model time on every turn, because the whole prompt and the whole conversation so far are reprocessed each time. An agent that felt quick in testing on turn two can feel sluggish on turn fifteen of a real call — and a 900-word prompt pays that tax on every single turn.
Endpointing: the invisible half
This is the part almost nobody measures and the part that most often ruins a deployment. Before the agent can respond it has to decide the caller has stopped — and a person pausing mid-thought sounds identical to a person who has finished, for the first several hundred milliseconds.
Every voice platform resolves this with a wait. Set it short and the agent interrupts people. Set it long and the agent feels slow. There is no setting that avoids both; you are choosing which failure your callers get.
On Vociply this is four separate timers rather than one, because the right wait depends on how the caller stopped:
| Timer | Default | Applies when |
|---|---|---|
| Wait seconds | 0.5s | The base pause after speech stops. |
| On punctuation | 0.2s | The caller clearly completed a sentence. Short is correct here. |
| On no punctuation | 1.5s | The caller trailed off mid-thought. This is the big one. |
| On number | 1.0s | The caller is partway through digits. |
That last one matters more than it looks. People read phone numbers and account numbers in bursts with gaps between them. A single global wait tuned for conversation will cut somebody off halfway through their own phone number, which is among the most infuriating things a system can do. Full detail on tuning these is in voice and conversation settings.
If your agent feels slow, drop the no-punctuation wait by 0.3s and re-test before you change models, providers or prompts. It is the cheapest 300 milliseconds available and most teams never look at it.
A methodology you can reproduce
Any benchmark worth reading states these six things. If a vendor’s numbers do not, you cannot reproduce them and should not weight them.
1. The script - exact caller utterances, word for word 2. The prompt - length in words, since it is reprocessed each turn 3. The model - and the speech-to-text and text-to-speech engines 4. Turn count - latency at turn 1 and at turn 10 are different numbers 5. The network - real PSTN call, or a websocket from a datacentre? 6. The percentile - P50, P95 or P99, and the sample size
Running it on your own agent
- Use a real phone call, not a browser test. Browser-to-server testing omits the telephony leg, which is exactly the part that varies most by market.
- Script the caller side. Same words, same order, every run. Improvised test calls produce unusable variance.
- Run at least 30 calls. Below that you cannot speak about P95 meaningfully.
- Measure at turn 1, turn 5 and turn 10. Reporting only the first turn hides the growth that callers actually feel.
- Record the calls and measure from the audio. The waveform is ground truth: caller’s last syllable to agent’s first syllable. Platform-reported timings measure the part the platform can see.
- Test at your busy hour. Latency under load is the number that describes your Monday morning.
Every Vociply call is recorded with a transcript in call history, so the audio you need for this is already there.
Reading your own numbers
| What you observe | What it means | Where to look first |
|---|---|---|
| P50 good, P95 bad | Something intermittent — usually tool calls or a slow downstream system. | Which turns are slow? If it is the ones after a question that needs a lookup, it is the integration, not the model. |
| Turn 1 fast, turn 10 slow | Context growth. Normal, but it can be reduced. | Prompt length. Move reference material out of it. |
| Uniformly slow, every turn | A fixed cost. Almost always endpointing. | The no-punctuation wait. |
| Fast, but it talks over people | Endpointing tuned too aggressively. | Raise the wait. You are on the other side of the same trade. |
| Slow only for some callers | Geography, or line quality. | Compare by country. Also check whether noise suppression is on. |
| Fine in testing, slow in production | Load, or a longer real prompt than the tested one. | Measure at your peak hour with the live agent. |
Nine ways to cut latency
In rough order of how much they return for the effort.
| # | Change | Typical gain |
|---|---|---|
| 1 | Lower the no-punctuation endpointing wait by 0.2–0.4s | Large. Usually the single biggest win available. |
| 2 | Shorten the system prompt; move reference material to a knowledge base | Large on long calls, compounding every turn. |
| 3 | Use a fast model rather than the most capable one | Moderate and consistent. |
| 4 | Remove integrations the agent does not need | Small per turn, but every attached tool is one more thing weighed on every turn. |
| 5 | Make your own webhooks fast, or make them asynchronous | Whatever your endpoint was costing. |
| 6 | Turn off noise suppression if you do not have a noise problem | Small, on every turn. |
| 7 | Cut the greeting down | Does not change latency, changes the first impression — which is what people are actually judging. |
| 8 | Tell the agent to narrate lookups | Zero milliseconds, large perceived gain. See below. |
| 9 | Use a local number and local routing | Material in markets far from your infrastructure. |
A long prompt costs latency on every turn and makes the agent follow instructions less reliably, because the important rules are diluted. Moving your price list and policies into a knowledge base — searched only when a caller asks something that needs it — usually improves speed and accuracy together.
Tool calls and the honest pause
When an agent checks a real calendar or looks a customer up in a CRM, that is a genuine round trip to another system. It adds real time, and no amount of platform optimisation removes it — the calendar takes what the calendar takes.
The fix is not technical. It is telling the agent to say something first:
Before checking availability or looking anything up, say a short line like "let me check that for you" - then make the call.
A narrated wait is enormously more tolerable than an unexplained one. Two seconds of silence feels broken; two seconds after “let me check that for you” feels like competence. This costs nothing and is the highest-return latency change most deployments never make.
It also sets the standard for your own endpoints. If you are calling your own systems through a webhook tool or through n8n, aim for under two seconds, and never chain six slow API calls into a single in-call workflow.
Geography, and why it bites emerging markets
Audio has to physically travel. A call between a caller in Nairobi and infrastructure in Virginia crosses an ocean twice per turn, and no model choice compensates for that.
This is systematically under-discussed, because most voice AI benchmarking is done by teams whose callers and servers are on the same continent. If you are deploying in Kenya, Nigeria or anywhere else distant from the major cloud regions, the same platform with the same settings will measure worse than the published figures — and the gap is not the platform’s model.
- Use a local number. It affects routing as well as answer rates — see phone numbers.
- Benchmark from the market you serve. Testing from your own office in another country tells you nothing about your callers.
- Budget more endpointing tolerance on poor lines. Noisy, lossy audio makes the end of speech genuinely harder to detect, and a slightly longer wait beats cutting people off.
- Ask about regional infrastructure if latency is critical at volume. It is one of the things dedicated Enterprise deployments exist to solve.
When lower is not better
Latency is a means, not the goal. The goal is a call that works. Three cases where pushing the number down makes the experience worse:
- Callers reading out numbers. Aggressive endpointing cuts them off mid-digit. Raise the on-number wait and accept the slower feel.
- Elderly, unwell or distressed callers. They pause more, and being interrupted is far more upsetting than waiting. A healthcare line should be tuned slower than a B2B qualification line, deliberately.
- Anything requiring a real lookup. An agent that answers instantly about your availability without checking the calendar is fast and wrong. Checking is worth the second, every time.
Latency is a proxy. What you are really optimising is whether callers complete the thing they rang about. Track that — bookings made, leads qualified, questions answered without a transfer — and treat latency as one of the inputs. An agent with a 400ms P95 that nobody books through is not a success.
If you want to see the settings discussed here in the product, they are all on the agent’s voice step and documented in voice and conversation. The fastest way to get a number for your own deployment is to build an agent and call it, then measure the recording.
Frequently asked questions
What is a good latency for an AI voice agent?
Under about 500ms from the caller finishing to the agent starting to speak feels conversational. Between 500ms and 1 second feels slightly slow but acceptable. Beyond about 1.2 seconds callers start talking again over the top of the agent, because they assume it did not hear them. Human conversation has a typical gap of roughly 200ms, which is why anything much beyond half a second reads as hesitation.
Why does my voice agent pause before answering?
Usually it is not thinking time at all — it is endpointing, the wait before the agent decides you have finished speaking. That is a configurable setting rather than a model limitation. Long system prompts are the second cause, because the entire prompt is reprocessed on every turn. Tool calls are the third: if the agent is checking a real calendar or CRM, that lookup is genuine round-trip time to another system.
Should I trust vendor latency benchmarks?
Only if they publish the methodology — the exact prompt, the model, the number of turns, the network path and the percentile. A number without those is unreproducible, and every vendor benchmarks in the configuration that flatters them. Run your own test on your own script; it takes an afternoon and the result is the only one that describes your deployment.
Does a faster model always mean a faster agent?
No, and this is the most common misconception. Model inference is often a minority of perceived latency. Endpointing wait, network round trips, text-to-speech time to first audio and any tool calls frequently add up to more. Switching models while leaving a 1.5 second endpointing wait in place changes very little.
What is P95 latency and why does it matter more than average?
P95 is the value that 95% of responses come in under. It matters more than the average because callers do not experience averages — they experience the bad turns. An agent averaging 400ms with a P95 of 1.8 seconds feels unreliable, because roughly one turn in twenty stalls noticeably, and one stall is enough to make somebody talk over it.
Does noise suppression add latency?
Yes, a little. Filtering the caller audio before transcription costs processing time on every turn, and aggressive filtering can clip quiet speech. Turn it on because you have a genuine noise problem — callers phoning from streets, markets or vehicles — rather than as a precaution.