The cost of conversational silence: latency budgets for Indian voice agents

“Hello? Hello?”

Your agent was thinking for 1.2 seconds. The caller thought the line had died.

Long enough.

That scene is fictional, and 1.2 seconds is not a universal impatience threshold for Indian callers. There is no magic national number. A Bengaluru developer on fibre, a shop owner calling over a noisy prepaid connection in Surat, and a worried parent booking a hospital appointment will tolerate silence differently.

Still, dead air costs trust. Teams feel it in demos, then track only average API response time and wonder why production calls contain so many repeats.

The pause may have come from a patient turn detector, a cold model connection, a CRM query in another region, speech generation waiting for punctuation, or several tiny buffers that look harmless on separate charts and feel awful when the caller experiences them as one unbroken silence.

Voxeval’s opinion: latency is a conversation outcome, not a single infrastructure metric. Budget every part of the turn, measure the tail, and test what the agent does while a slow step is happening.

The silence has owners

A caller finishes speaking. Several clocks start or keep running:

• When did the system decide the turn was over?

• Speech recognition may still be finalizing the last words, especially an address or code.

• The model reads context and plans a response.

• A tool call checks a booking, payment, or policy.

• Text-to-speech needs enough output to begin audio.

• Telephony buffers and carries it back to the phone.

If your dashboard rolls all of that into “response time,” nobody knows what to fix.

Track at least four timestamps: end of caller speech, detected end of turn, response decision ready, and first audible agent audio. Add tool start and finish when a tool sits on the path.

Then split by call route, language, workflow, and audio condition. One fast median can hide miserable tail calls.

Average latency is the polite lie

Suppose 90 calls answer in 500 milliseconds and 10 answer in 4 seconds. The average sounds tolerable in a slide. Those 10 callers are saying “are you there?” over the agent’s late reply, which triggers another interruption and makes the call slower again.

Look at p50, p90, p95, and p99. More important, listen to samples around each band.

A high p95 can come from a cold model connection, a slow CRM query, packet loss, a turn detector waiting too long, or text-to-speech buffering a full sentence. The remedy depends on the cause.

At Voxeval, we would also measure:

• False starts: the agent begins, stops, then restarts.

• How often did the caller repeat words during the gap?

• Overlap. First audio arrives after the caller has begun a new turn.

• “Hello” count?

• Useful first audio: not just a filler sound, but the first audio that helps the caller understand what is happening.

That last one matters. A system can game time-to-first-audio by saying “Sure” instantly, then leaving a three-second crater.

Detection delay is still delay

Turn detection makes a trade. Wait longer and you are less likely to cut off a slow speaker. Answer earlier and the conversation feels quick, but you may treat a mid-sentence pause as the end.

Deepgram’s Flux exposes end-of-turn thresholds and timeout settings. Similar controls exist in other stacks. Do not copy one setting across every lane.

A caller spelling an IFSC needs room between characters. A restaurant caller saying “yes” does not. A Hindi speaker may pause before switching into an English product name. A customer in a loud market may produce noise that keeps the detector open.

Build different profiles for different jobs, or let the conversation state choose from a small, tested set. Keep the selection logic observable. If every slow turn is marked “model latency,” the turn detector gets away clean.

A budget has to add up

Here is a fictional target for a simple account-balance question. It is an example, not an industry standard.

Segment	Example p95 budget
End-of-turn decision	350 ms
Final transcript and routing	180 ms
Model to first response token	320 ms
Speech generation to first audio	220 ms
Telephony delivery buffer	130 ms
Total first useful audio	1,200 ms

Now compare a booking lookup. The tool may need another second. Do you keep the caller in silence? Probably not.

Say something honest and short: “I’m checking Saturday slots.” Start the lookup in parallel only when the workflow makes that safe. Do not make the filler sound like success. “Done” before the tool result is a nasty little lie.

Streaming helps. So does saying less.

Recent realtime voice work has pushed model and audio latency down. Streaming input, early response generation, cached context, warm connections, and incremental speech output can all help.

But teams sometimes burn the gain with a 180-word system prompt repeated on every step, giant tool descriptions, three sequential model calls, and an opening sentence that sounds like a call-centre handbook.

Cut the path.

Keep stable instructions cached where the platform supports it. Send the smallest tool schema that still protects the job. Parallelize independent reads. Preload likely customer context after authentication. Let text-to-speech begin on a clause boundary instead of waiting for a speech.

And edit the copy. “Let me check that” reaches the caller faster than “I would be happy to assist you with checking the current status of that request.” It also sounds less strange.

Do not buy speed with errors

An agent that answers before it understands the address is not fast. It is wrong sooner.

τ-Voice reported a marked performance drop between clean and realistic audio conditions across tested agents. That is a warning against tuning only on studio speech. Low latency under packet loss, background noise, and interruption has to preserve task behavior.

Test these pairs together:

• first-audio latency and correct end-of-turn detection;

• tool response time and final business state;

• fast barge-in recovery, but did the agent keep the caller’s correction?

• speech streaming and pronunciation of the entity at the sentence boundary;

• retries. Did they duplicate the action?

No latency win gets credit if the verified outcome drops past its release threshold.

India is not one network condition

Do not write “Indian mobile network” as a single test setting. Sample the routes your customers use.

You may find different behavior across operators, regions, handset ages, Wi-Fi calling, 4G and 5G coverage, or international termination. Measure lawful, consented production traces in aggregate, and recreate representative delay, jitter, loss, and codec conditions in test.

Language can interact with the stack too. Code-switched turns may finalize later. Names and addresses may trigger retries. A text-to-speech voice may start quickly in English and buffer longer in another language. Slice it.

The metric that makes people move

Give every slow segment an owner and every key workflow a latency envelope. The voice latency budget template has columns for call condition, p50, p95, first useful audio, repeats, overlap, outcome pass rate, and the team responsible.

Fill it with traces, not guesses.

Then sit with headphones and play the p95 calls. The silence will tell you who owns Monday morning.