Always-on VAD or push-to-talk? Build for noisy Indian streets

The horn wins.

Your agent is midway through “I found your booking” when a bus leans on its horn. The voice activity detector calls it human speech, cancels the reply, and waits. The caller hears half a sentence and then nothing.

“Hello?”

That is not an exotic audio lab case in India. It is a Tuesday near a junction.

The ugly part is what follows: one false sound event cancels speech, the cancellation throws away a model response, the caller’s “hello” becomes a fresh turn, and a retry may fire the same booking tool again while everybody is still staring at a VAD graph.

Teams often frame the fix as always-on voice versus push-to-talk. Wrong fight. The real question is how much uncertainty your interface hides from the caller. Always-on can feel effortless in a quiet room. Push-to-talk can be wonderfully honest beside a market, on a construction site, or inside a moving auto.

Voxeval would support both and switch by evidence, not fashion.

Three detectors are probably arguing

In a typical voice stack, “the VAD” is not one thing.

The device may run acoustic echo cancellation and noise suppression. Your speech provider marks speech start, partial words, and end of turn. The application then decides whether a detected sound should interrupt TTS, cancel an LLM request, or merely be logged as possible overlap.

Those decisions need separate names.

Scroll diagram horizontally on smaller screens.

A horn can trigger acoustic energy without producing a transcript. Background speech can produce real words that do not belong to the caller. Echo can look exactly like the agent’s own voice returning through a cheap speakerphone.

If every signal maps directly to “barge-in,” the call becomes twitchy.

Flux gives you knobs, not answers

Deepgram’s current Flux documentation exposes eot_threshold, eager_eot_threshold, and eot_timeout_ms. Its quickstart lists a default end-of-turn threshold of 0.7, an eager threshold range starting at 0.3, and a default hard timeout of 5000 milliseconds. It also recommends 80 millisecond audio chunks for the stated setup.

Useful. Still not a preset for Chandni Chowk.

Higher end-of-turn confidence can reduce early cutoffs while adding wait. An eager threshold can begin response work sooner, but a false early guess wastes LLM calls and may start speech over the caller. A hard timeout protects the system from waiting forever, yet five seconds feels enormous in a live support call.

Sweep these values on your own recordings. Do not copy the example configuration into production and call it tuning.

The interruption policy needs memory

Suppose the caller makes a soft “hmm” while the agent reads a delivery date. Is that a request to stop? Maybe not.

Suppose they say “No, Thursday.” That absolutely is.

Use at least two interruption stages:

• Possible overlap. Lower the TTS volume a little or keep listening without canceling the response.
• The caller produced meaningful speech, so stop playback and preserve where the agent was interrupted.
• Noise-only burst? Ignore it, but keep the event for later scoring.

The Twilio Conversation Relay updates announced in April 2026 include adjustable interruption sensitivity, aimed at tuning behavior in noisy backgrounds. Provider controls help. Your dialog policy still decides what to do after the event.

And remember the canceled response. If the agent was reading three appointment options and the caller interrupted after option two, resuming from the beginning is irritating. Resuming from a fabricated location is worse. Track tokens played or use a simpler semantic checkpoint.

A fictional call outside Dadar station

This is an invented scenario for explanation, not a claim about a real deployment.

A caller is trying to reschedule a diagnostic appointment. The ambient track contains platform announcements, horns, and a friend saying, “Tuesday kar lo.” The caller says, “No, Friday morning.”

What belongs to the caller?

Speaker separation may help, but phone calls are often mono and the friend is near the same microphone. So the agent should treat conflicting date phrases as ambiguity, not as a confidence contest. “I heard Tuesday and Friday. Which date should I use?”

That repair is a product decision. No VAD threshold can invent it for you.

Push-to-talk is not a defeat

On a web or mobile experience, expose a press-and-hold option when the system detects repeated false speech starts, poor signal, or several failed repairs. Tell the person why: “There’s a lot of background sound. Hold the button while you speak.”

Clear. No shame.

For phone calls, the equivalent may be DTMF confirmation, a secure SMS link, or a human transfer. The microphone cannot become push-to-talk on an ordinary PSTN call, but the interaction can still become more explicit.

Use always-on where it earns its place. Quiet home calls, accessible hands-free flows, and experienced repeat users may benefit. In a warehouse with alarms and forklifts? Give the worker control.

Score the bad moments

Average latency hides the failures people remember.

Track false barge-in rate by noise type. Count missed genuine interruptions. Measure how often the system cuts off the final word, especially after Indian English discourse markers and Hindi particles. Capture the time from real end of speech to first agent audio. Then record canceled-response cost and repeat requests.

The 2026 τ-Voice benchmark adds controllable audio environments and turn-taking behavior to grounded tasks. Its results show a drop from clean to realistic conditions, with a large share of failures tied to agent behavior under the benchmark setup. That is the useful lesson here. Audio damage does not stay in the audio layer. It changes tool choices, confirmation, and policy behavior.

Build the noise set yourself

Use recorded room tones and legally sourced environmental audio, then mix them at several signal-to-noise levels. Add device echo, packet loss, clipped first syllables, and second speakers. Keep the clean source so you can compare.

Your set should include sudden horns, steady fan noise, market babble, station announcements, television speech, and silence that is not actually silence because an auto-rickshaw engine is idling underneath it.

But do not turn one Mumbai street recording into “India noise.” Hyderabad traffic, a rural pump, a call-center floor, and a kitchen pressure cooker create different errors.

Our gate would be unforgiving

Voxeval would block launch if a noise burst can authorize a tool action, if real caller corrections are repeatedly ignored, or if the agent has no usable recovery after several false barge-ins.

We would also block an always-on design whose p95 end-of-turn wait feels fine only because early cutoffs are missing from the latency chart.

Take the agent outside. Play the horn track. Interrupt it on purpose.