The Press 1 for Hindi era is dead: build fluid Hinglish voice AI

“Press 1 for Hindi.”

Then the caller presses nothing and says, “Mera refund abhi tak nahi aaya, can you check once?” The menu wanted a language. The person gave you a problem.

That little mismatch is where a lot of India voice deployments start wobbling. Not because the speech model has never heard Hindi. Because the system was built around one language label per call, while the caller is switching mid-sentence, carrying the same intent across both languages, and expecting the agent to keep up without a tiny committee meeting.

Voxeval’s view is blunt: a Hindi-or-English router is now a product smell. Keep it as an escape hatch for accessibility or caller preference. Do not make it the brain of the conversation.

One call, several language states

A bilingual call does not move cleanly from hi-IN to en-IN and back. A caller may use Hindi grammar, English product names, digits in either language, and a proper noun that belongs to neither model’s tidy vocabulary.

So stop storing “call language” as one mutable string.

Store language evidence at three levels:

• Turn level. Useful, but slow to change.
• A switch point can sit inside four words: “payment kal reverse hoga.”
• Entity level? Absolutely. Keep the language guess, original audio span, raw transcript, and normalized value together.

That last part saves investigations. When do hazaar becomes 200 in the tool payload, a reviewer should be able to trace the conversion. If the agent later speaks “two hundred,” you can see exactly which boundary broke.

Scroll diagram horizontally on smaller screens.

Don’t bolt translation into the hot path

Here is the tempting build: detect language, transcribe, translate everything to English, ask the model, translate the answer to Hindi, then synthesize speech.

It works on a whiteboard. On a phone call it can feel like waiting for a lift that stops on every floor.

Each serial stage adds delay and another chance to flatten meaning. Brand names get translated. Address fragments move. The caller says kal, which needs conversation context, and an early translation commits to “tomorrow” before the rest of the turn arrives.

A better pipeline keeps the original mixed transcript in the model context. Normalize only the fields the business tool needs. If your chosen model needs a translated shadow copy, produce it beside the original rather than replacing it.

Deepgram’s current voice-agent documentation makes the engineering trade-off fairly plain. Flux offers model-integrated turn detection and a multilingual model with language hints. Nova exposes broader formatting and multilingual code-switching features. That is a provider capability statement, not a pass certificate for your calls. Run both on the same telephony audio and score the final workflow.

And do not switch the TTS voice on every detected English noun. Nobody wants the voice persona to change because the caller said “refund.” Use a stable bilingual voice where it works. If you must switch voices, wait for a meaningful phrase or turn boundary and test the seam by ear.

A fictional courier call, because this gets real fast

Picture an invented courier-support call in Gurugram. No real customer or production call is being described here.

The caller says:

“Bhaiya parcel Sector 46 jaana tha, but app mein Sector 40 show ho raha hai. Flat is B-two-zero-four.”

The useful output is not a pretty English transcript. It is a verified state change:

Boundary	Evidence to keep	Failure that matters
Speech	Audio span around 46, 40, and B-two-zero-four	Traffic clips the correction
Transcript	Original mixed words and timestamps	forty-six collapses to forty
Entity layer	Old sector, corrected sector, flat B-204	New and old values get merged
Tool call	Address-update arguments and policy result	Agent updates an ineligible shipment
Spoken reply	Masked confirmation in the caller’s chosen style	It reads back the wrong flat

Notice the two sector numbers. A system that extracts entities without correction order can store both, pick the first one, and still produce fluent speech. Fluent. Wrong.

Let the model hear uncertainty

Many pipelines clean interim transcripts too aggressively. They collapse alternatives, remove partial words, and hand the language model a sentence that looks far more certain than the recognizer ever was.

Keep uncertainty around risky entities. You do not need to dump every token probability into the prompt. A small contract is enough:

{
  "raw": "flat is B two zero four",
  "candidate": "B-204",
  "field": "flat_number",
  "confidence_lane": "confirm",
  "audio_span_ms": [8420, 10360]
}

The confirm lane is deterministic. The model can choose natural wording, but it cannot skip the check or silently invent a better-looking flat number.

Who picks the reply language?

Not the last noun.

If a caller says, “Refund kab aayega?” the word refund does not mean they want an English reply. Follow the language pattern of the whole turn, the caller’s explicit preference, and the vocabulary that makes the job easiest to understand.

Keep the choice sticky for a few turns. Otherwise the agent can wobble between formal Hindi and English every time a product name appears. That sounds less like a bilingual person and more like two bots sharing one phone.

A useful reply policy can stay small:

• Explicit request wins: “English mein batao.”

• The caller has used mostly Hindi grammar for three turns, so keep Hindi structure while leaving familiar product names alone.

• Safety or money language? Use the version your company has reviewed, then offer to repeat it in the other supported language.

• Unsure?

Ask once. “Hindi or English, whichever is easier for you?” Then remember the answer.

Do not translate names, addresses, plan labels, or legal phrases just to make the sentence look consistent. Consistency is not the goal. Understanding is.

Repairs tell you whether the agent was listening

Code-switching is often blamed when the real bug is correction handling.

The caller says, “Sector forty-six… nahi, forty-three.” Both numbers may be transcribed perfectly. The failure comes later when a summary keeps the first value, a tool payload takes the second, and the spoken confirmation somehow returns to the first.

Make corrections first-class events:

{
  "field": "sector",
  "old_candidate": "46",
  "new_candidate": "43",
  "evidence_turn": 18,
  "state": "awaiting_confirmation"
}

Do not erase the old candidate. Mark it replaced. That gives reviewers a trail and prevents a stale summary from bringing it back.

Now test the messy versions. The caller corrects the number in Hindi after saying it in English. They interrupt the read-back. A friend in the room offers a different value. The agent begins a tool call while the correction is still arriving.

Voxeval treats those as conversation-state tests, not speech-to-text tests. The recognizer can pass while the agent fails.

One state object, limited readers

A shared state does not mean every provider receives the whole call.

The speech service may need audio and language hints. The model needs the recent mixed-language turns and safe customer context. The address tool needs a normalized sector and flat number, not the caller’s full transcript. Text-to-speech needs the approved reply, not raw account data.

Split access by job. Keep provenance on the fields that cross boundaries.

This also makes model swaps less painful. When speech recognition, language reasoning, and tool validation have separate contracts, you can compare two recognizers on the same audio without changing how an address gets committed. You can try a new bilingual voice without giving it more customer data.

The fashionable phrase is “multilingual stack.” The practical version is a handful of boring field contracts that survive a language switch.

Latency hides at the switch point

The 2026 τ-Voice paper found a large gap between text-agent task completion and voice-agent performance, even in clean conditions. Its simulator also varies accent, audio conditions, and turn-taking. That does not measure Hinglish support directly. It does expose the trap: voice failures often show up in agent behavior, not only transcription.

Language switching can add latency in places your top-line timer misses:

• A language detector waits for more speech before committing.
• The TTS layer reloads a voice. Maybe 300 milliseconds. Maybe much worse on a cold path.
• What if the agent starts an English reply while the caller is still finishing a Hindi correction?

Measure switch-to-first-audio, not only end-of-turn-to-first-audio. Also measure false starts, canceled responses, and how much speech gets thrown away after a correction.

Build a test set that sounds like people

VoiceAgentBench includes English and six Indic languages with tool use, multi-turn tasks, and safety cases. Helpful. Still, its synthetic queries are not your caller mix, your prompts, or your backend.

Your private set should have full-turn switches, mid-phrase switches, English nouns inside Hindi syntax, Hindi repair phrases inside English turns, numbers spoken across both languages, names with several accepted spellings, and third-language intrusions that should trigger a safe recovery.

But vary the people too. Delhi Hindi is not the master template for India. Include first-language differences, age bands, phone routes, speaking pace, and the actual vocabulary of the job. A collections agent hears different language from a hospital scheduler.

Then stratify. One shiny “Hinglish accuracy” average can hide a 0% pass rate on mixed-language amounts.

The release gate we would use

At Voxeval, we would block a release if any high-risk Hinglish lane fails one of these checks:

• Wrong business action after a language switch.
• A correction leaves the stale value in the final tool payload, which is the sneaky one.
• Sensitive data gets read back without the approved mask.
• “Human please?” in either language does not transfer.
• Third-language speech is confidently treated as Hindi or English.

No weighted average should rescue those failures. The system either preserved the caller’s meaning through the switch or it did not.

Start with the Hinglish switch-point test cards. Replace the fictional delivery and payment cases with your own high-loss workflows. Record several speakers, keep the ugly phone routes, and score the tool state as carefully as the transcript.

Put the switch point in the test set before you put the agent on a phone line.