Beyond vibe-testing: regression gates for Indic vernacular voice agents

Three people from your office called the bot. It worked. Ship it?

No.

They probably used good Wi-Fi, knew what the bot expected, and spoke the kind of Hindi or English your team hears every day. That is a demo circle, not evidence for callers in Kochi, Guwahati, Jaipur, Srinagar, or a bus crawling through Bengaluru traffic.

The industry is finally publishing better voice-agent benchmarks. VoiceAgentBench covers more than 5,500 synthetic queries with tool use, multilingual tasks, and safety cases. EVA-Bench brings 213 scenarios, repeated trials, audio effects, and several ways to measure reliability. The 2026 Voice of India work evaluates unscripted telephone speech across 15 Indian languages and 139 regional clusters.

Good. Use them.

But Voxeval would never turn a public leaderboard into your release gate. Your agent has its own policy, tools, customers, telecom routes, and expensive ways to fail.

“Tamil passed” tells us almost nothing

Which Tamil? From whom? Over which phone route? With what task? Did the transcript look right while the tool call sent the wrong date?

Language labels are too broad for regression planning. Break coverage into strata that can expose a business risk.

• Chennai Tamil on a quiet Android handset.

• A caller from Madurai uses a regional pronunciation, then inserts an English product name and a six-digit PIN code.

• Old handset. Narrowband audio. Two corrections.

• What happens when the customer says the date in words?

• The tool times out after the agent says it is checking.

One case may belong to several strata. That is fine. The goal is not a perfect taxonomy. It is being able to say, “This release broke Malayalam address capture on narrowband calls,” instead of, “The average moved by 1.7 points.”

Public benchmarks are maps, not permission slips

VoiceAgentBench is useful because it moves beyond transcript accuracy into agent behavior, tool calls, multilingual queries, and adversarial safety. EVA-Bench adds repeated-trial thinking: pass once, pass at least once across attempts, and pass every time across attempts are very different bars. τ-Voice shows how sharply voice-agent performance can fall under realistic acoustic conditions.

Each paper also has boundaries. Synthetic queries are not your live customers. Bot-to-bot calls do not recreate every carrier codec, speaking habit, interruption, or emotional caller. A model that does well on general tasks may still mishandle your loan-status policy in Marathi.

Use public results to find test dimensions you forgot. Then rebuild those dimensions around your own workflow.

Start with the loss, not the language count

Imagine a fictional diagnostics chain with booking agents in six Indian languages. A Kannada caller asks to move a fasting blood test from Friday to Saturday. The bot hears the dates correctly, calls the reschedule tool, but drops the fasting_required instruction from the confirmation.

Word error rate may look fine. Task completion may even show a successful reschedule. The customer arrives after breakfast.

That workflow needs a verified outcome with several assertions:

• the original appointment was changed once;
• Saturday is the stored date;
• the fasting instruction remains attached;
• the caller heard it in the chosen language;
• the old slot is no longer active.

Now you have a test worth automating.

A matrix your team can maintain

For every release-sensitive workflow, capture at least these fields:

Caller slice. Language, region or accent proxy where lawfully collected, age band if relevant and permitted, device or route, and speaking pattern. Avoid turning accent labels into identity claims. They are test coverage tags, not truths about a person.

Audio condition. Clean, narrowband, packet loss, crowd noise, fan, television, echo, another speaker, or synthetic perturbation calibrated against real call samples.

Conversation behavior. Interruption, hesitation, code-switch, correction, topic change, refusal, repeated question, silence, anger, or a request for a human.

System event. Slow model, tool error, stale customer state, transfer failure, webhook replay, or policy denial.

Outcome proof. Database state, payment state, booking state, consent record, transfer receipt, and what the caller was told.

The downloadable Indic voice regression matrix includes fictional starter rows. Delete them after your team understands the shape. Real coverage should come from your call reasons and incident history.

Repeated trials. Because the model is not a toaster.

A single pass can flatter a variable system.

Suppose a safety-sensitive case passes nine times and fails once. A dashboard showing 90% may sound acceptable. It may be awful if the one failure disclosed account data to the wrong caller.

Use different repeated-trial views:

• pass@1: what happens on the first attempt?

• At least one success across retries can matter for a recoverable information request.

• Every run passes. That is closer to the bar for a hard policy boundary.

Do not average those ideas into one number. Label them.

For high-impact cases, Voxeval prefers a no-regression count and a minimum repeated-pass bar. The exact threshold belongs to the risk owner. A restaurant booking and a debt-collection disclosure should not share one release rule.

Human recordings, synthetic audio, and neither pretending to be the other

Synthetic speech is fast, cheap, and easy to vary. It is useful for broad regression. It can also sound far cleaner than a real uncle calling from a veranda with a ceiling fan, a television, and three relatives answering in the background.

Human recordings bring real pronunciation, timing, breath, self-correction, and hesitation. They also bring consent, privacy, sampling, labeling, and storage work.

Use both, with labels.

A practical stack might look like this:

1. Thousands of synthetic runs on every change.
2. A smaller consented human-audio set across target language and region slices.
3. Periodic shadow analysis of lawfully sampled production failures, redacted and governed.
4. Manual listening for the odd cases the metrics cannot explain.

Never pass synthetic audio off as evidence that real regional callers are covered. And never keep raw production audio forever because “we may test with it later.”

The gate needs teeth

A report that arrives after deployment is archaeology. A release gate runs before the change reaches callers.

For each candidate build:

• run the fixed regression set;
• compare against the approved baseline by stratum;
• block on any hard-policy failure;
• require review when a slice drops beyond its allowed band;
• store the exact prompts, models, voices, tool mocks, audio, seeds where available, and evaluator version;
• rerun a sample to catch evaluator flukes.

Then have a named person approve exceptions. “Known issue” without an owner and an expiry date is just a future incident.

Listen where the average is quiet

The most valuable chart is often not the top-line pass rate. It is the little red cell for one language, one carrier condition, one correction pattern, and one workflow.

Build that cell before your next prompt change. Put a real loss behind it. Make the release fail when the loss comes back.