From production call failure to reproducible regression case

A voice agent fails in production. The caller corrects an address, the agent confirms the correction, and the delivery tool still receives the old flat number. Someone patches the prompt. The next week, a similar correction fails in a different workflow.

That is not a learning loop. It is a memory problem.

This guide shows engineering, product, and operations teams how to turn an approved production failure into a reproducible regression case. You will preserve the evidence needed to explain the incident, remove or replace customer data, isolate the triggering behavior, write layered assertions, and attach the case to a release gate.

The short answer

Stabilize the incident first. Then preserve an access-controlled evidence bundle, reconstruct the session timeline, identify the smallest triggering pattern, create a privacy-safe replay, and assert both the expected conversation behavior and final business state. Prove that the case fails on the affected version and passes on the candidate fix.

Do not paste a raw recording into a shared test repository. Do not “fix the transcript” until it passes. Do not turn one strange call into a global prompt rule before finding the owning layer.

A good regression case keeps the failure’s causal shape while removing the person’s identity.

What counts as a production failure?

Not every awkward call deserves a permanent release gate. Start with impact and recurrence risk.

Strong candidates include:

• Wrong-person or wrong-account access
• Unauthorized or duplicate tool effects
• A false claim that an action completed
• A missed mandatory disclosure or escalation
• Critical entity corruption
• Failed handoff with no recovery
• Repeated caller effort that causes abandonment
• Turn-taking behavior that clips decisive information
• A model, tool, or evaluator regression likely to recur

Low-impact wording issues can still enter a quality backlog. Keep critical regression suites focused enough that teams trust them.

NIST’s AI RMF Core calls for production monitoring, feedback mechanisms, incident response, recovery, and continual improvement. Its Playbook discusses monitoring system behavior and responding to errors. Those are voluntary risk-management resources, not a prescribed voice-agent incident process. The workflow below is Voxeval’s engineering interpretation for this domain.

Scroll diagram horizontally on smaller screens.

Step 1: contain impact before improving the model

The first job is not to create a test. It is to protect callers and the business.

Depending on the incident, containment may mean disabling one tool, narrowing traffic, routing the workflow to humans, rolling back a version, adding a confirmation, or pausing the lane. Keep a record of what changed and when.

Do not overwrite logs, prompts, or configuration while investigating. Preserve the affected version and deployment metadata.

Name an incident owner and a decision owner. The person debugging recognition may not have authority to decide whether a refund lane can reopen.

Step 2: preserve a controlled evidence bundle

Collect only what the incident needs, under the organization’s approved access and retention rules:

• Original caller and agent audio, where permitted
• Partial and final transcripts
• Voice activity, playback, interruption, and timing events
• Prompt, model, voice, transport, and agent versions
• Tool requests, results, retries, and request IDs
• Business state before and after the call
• Policy and workflow version
• Telephony route and relevant network metadata
• Human review notes and customer-reported impact

Hash or otherwise identify the immutable originals. Work from controlled copies.

LiveKit’s prompting guidance recommends using real-world observability, including transcripts, observations, and audio recordings, to find issues and inspire new tests. That advice still sits inside your legal, privacy, contractual, and security obligations. Availability of a recording does not grant permission to reuse it.

Step 3: reconstruct one session timeline

Put the evidence on a shared monotonic timeline. The order should answer:

1. What the caller said and heard
2. What speech recognition emitted over time
3. What the agent believed and generated
4. Which tool request was prepared and committed
5. What the backend accepted
6. What the agent claimed
7. What final state remained

For a fictional delivery incident:

• 10.20s: Caller says, “Flat twenty-six.”
• 11.05s: Interim transcript emits flat 26.
• 12.10s: Agent confirms flat 26.
• 12.40s: Caller says, “Sorry, sixteen, one-six.”
• 12.85s: Interruption is detected.
• 13.10s: Playback stops.
• 13.30s: Final transcript contains flat 16.
• 13.42s: Tool request still contains 26.
• 14.00s: Backend updates the address to 26.
• 14.30s: Agent says, “Updated to sixteen.”

This is invented to show the method. It is not a real Voxeval customer call.

The timeline tells you the recognizer heard the correction and the business action did not reflect it. The likely owning layer is state invalidation or tool preparation, not general speech recognition.

Step 4: separate the trigger from the surrounding call

The full call may contain twenty minutes of irrelevant detail. Find the smallest sequence that still produces the failure.

For the fictional case, the trigger may be:

1. Capture a critical value.
2. Begin confirmation.
3. Caller interrupts with a correction.
4. Agent acknowledges the new value.
5. A tool request prepared before the interruption commits afterward.

That pattern could affect address updates, appointment times, order quantities, and payment amounts. The first regression case should stay tied to the original workflow. A separate generalized case can test the shared state mechanism.

Do not delete timing from a timing bug. A neat text transcript may not reproduce an interruption race. Preserve or simulate the event sequence that caused stale state.

Step 5: choose the safest replay form

There are three common options.

Synthetic reconstruction

Re-record the words with approved speakers or synthesize audio, replace identifiers with fixtures, and reproduce the timing and overlap. This is often the safest reusable form.

Check that the reconstruction still triggers the failure. Synthetic timing, pronunciation, or acoustics may remove the key condition.

Transformed production excerpt

Use a tightly cut, redacted, approved excerpt when the original acoustic condition is essential and your organization permits that reuse. Restrict access and retention. Document every transformation.

Event-level simulation

Recreate partial transcripts, interruptions, state updates, tool timing, and backend responses without caller audio. This is ideal when the bug sits after recognition and the acoustic signal is not relevant.

A mature suite may keep two versions: a fast event-level regression on every change and an end-to-end audio replay for release candidates.

Step 6: redact without destroying the bug

Redaction is not a find-and-replace exercise.

Inventory sensitive data across:

• Audio waveform
• Transcript and subtitles
• File and participant names
• Phone numbers and SIP headers
• Tool arguments and results
• URLs, storage paths, and request IDs
• Reviewer notes and screenshots
• Business-state snapshots

Replace live records with dedicated fixtures. Preserve format, length, pronunciation difficulty, and validation behavior when they matter. Replacing a complex name with “John Doe” can make a recognition bug disappear. Replacing a 12-digit policy number with 123 can change chunking and confirmation.

Run a separate privacy review on the derived artifact. Keep provenance back to the controlled incident record without exposing that link to every test runner.

Step 7: write layered assertions

One regression case can prove several boundaries.

For the correction incident:

• The correction is present in final recognition.
• The active conversation state changes from 26 to 16.
• Any pending tool request tied to 26 is cancelled or invalidated.
• The committed tool argument is 16.
• The authoritative address becomes 16.
• The agent’s spoken confirmation matches 16.
• No duplicate update occurs.

Keep the primary assertion focused on the original impact. Secondary assertions help localize future failures.

Attach severity and a release rule. A stale address update may be critical in a medication-delivery workflow and major in a low-risk brochure request. Context matters.

Step 8: reproduce before changing the agent

Run the derived case against the preserved affected version. If it passes, the reproduction is not ready.

Check:

• Did redaction remove the trigger?
• Is the affected model or tool version still available?
• Did the original failure depend on load, region, timing, or a transient backend response?
• Is the evaluator missing the actual failure?
• Was the incident caused outside the agent stack?

Record the reproduction rate over repeated runs. A race that fails once in ten should not be represented by one lucky pass.

Only after reproduction should the team implement and evaluate a fix.

Step 9: verify the fix did not move the failure

A targeted patch can create caller friction elsewhere. Adding confirmation after every correction may prevent stale state while doubling call time and annoying users.

Run:

• The incident regression
• Neighboring happy paths
• Other correction types
• Duplicate and delayed tool cases
• Caller-effort and latency budgets
• Policy and final-state gates

Review the audio. A correct backend state with chaotic conversational repair is an incomplete fix.

The ten-metric evaluation framework helps catch that tradeoff. The full-duplex guide covers cancellation and stale actions around interruption.

Step 10: give the regression a lifecycle

Store:

• Stable case ID and title
• Incident and source-policy references
• Approval and data-handling basis
• Redaction or synthesis method
• Minimal trigger description
• Fixture and agent version requirements
• Assertions, evidence, and severity
• Reproduction rate on the affected version
• Fix version and review record
• Owning team
• Retention and revalidation date

Do not delete the test because the original model is gone. Keep it while the failure pattern remains possible. Update the replay when an infrastructure change makes the old fixture meaningless, and preserve the history.

Avoid five tempting shortcuts

Patching the prompt immediately

The bug may live in endpointing, state, a tool adapter, backend transaction logic, or the evaluator. A prompt patch can hide the symptom.

Fine-tuning on the raw call

One incident is not automatically valid training data or enough evidence for a model change. Resolve rights, privacy, representativeness, and expected behavior first.

Keeping only the transcript

You may lose timing, overlap, pronunciation, and what the caller heard.

Testing only the final answer

The same words can accompany the wrong tool effect. Verify the system of record.

Closing the incident when the patch ships

Close it when containment, root cause, regression, fix evidence, rollout, and follow-up ownership are complete.

Where Voxeval fits

Voxeval is being built to connect incident evidence, redacted fixtures, scenario versions, failure attribution, and release gates. The goal is not to become a warehouse of raw calls. It is to preserve reusable evidence with less data and more meaning.

The fastest-improving teams do not avoid every production surprise. They make sure the same surprise does not become folklore and happen again unnoticed.

Common questions

Should every production failure become a regression case?

No. Prioritize impact, recurrence, shared mechanism, and diagnostic value. Group duplicates under one well-designed case when they share a trigger.

Can we use customer call audio in tests?

Only under your organization’s approved consent, purpose, contractual, privacy, security, access, and retention rules. Prefer synthetic or event-level reconstruction when it preserves the failure.

What if the incident cannot be reproduced?

Keep the evidence and investigate missing conditions such as load, network route, timing, provider version, or backend state. Do not invent a deterministic story. Add monitoring that can capture the condition next time.

Should the regression test the old failure wording exactly?

Test the causal behavior and outcome. Preserve exact wording only when it is part of the trigger, such as a pronunciation, correction form, or policy phrase.