Full-duplex voice changes the eval plan

A voice agent can produce the right transcript and still mishandle the conversation. It can start a correct answer while the user is still naming an account, acknowledge at the wrong moment, keep speaking after a barge-in, or trigger a tool from an abandoned request. Full-duplex systems make those failures more visible because listening, speaking, deciding, and acting can happen concurrently.

That is the evaluation shift. Transcript accuracy remains useful, but it is one projection of a timed, stateful interaction. A release plan also needs audio events, model and tool traces, environment state, and outcome evidence aligned to a shared clock.

The current evidence should be read in layers. Vendor announcement: in its July 8, 2026 GPT-Live announcement, OpenAI described an architecture that can process input while generating output, choose whether to speak or listen, produce short acknowledgements, wait through pauses, and delegate deeper work. Those are OpenAI’s product claims, not independent measurements. GPT-Live powered ChatGPT Voice and was not yet offered through the API in that announcement. OpenAI said API access was planned soon, so the post was not evidence that an API team could deploy GPT-Live on announcement day.

Engineering account: OpenAI’s May 4 WebRTC article explains how its media infrastructure carries continuous audio and how its relay-and-transceiver design handles routing and session ownership. It is a useful account of one provider’s deployed infrastructure, not a comparison proving that one transport or topology fits every workload.

Research preprint: τ-Voice was submitted to arXiv in March 2026. The authors combine grounded task completion, realistic audio, and full-duplex interaction in a simulator-backed benchmark. They report results over 278 tasks, including lower task completion for the evaluated voice agents under noisy, varied-accent conditions than under clean conditions. The study uses English-only synthetic TTS personas rather than recorded speakers, so its accent findings are indicative, not definitive. Its simulator has perfect memory and instantaneous tool calls. Measures outside its scope include tool-call latency, user satisfaction, partial task success, and agent speech-generation quality. Those are measurements within the paper’s selected agents, tasks, simulator, and scoring setup. The preprint is not an industry-wide production study, and its results should not be generalized to an untested stack.

Full-duplex changes the system under test

A turn-based test usually has a simple sequence: user audio ends, the agent responds, then a scorer compares content with an expected answer. Full-duplex breaks that sequence into concurrent event streams. User speech may continue while agent audio begins. A backchannel may be appropriate even though a substantive answer would be premature. A tool may run while the agent fills the conversational gap. The user may interrupt while synthesized audio and a downstream action are already in flight.

The unit under test needs to be the session, not a list of utterances. At minimum, record user voice activity, agent audio start and stop, partial and final recognition, model decisions, interruption signals, tool request and result events, playback state, and the authoritative business state. Preserve a monotonic session clock and stable event IDs, then add explicit parent and causation links between derived events. The clock establishes ordering; ordering does not establish causation.

Latency is still a dimension, but one aggregate number hides different risks. Measure connection setup, end-of-utterance detection where relevant, first agent audio, interruption detection, playback stop, and tool completion separately. Report distributions by network and device condition. A quick first sound does not compensate for slow cancellation, and a fast tool response does not show that the right tool was called.

This event model is our engineering inference from the concurrency described in the sources. Neither OpenAI post nor the τ-Voice preprint defines a universal release schema. Teams should choose clocks, traces, and thresholds that match their own transport, model, tools, policies, and customer workflow.

Scroll diagram horizontally on smaller screens.

The failure modes a transcript cannot show

Overlap is not merely two strings sharing a timestamp. The evaluator needs the audio mixture and playback state. Did the agent mask a critical account number? Did echo cancellation turn its own speech into apparent user input? Did it start a long response during a brief hesitation? Score who held the conversational floor, whether information was lost, and whether recovery was coherent.

Backchannels need their own label. A short acknowledgement can signal attention without taking the floor, but the same phrase can sound like confirmation of a transaction or interrupt a sensitive disclosure. Test semantic neutrality, placement, frequency, and whether acknowledgements leak into the user’s recognized text. Do not count every emitted sound as a successful response.

Barge-in spans several clocks. Detection is only the first event. Playback must stop within the product’s budget; the agent must preserve what the user heard; obsolete generation must be discarded; and any tool action tied to the interrupted intent must be cancelled or made safe. A transcript can look orderly after post-processing while the user actually heard several unwanted words or the backend completed a stale action.

Silence is ambiguous. It can mean thinking, turn completion, distraction, a dropped connection, or consent to proceed. Build cases with short and long within-utterance pauses, background speech, packet gaps, and explicit requests to wait. The expected behavior may be continued listening, a neutral check-in, or a safe timeout; it should come from the workflow policy rather than a model’s default endpointing behavior.

Tool timing couples conversation state to external state. Capture when the intent became sufficiently specified, when the tool request was committed, whether it was idempotent, what the user had heard at that point, and how late or failed results were presented. OpenAI says GPT-Live can delegate work while maintaining conversation, but that announcement does not establish correct tool behavior for another application.

Business outcome is a separate axis. A session may feel fluid yet violate policy, update the wrong record, omit required disclosure, or fail to resolve the task. It may also sound awkward and still complete the workflow correctly. Keep interaction quality, policy compliance, tool correctness, and verified task outcome as distinct scores so a pleasant conversation cannot wash out a consequential failure.

A practical full-duplex eval matrix

Start with a small matrix that produces replayable evidence. Replace the example release rules with budgets derived from user research, risk tier, and measured baselines; there is no source-backed threshold that applies to every voice product.

Stable network, clean speech

Expected behavior. Respond at the intended boundary without clipping the user. Evidence. Retain aligned user and agent audio, voice activity, and the first-audio trace. Release rule. The latency distribution stays inside the lane’s approved budget.

User speaks during agent audio

Expected behavior. Yield or continue according to the scripted intent and policy. Evidence. Retain the interruption signal, playback-stop event, and audible recording. Release rule. No critical user content is masked, and the cancellation budget passes.

Neutral listening cue is appropriate

Expected behavior. Use a non-committal backchannel without taking the floor. Evidence. Retain the audio label, timing, and recognition stream. Release rule. There is no false confirmation or user-transcript contamination.

Mid-utterance thinking pause

Expected behavior. Continue listening without inventing completion. Evidence. Retain the pause annotation, endpoint decision, and agent audio. Release rule. Behavior matches the scripted pause policy.

Background talk or burst noise

Expected behavior. Keep foreground intent separate or recover safely. Evidence. Retain mixed audio, diarization or input events, and the recovery trace. Release rule. There is no unauthorized action, and the recovery path completes.

Slow, failed, or late tool

Expected behavior. Explain or recover without fabricating a result. Evidence. Retain tool request and result timestamps, spoken claims, and final state. Release rule. Spoken status matches authoritative tool state.

Barge-in after action preparation

Expected behavior. Stop obsolete speech and prevent stale side effects. Evidence. Retain the playback trace, cancellation event, and audit log. Release rule. There is no duplicate or abandoned-intent mutation.

End-to-end business task

Expected behavior. Complete the permitted task with required policy steps. Evidence. Retain final system state, policy assertions, and the session recording. Release rule. Outcome and policy gates both pass.

Run each condition across representative accents, speech rates, devices, codecs, network impairment, and tool response profiles. Treat these as controlled test strata, not a claim that a finite matrix reproduces production. Add real, consented failure patterns after launch, with sensitive data removed and access controlled.

What to gate before release

Averages alone can hide rare but severe failures and are weak release gates for consequential lanes. Define lane-specific budgets and inspect tails; then pair quantitative checks with reviewed recordings for timing and conversational meaning. Safety- or money-changing workflows need stricter side-effect and policy rules than a low-risk informational assistant.

Vendor-neutral pre-release checklist

• Define the intended floor-taking, backchannel, pause, and interruption policy for each workflow.
• Align input audio, agent playback, model decisions, tool events, and business state on one session timeline.
• Measure setup, response, interruption, playback-stop, and tool timing as separate distributions.
• Test clean speech, overlap, pauses, noise, packet impairment, varied speech patterns, and late tools.
• Verify that barge-in cancels obsolete output and prevents stale or duplicate side effects.
• Score interaction quality separately from transcript content, policy compliance, tool correctness, and task outcome.
• Review failures by severity and condition; do not let a high average hide a critical lane.
• Record the model, prompt, voice, transport, tool versions, and evaluator configuration for reproducibility.
• Convert consented production incidents into redacted regression cases with an owner and release rule.
• Document untested populations, languages, devices, networks, workflows, and residual risks in the release decision.

Where Voxeval fits

Voxeval is intended to help teams organize these session-level cases, evidence, and release rules across a changing voice stack. The matrix and checklist do not depend on Voxeval: they can be implemented with an internal harness, vendor tooling, or a combination. The important step is to make concurrent behavior and business state observable enough that a release decision can cite evidence rather than a polished transcript.