This guide is for developers and QA engineers building a voice agent. The reference kit assumes you can run Python commands and inspect JSON events. For an agent that only answers questions, adapt the prompts and case checklist to that task, use the speech and correction cases, and mark booking checks as inapplicable. Start with the local checks, capture one matched pair, then use the eight-case rubric to plan the wider test.
Define the decision and pass conditions first
For a booking assistant, the useful question is whether a caller can change Thursday to Tuesday while the assistant is speaking and still get the right proposal. For a tutor, a brief “mm-hmm” may need no response at all. For a phone menu, stopping promptly can matter more than conversational overlap. Write down which interaction your product needs before collecting a latency number.
Write a short pass/fail checklist for each case: the words the caller says, the information that changes, the information that must survive, the action that remains allowed, and the behavior that counts as a failure. The kit stores this checklist as a scenario contract. Set stop and recovery limits from your product requirements or a listening pilot, before comparing the models.
Choose limits around the interaction. After “Stop,” how much unwanted speech can the caller tolerate, and is an acknowledgment allowed? After a correction, how long can the caller wait before repeating it? Review recorded examples with the team or representative users, then write down the agreed limits. Require explicit approval before a booking commits, regardless of which voice responds faster. A faster system can still fail your minimum requirements.
Keep four judgments separate. Did output stop when it should? Did the assistant capture the complete new request? Did its response and proposed tool arguments use the corrected values? Did the application prevent an action whose permission had changed? A system can succeed on one judgment and fail another. Record an attempted stale booking even if a guard successfully blocks it.
Match the task across the two architectures
GPT-Live-1 can listen and speak simultaneously and delegate reasoning and tool work to a backend. GPT-Realtime-2.1 supports speech-to-speech interaction, configurable reasoning effort, and function calling in its Realtime workflow. These are different system designs, so each needs its own integration and prompt.
| Control | GPT-Live-1 | GPT-Realtime-2.1 |
|---|---|---|
| Conversation and task work | Live voice instructions plus a Responses backend or client-managed backend | Realtime session instructions and direct function schemas |
| Record in every run | Delegation mode, backend identity, backend instructions, and any returned state | Reasoning effort, turn-detection settings, function definitions, and tool results |
| Keep equivalent | User goal, starting facts, correction policy, and authorization rules | The same goal, facts, policy, and rules, with matched fake data, recording, trigger, and pass criteria |
| Keep paired | Transport, client revision, environment profile, and run slot | The same transport, client, profile, and slot, with matching input-audio and task-contract hashes |
Use architecture-specific wording to express the same policy. Both paths should preserve Boston when the caller changes the date, and both should require confirmation for the current proposal. Sending identical JSON to incompatible APIs does not establish fairness. Conversely, changing the task instructions while changing the model makes the result hard to interpret.
The reference kit uses client-state-machine-v1 as GPT-Live's backend. It is a deterministic service, not a second language model. Replacing it with a reasoning agent changes the tested system, including its latency and cost. OpenAI's delegation guide explains the two delegation modes and the application's responsibility for task state and permissions. Record the backend you actually use.
Know what the reference kit measures
Download the interruption benchmark kit. It contains eight frozen recordings, architecture-specific prompts, one-row WebSocket capture, human-review tools, an 80-row scorecard, and evidence checks. The recordings are synthetic English speech at 24 kHz, mono PCM16. You can use them without rebuilding them with macOS speech synthesis.
Its audio player is an in-memory queue: it does not send sound to a speaker. Its input timestamps mark the start and end of sending a WAV, including any silence. These records help you check capture and queue behavior. To measure what a caller hears, add real playback capture and mark where speech actually begins and ends.
The runner also installs the expected answer when it starts sending the interruption. If the caller says Tuesday, Tuesday is already in the application's state before the model has demonstrated understanding. GPT-Live can receive it through delegation. A correct final date could therefore come from the supplied answer. For an understanding test, keep the expected answer in a separate scorer and capture the model's proposed correction before returning any corrective backend result.
There is also a tool-path difference: the reference Realtime path processes function calls, while the Live client-delegation handler returns state and proposal information. The latter does not exercise an equivalent model-driven booking workflow. Use the shipped runner for capture practice and a controlled pilot. A product comparison needs both backends to perform the same permitted task and expose attempted arguments and rejected actions.
Most recordings are injected at an offset after assistant audio starts. A booking-cancellation test needs a trigger tied to the actual proposal becoming ready. The kit's review-guide.md lists these integration changes and the limits of its timing helpers and gates. Complete them before using the results to rank systems; the local self-tests check capture mechanics.
Set up locally and run one matched pair
Extract the kit into a private working directory, enter the folder containing benchmark-runner.py, and use Python 3.10 or later. Keep evidence outside a served website directory. The runner needs websocket-client for network capture; the local checks below do not make model requests.
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements-runner.txt
python3 benchmark-runner.py --self-test
python3 review-captured-run.py --self-test
python3 verify-evidence.py --self-test
python3 interruption-test-gate.py --self-test
python3 benchmark-runner.py --model gpt-live-1 \
--scenario date-correction --run-id 1 \
--environment-id lab-direct-v1 --dry-run
python3 benchmark-runner.py --model gpt-realtime-2.1 \
--scenario date-correction --run-id 1 \
--environment-id lab-direct-v1 --dry-run
python3 interruption-test-gate.py scorecard.csvThe dry runs should report network_request: false, the selected endpoint, input hash, and planned evidence path. The final command should return HOLD on the untouched scorecard because the 80 model rows have not run. That command exits unsuccessfully by design until the matrix is ready; it is not an installation failure. An offline self-test cannot verify your account's model access or the live event schema.
Copy environment-profile.example.json to lab-direct-v1.json. Set its environment_id to lab-direct-v1 and replace every RECORD_ and REPLACE_ value with the actual OS, Python and WebSocket-client versions, client source revision or archive hash, network, and routing information. Record automatic routing if you do not control the region. Keep the declared audio path as the simulated queue for this pilot.
Supply OPENAI_API_KEY through your normal secret mechanism. The following commands make paid requests and capture one row each. Start with this pair and inspect its evidence before committing time and budget to the remaining cases.
python3 benchmark-runner.py \
--model gpt-live-1 --scenario date-correction --run-id 1 \
--environment-id lab-direct-v1 \
--environment-profile lab-direct-v1.json \
--evidence-root ./private-evidence \
--scorecard scorecard.csv --confirm-paid-api-run
python3 benchmark-runner.py \
--model gpt-realtime-2.1 --scenario date-correction --run-id 1 \
--environment-id lab-direct-v1 \
--environment-profile lab-direct-v1.json \
--evidence-root ./private-evidence \
--scorecard scorecard.csv --confirm-paid-api-runThe evidence directories follow private-evidence/voice-interruption-v1.1/lab-direct-v1/date-correction/<model>/run-1/. Check run.json, events.jsonl, state.json, input.wav, and assistant.wav, along with the retained contract, prompt bundle, and environment. CAPTURED means evidence was collected. It says nothing yet about whether Tuesday was understood.
A connection, authentication, or server error needs inspection even if a file was written. The runner can retain a failed request; it can also accumulate server errors inside a captured session. Read the error fields and session events. If there is no usable response, preserve the failed attempt and its available usage instead of assigning it a successful outcome.
If the first pair will not run
| Symptom | Check before retrying |
|---|---|
| Python or package import error | Check python3 --version, activate the virtual environment, and install with that interpreter's python3 -m pip. Run from the extracted kit folder. |
| Environment-profile validation fails | Match the profile's environment ID to the command and replace its placeholder values. Describe the audio path you actually used. |
| Authentication or model-access error | Check that the key is available to the process, belongs to the intended project, and has access to the requested model. Read the provider's error message; keep the key out of logs. |
| Rate limit, disconnection, or server error | Retain the failed attempt and any usage. Follow the reported retry guidance, then record a new attempt. Do not repeatedly rerun into the same evidence directory. |
| No sound comes from the computer | The reference runner has no speaker output. Inspect the saved WAV and received audio events. Live listening requires the product's real player. |
| Audio is distorted or unexpectedly fast | Check sample rate, channel count, and PCM format against the session configuration before blaming the model. The shipped inputs are mono, 24 kHz PCM16. |
| No valid overlap was captured | Check whether the assistant was still speaking at injection. An interruption after the answer ends cannot supply a stop-latency result. |
After the pilot, use the five supplied run slots for each of the eight scenarios and both models: 80 attempts. Alternate which model runs first within each pair, and vary the starting model across scenarios. Keep each pair close in time. Five runs per case are exploratory, so expand the sample for consequential decisions. A changed prompt, fixture, backend, player, or VAD setting needs a separately identified condition and fresh scorecard; do not mix it into the original pair.
The original scorecard has only five slots per model and case. If a slot failed, keep it. Additional attempts need a separate expanded scorecard or scorer with new identities; the supplemental outcome sheet can retain them, but it does not extend the capture runner's slot handling.
Test eight conversation cases
| Case ID | Input and context | Check in the response, state, and logs |
|---|---|---|
hard-stop | “Stop” after assistant speech has started | Output stops within your limit. The abandoned explanation stays stopped; silence after the command can be correct. |
date-correction | Thursday to Tuesday; destination remains Boston | Tuesday replaces Thursday, Boston survives, and any later date argument uses Tuesday. |
budget-correction | $800 to $500; keep Harbor House | The hotel survives and the next relevant proposal uses $500. A correct spoken budget alone is insufficient. |
backchannel | “Mm-hmm” during an explanation | The answer continues coherently without a new task. Note an unnecessary cutoff, restart, or repeated section. |
thinking-pause | A flight request generated with a 1.4-second pause instruction | No premature answer during the pause. The final request includes Seattle, next Friday, and morning. |
background-speech | “Alex, the package is by the door” during another task | No task replacement or unrelated tool request. Test spatial speaker rejection separately in a room. |
identifier-repair | B74 corrected to D74 using phonetic letters | The final value is exactly D74. Check letters and digits independently, and inspect any lookup argument. |
action-cancellation | “Do not book it” after a proposal, before authorization | The proposal loses permission to commit. Inspect attempts, rejection, pending work, and the final record. |
For the hard-stop case, decide whether a short acknowledgment is allowed. If silence is allowed, a missing next response is not automatically a failure and recovery latency is not applicable. Continued speech, a restarted explanation, or an invented new request is a failure. Put that policy in the contract so two reviewers do not score the same silence differently.
Date and budget corrections need both replacement and preservation. “Tuesday” with the destination lost is partial recovery. Keeping the hotel while searching under the old budget is a stale-argument failure. A model that asks for clarification can be correct when the caller's correction is ambiguous; for the explicit shipped recordings, unnecessary repetition adds recovery work and should be recorded.
Keep ambiguous requests in separate cases. “Make it earlier” does not identify a new date; “not that one” may not identify a proposal. Specify what the agent should ask before acting. For relative dates such as “next Friday,” give both systems the same reference date and time zone and check the resolved date. A benchmark should distinguish a sensible clarification from failure to hear an explicit correction.
For backchannels, distinguish a short conversational pause from abandoning the answer. A server may signal speech onset for “mm-hmm” even though the product policy calls for continuation. Score what the user receives, including whether the assistant restarts a paragraph, asks an irrelevant question, or loses its place.
The thinking-pause fixture is sent as one user turn, not as a barge-in over assistant speech. Stop latency does not apply. Listen across the internal pause and score whether the complete request survives. The reference timing helper assumes assistant speech was active at interruption onset, so its automatic stop and next-response fields do not provide a complete timing analysis for this case.
The background-speech recording tests whether unrelated words hijack a task when injected into the input stream. It does not test distance, microphone direction, or speaker identity. For a product-room test, keep the primary caller's position fixed and record the secondary speaker's position and level. Test a caller addressing the agent directly as a control, so rejecting all speech cannot look like a good result.
Identifier repair is useful for booking codes, order numbers, and account lookups. Preserve the correction's exact characters. “D seventy-four” may sound right but still produce the wrong structured identifier. Add separate cases for an uncertain character that should trigger clarification and for an unambiguous character that should not.
Measure speech boundaries and actual playback
For an intended interruption, place four markers on one monotonic timeline: the first audible sample of the caller's interruption, the last audible sample of the abandoned assistant response, the final audible sample of the caller's new turn, and the first audible sample of the useful repaired response. State whether a filler such as “one moment” counts; measuring only that filler can hide the time until the correction is actually addressed.
audible_stop_latency_ms = assistant_audio_stop_ms - user_speech_onset_ms
recovery_latency_ms = next_assistant_audio_ms - user_speech_end_msThe reference runner's input markers describe sending the fixture, and its output markers describe queue frames. A WAV may start or end with silence, and a queued chunk may contain silence between words. Annotate the speech boundaries before calculating caller-audible latency. Retain both the original transfer markers and the corrected annotations; document the boundary rule and how reviewers resolve uncertain samples.
Capture product output after the player or through a loopback path, with a synchronized input reference. For a telephone product, record the endpoint the caller hears. Log queue depth, clear time, and transport events alongside that audio. The reference assistant.wav contains simulated delivery and is useful for listening to content, but the recorder is not connected to a sound device and does not prove actual audible stop time.
Use the abandoned response's identity to separate its tail from a new acknowledgment. A short gap followed by more of the old explanation is resumed unwanted speech. If input and output are recorded on different devices, align them with a shared reference before subtraction; subtracting unrelated wall clocks can manufacture a latency difference.
Missing values need a reason. No assistant speech at the scheduled injection point means that attempt did not exercise the intended overlap. No next response after a correction may be a failure. No next response after a successful “Stop” can be the specified outcome. Keep these cases in the outcome record and calculate timing only where it is defined; never enter zero merely to complete a table.
If the repaired response starts before the caller finishes, the recovery calculation can be negative. Inspect that overlap: an acceptable brief acknowledgment and an answer that talks over an unfinished correction are different outcomes. Keep the signed measurement and score the behavior under your turn-taking policy; do not call every negative value faster recovery.
Separate turn detection from cancellation and truncation
In Realtime's interruption flow, automatic VAD interruption can cancel a response while the client still has audio queued. WebRTC and SIP can automatically truncate unplayed output because the server manages their audio buffer. A WebSocket client owns playback and needs to stop it, determine how much of the response was played, and send conversation.item.truncate.
{
"type": "conversation.item.truncate",
"item_id": "THE_ACTUAL_INTERRUPTED_ITEM_ID",
"content_index": 0,
"audio_end_ms": 1500
}This is a schema example, not a universal timestamp. Replace the item ID and audio_end_ms with the interrupted item's identity and actual played duration. The 1,500 ms value here describes output played within that item; it is different from the fixture trigger relative to the start of the assistant's speech. Truncation also removes the transcript of unplayed content, helping keep unheard speech out of later conversation context.
Realtime VAD offers server_vad, based on silence, and semantic_vad, based on whether the utterance appears complete. Server VAD exposes threshold, prefix_padding_ms, and silence_duration_ms; semantic VAD exposes eagerness. Lower semantic eagerness allows more time to finish speaking. Record create_response and interrupt_response independently.
The kit starts Realtime with semantic VAD, medium eagerness, and both automatic-response and interruption flags enabled. To compare alternative settings, create separate conditions. Check hesitation and backchannel cases as well as hard stops: a setting that yields quickly can still interrupt a sentence before the user is finished. Change one control at a time and rerun the same recordings.
GPT-Live has its own conversation flow. Its WebSocket output lacks timing fields and an output-audio-done event, according to the voice WebSocket guide. Track delivery with your player and keep transcript timing separate. The reference Live path queues the received chunks; it does not reproduce the same Realtime speech-start clear-and-truncate sequence. Record each application's stop policy before interpreting a difference as model behavior.
Test cancellation during backend work
A booking can be prepared, authorized, submitted, or already confirmed when an interruption arrives. Those states need different outcomes. The shipped cancellation case starts with a fake prepared proposal and no confirmation. It does not call a real booking service or demonstrate rollback of a submitted action.
Use a proposal revision and a confirmation tied to that revision. After a correction, invalidate the old proposal and require approval of the new one. When a pending operation returns, compare its revision with the current revision before applying it. Record a rejected stale result as a rejected attempt, not as a correct model proposal.
# Evaluation design; adapt to your application's transaction model.
on_correction_or_cancellation:
active_revision += 1
invalidate_previous_proposal_and_confirmation()
before_commit:
require(proposal.revision == active_revision)
require(confirmation.matches(proposal))
require(action_is_still_authorized)Keep separate fields for what the model proposed and what the application allowed. A guard blocking a Thursday proposal after a Tuesday correction is a successful guard and a failed semantic or tool-argument outcome. Likewise, no committed booking proves little about cancellation handling if the test never created a proposal or never exercised a permitted commit path.
| When the caller cancels | Expected application behavior | Evidence to inspect |
|---|---|---|
| During preparation | Invalidate the pending revision and reject its late result | Operation ID, revision, completion time, and rejection reason |
| After preparation, before confirmation | Invalidate the proposal and its approval path | Proposal status, confirmation state, and any commit attempt |
| After confirmation, before submission | Recheck permission at the commit boundary | Ordering of cancellation, permission check, and submission |
| After external submission | Determine whether the action happened; use the service's cancellation or recovery flow | External action identity and authoritative status, including uncertainty |
Add a positive control in which the caller confirms the current proposal and a negative control in which an old confirmation is reused after a correction. Also test a delayed tool result and duplicate delivery. Use fake side effects while developing these cases. OpenAI explicitly notes that interrupting Live speech leaves backend work running; stopping the audio therefore cannot stand in for stopping an operation.
Review each run before marking it complete
Listen to the retained response, inspect the input and output transcript fragments, and compare the actual tool history with the scenario contract. Use the transcript as a navigation aid. It cannot establish that a consonant was audible, that the first correction word survived, or that buffered speech stopped at the time claimed.
Use the six review fields to record whether the correction was heard, unchanged facts survived, the old answer stopped, an unwanted interruption occurred, an action committed, and tool arguments used the latest request. Their exact flag names appear in the command below. Use na only when the question does not apply. If a required tool was never exercised, explain the gap and leave the outcome incomplete.
For naturalness, use the same listening rubric throughout: 1 for an unusable transition; 2 for a disruptive restart or repeated talking over the caller; 3 for understandable recovery with noticeable awkwardness; 4 for a coherent transition with minor roughness; and 5 for coherent recovery without a noticeable transition problem. These are suggested local anchors. Score content, timing, and action behavior separately, so a smooth voice cannot offset a wrong date.
For the listening review, make copies labeled A and B, hide the model names, and vary which clip is heard first. Keep the original evidence intact and retain the label mapping separately. Include enough context to hear what was interrupted. For important or disputed cases, have a second reviewer score independently before resolving the disagreement. Record voice preference separately from whether the turn worked.
How to judge the Tuesday correction
Start from the supplied date-correction case: Thursday changes to Tuesday, and Boston must survive. The table below is a decision guide for evidence you might encounter, not a set of model results. Apply it to the independently observed response and tool events, after removing the supplied-answer shortcut described earlier.
| If the evidence shows | What to record |
|---|---|
| The reply uses Tuesday and Boston, and any required tool uses both | Those content and tool checks pass. Check stopping, recovery, and authorization separately before passing the task. |
| The reply says Tuesday, but the tool still uses Thursday | Spoken correction captured; tool-argument failure. A convincing acknowledgment does not repair the booking. |
| The model proposes Thursday and the application blocks it | Incorrect proposal; successful guard; failed task correction. Preserve both outcomes. |
| The reply uses Tuesday but loses Boston | Correction captured; unchanged destination lost. The task remains incomplete or failed under your checklist. |
| State says Tuesday only because the runner inserted it | Understanding is unproven. Do not fill the observed-model field from this state or award a semantic pass. |
The command below illustrates review of a date-correction row whose evidence supports the six listed judgments. Fill the reviewer and actual reconciled call cost first, and change every judgment to match what you observed. Do not copy these values into a run you have not inspected.
VOICE_REVIEWER='Your actual reviewer name'
VOICE_CALL_COST_USD='Replace with the measured numeric call cost'
VOICE_RUN_DIR='./private-evidence/voice-interruption-v1.1/lab-direct-v1/date-correction/gpt-realtime-2.1/run-1'
python3 review-captured-run.py \
--scorecard scorecard.csv --evidence-dir "$VOICE_RUN_DIR" \
--scenario date-correction --model gpt-realtime-2.1 --run-id 1 \
--reviewer "$VOICE_REVIEWER" \
--correction-captured yes --protected-fields-retained yes \
--abandoned-content-removed yes --false-interruption na \
--tool-action-committed na --tool-arguments-current na \
--naturalness 4 --cost-usd "$VOICE_CALL_COST_USD" \
--attest-listenedUse the matching Live directory and model ID for its review. Add --notes for missing or inapplicable timing and any ambiguity. A reviewed connection failure uses --attest-inspected, descriptive notes, and --cost-usd na only if final usage is unavailable. Preserve the attempt as FAILURE_REVIEWED; rerunning it is a new attempt, not a replacement for the failure.
The reference gates have a limitation worth checking before extending them to your product: the reviewer and scorecard gate reject a cancellation row that records a committed action. Preserve such a failure in the evidence and supplemental outcome record. Do not change “yes” to “no” to satisfy the gate. A product evaluator must accept valid evidence of unsafe behavior and mark the task failed while blocking release.
python3 interruption-test-gate.py scorecard.csv
python3 verify-evidence.py scorecard.csv ./private-evidenceThe two commands report review completeness and file integrity, respectively. Their success verdicts do not apply your product's pass/fail checklist. The gate's aggregate timing output also blends intended-interruption scenarios; calculate date correction, hard stop, and other cases separately when making a decision.
Repeat the relevant cases through the product
After the digital pilot works, capture the same cases through the microphone, player, echo cancellation, and transport your users will encounter. Give this condition a new environment ID. A browser headset test, a laptop speaker test, and a telephone test are separate conditions because each contains different capture and playback behavior.
Start with a quiet headset control. Then test speaker playback with the user's microphone open, so you can detect the assistant hearing its own output. Compare a near caller with a quieter or more distant caller, and add a secondary person speaking to someone else. Keep the positions and volume documented. A noise-processing change belongs in a new condition.
Use consenting speakers and synthetic task details for accent, speaking-rate, and hesitation coverage. Include soft-spoken corrections, an interruption that starts with “no,” and a caller who changes a digit midway through a code. For telephony, repeat through the actual codec and connection. Synthetic English recordings cannot establish performance for another language, a speech impairment, a particular accent, or natural overlapping speakers.
Vary when the interruption arrives: just as the assistant starts, in the middle of a sentence, and near the end. Verify that it actually overlaps speech in each attempt. Listen especially for a lost first word such as “not”; the remaining sentence can mean the opposite. Also test a request after the assistant has finished, where stop latency does not apply. Give changed recordings and triggers their own case identities.
Extend the conversation beyond the first correction: Tuesday becomes Wednesday, the caller says “Actually, cancel it,” and a previously prepared result arrives late. Check whether the latest request wins without losing unchanged facts. Add reconnect and dropped-connection cases if your product resumes calls; inspect the external operation's status before retrying an action whose result was lost.
Interpret per-case outcomes and cost
For each scenario and condition, report attempted calls, valid overlap trials, passes, failures, missing measurements, and provider errors. Show individual values with the median and worst observed latency in a five-run pilot. Avoid a precise tail-latency claim from five observations. A lower overall median can hide poor identifier repair or an unsafe cancellation attempt.
Compute paired differences within the same scenario, input, and condition. If one attempt did not start speaking in time, keep its task outcome but exclude it from a paired stop-latency calculation with an explicit reason. Report the timing denominator alongside the attempt denominator. Otherwise, missing hard cases can make the remaining times look artificially good.
Use a result record with separate cells for timing, correction or restraint, preserved fields, tool attempt, committed action, listening quality, and measured cost. Keep observed state distinct from expected state. The kit's supplemental product-review.csv provides blank slots for this review; its values need evidence and are not automatically inferred from the capture scorecard.
Billing uses different units: Live charges for session duration with backend and tool usage separately, while Realtime publishes token-based text and audio pricing on its model page. Retain the usage events, reconcile total call cost, and include delegated work and billable failures. A lost usage record makes the cost comparison incomplete; it is not a free call.
# Apply only the checks relevant to the scenario.
task_pass = timing_pass AND semantic_or_restraint_pass
AND required_state_checks_pass AND required_tool_checks_pass
cost_per_successful_task = total_measured_cost / successful_tasksIf successful_tasks is zero, report “no successful tasks” and the total measured spend; the ratio is undefined. Keep the cost numerator and success denominator within the same workload and condition. Use the full call, including recovery and retries, when comparing a support task. A warm cached short confirmation and a long fresh conversation should not share one cost estimate.
Use the failure pattern to choose the next change
| Observed pattern | Inspect next | Retest before choosing |
|---|---|---|
| Cancellation event arrives quickly, but speech continues | Queued playback, clear behavior, transport buffers, and actual output capture | The same recording with corrected playback instrumentation |
| Output stops, but Tuesday or “not” is missing | The received input, speech boundary, transcript, and any leaked expected state | Correction capture before backend correction is returned |
| Good hard stops, poor “mm-hmm” handling | Turn policy, automatic interruption, and loss of answer position | Both substantive interruptions and listener signals under each setting |
| Premature response during a hesitant request | Silence-based turn detection or semantic eagerness, then the complete received utterance | The same pause with the full destination and date checked |
| Correct speech, stale tool arguments | Proposal revisions, confirmation binding, and late-result handling | A corrected proposal plus stale-result and reused-confirmation controls |
| Digital pilot passes, speaker or phone test fails | Microphone pickup, echo cancellation, resampling, codec, and network path | The failing product condition against the quiet control |
| Differences change sign across paired runs | Run order, configuration drift, timing validity, and sample size | More matched attempts before naming a performance winner |
Choose a path only for the workloads it meets. A support conversation needs reliable corrections and tolerable backchannels; a structured booking needs current tool arguments and working approval boundaries; a tutor needs to respect unfinished speech. If neither path meets those requirements, keep the action behind explicit confirmation, change the interaction, or retain a human handoff while you repair and retest the failing cases.
Keep the final decision beside the evidence: model and backend identities, prompts and settings, transport, capture method, test date, speakers and conditions, attempts per case, failure counts, cost coverage, and the next retest trigger. A corrected player, a different backend, or a new telephone path can change the result, so rerun the affected cases before carrying the choice into that configuration.
