Part 8 of 12 · 2 min read
Answering voice-agent questions in an interview
How would you design the latency budget for a voice agent?
"Define the turn gap as silence from the end of the user's speech to the first agent audio, and set p50 and p95 targets. Break it into endpointing, transcript finalisation, model start plus first sentence, first audio chunk and network. Stream every stage and start speaking at the first sentence. In the example's simulation streaming saved about 600 ms and a faster model about 200 more, but endpointing was the largest single item, so I tune it against the cut-in rate." Follow-up: "What else cuts latency?" Smaller or faster models, caching the prompt prefix, regional hosting, pre-warmed connections, and speculative responses where safe.
How do you handle a user interrupting the agent?
"Detect speech, stop playback and flush the buffer within a couple of hundred milliseconds, cancel generation, and record in the history only what the user actually heard, marked as interrupted. I add a minimum speech duration and echo cancellation so noise and the agent's own audio do not trigger it." Follow-up: "Why truncate the history?" Otherwise the agent believes it delivered information the caller never heard. In the example's simulation that was about 17 of 35 words.
When is voice the wrong interface?
"When the information is dense or needs reference: speech runs at about 150 to 180 words per minute against 250 to 300 for reading. I use voice for hands-free, quick, low-complexity or accessibility needs, and send details by text." Follow-up: "How do you support both?" Voice for the conversation. A text or email summary and a link for the dense parts.
A voice agent executes a payment the caller did not intend. What failed?
"Likely a mis-transcription plus no confirmation before an irreversible action. I add read-back and explicit confirmation, limits and approval for risky actions (Level 10 module 5). I also add word-error evaluation on domain terms, and logging to reconstruct what was heard." Follow-up: "What about fraud and voice cloning?" Do not treat a voice as authentication; use a second factor for sensitive actions.