CSE Roadmap

Specialist tier · about 3 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that make a caller talk over, or wait in silence

Common mistakes, and what the example does not do.

Mistakes that make a caller talk over, or wait in silence

  • Batch pipeline: waiting for the full reply before speaking.
  • Chasing the shortest endpoint until the agent cuts people off.
  • Ignoring barge-in so callers talk over a monologue.
  • Leaving the full reply in history after an interruption.
  • Replies written for reading: long, with lists and URLs.
  • Reading numbers as one long string.
  • No confirmation before irreversible actions.
  • Silence during slow tool calls.
  • Testing on clean studio audio instead of phone-quality recordings.
  • No AI disclosure or recording consent where required.

What the example does not do

About this example What the example does not do
  • No audio at all: every timing is a number drawn from an invented distribution, not a measurement.
  • No echo, background noise, codec or packet-loss modelling; the false barge-in model is a single duration threshold.
  • Endpointing is a silence threshold; real systems use learned end-of-turn detection.
  • The model reply is a fixed number of tokens; real replies and tool calls vary.
  • No concurrency or cost model: real voice systems pay per minute and handle many simultaneous calls.
  • No dialog management (clarification, repair, corrections such as "make that tomorrow").
  • Speaking rate is fixed at 165 words per minute, ignoring accent, voice and context.
  • Optional real experiment: record ten real calls with your stack, timestamp user-stop, first audio and each stage, and compute your own p50 and p95 gap and cut-in rate.

Check yourself

Where does the delay in a voice agent come from?

Show answer

The turn gap, the silence between the user stopping and the agent first audio, is the sum of stages. In the simulation a streaming pipeline had a mean gap of about 1.8 s, and endpointing at 600 ms was the largest single item.

For the curious What the user heard is what the history should say

When the user interrupts, stop playback and flush the audio buffer. If the history kept the full reply, it recorded on average 17 words as said that the user never heard; truncating the history at the interruption cut that to 0. Every stage latency here is an assumed round number, not a measurement of any product.

What you will use daily at work

Practice What it looks like
Budget the turn gap Stream every stage and tune endpointing on real calls.
Support barge-in Stop quickly, and record only what was heard.
Write for the ear Two sentences, then ask if they want the rest.
Confirm names, dates and amounts And disclose that it is an AI.