CSE Roadmap

Specialist tier · about 3 hours · 12 parts

Part 2 of 12 · 2 min read

Every stage of the pipeline adds to the silence the caller hears

A reader tolerates a paragraph loading slowly; a listener notices a half-second of dead air.

Think of a phone call where the other person pauses before answering. A short pause reads as thinking, a long one reads as "did the call drop?" There's no progress bar to reassure you it's still working.

In plain words: the turn gap is the silence between the caller finishing and the agent's first audio, and it's the sum of several stages. A text-based chat agent can hide slow stages behind a spinner. A voice agent cannot hide them at all.

A worked example. Waiting for the whole reply before speaking it (a batch pipeline) gives a median turn gap of 2,399 ms and a p95 of 2,885 ms. Starting to speak as soon as the first sentence is ready (streaming) cuts that to 1,764 ms / 2,268 ms. Pairing streaming with a model that starts generating in 250 ms instead of 450 brings it down further, to 1,564 ms / 1,861 ms — streaming alone saved more than the faster model did.

The key lines. From code/mini-voice/latency.py:

# from code/mini-voice/latency.py
if c["streaming"]:
    gen = c["first_sentence_tokens"] / c["tokens_per_s"] * 1000
    speak = tts_first
else:
    gen = c["reply_tokens"] / c["tokens_per_s"] * 1000
    speak = c["tts_full_ms"]
parts = dict(endpoint=c["endpoint_ms"], stt_tail=c["stt_tail_ms"], llm=ttft + gen, tts=speak, network=c["network_ms"])

Read it as: streaming only has to wait for the first sentence's worth of tokens and the first chunk of audio, not the whole reply. The gap is literally the sum of five named stages. That's why fixing one stage at a time has a visible, measurable effect.

Turn gap, endpointing

The turn gap is the silence between the caller finishing speaking and the agent's first audio. Endpointing is the decision that the caller has finished their turn, usually from a silence threshold or a learned end-of-turn model.

Example: the endpoint_ms term in the code above is the single largest named stage in the 1.8-second streaming gap.