Latency: waiting for the first token and for the rest: Cost, latency and limits

Why a long input delays the start, a long answer lengthens the stream, and an agent adds up every step.

Time to first token (TTFT)

The wait between sending a request and seeing the first token. It covers the prefill phase, where the model reads the whole input, so it grows with input length.

Time per output token (TPOT)

The time to produce each further token. The total output time is the number of tokens times the TPOT: at 100 ms per token, a 1,000-token answer takes 100 seconds.

The example has a function for this. Its numbers (0.3 s base wait, 5,000 input tokens read per second, 0.02 s per output token) are placeholders I chose, not measurements of any model. For a 2,000-token input and a 100-token reply it gives a first token at 0.7 s and a complete answer at 2.7 s, so the output dominates. For a 40,000-token input and the same reply, the first token takes 8.3 s and the whole answer 10.3 s, so now the input dominates.

A long input mostly makes you wait before anything appears; a long answer makes the stream last.

What helps, by cause:

Slow because of	Try
Long input (large TTFT)	Shorter prompts, retrieval instead of pasting everything, prompt caching
Long output (large output time)	Ask for concise output, set a maximum, use structured fields instead of prose
Many sequential steps	Run independent tool calls in parallel (Level 0, module 4), reduce the number of steps
Model size	A smaller model for easy steps
User perception	Stream the tokens, so the user sees progress

For an agent the user waits for the sum of every step: ten steps of four seconds each is forty seconds. Plan for this in the design, for example by showing what the agent is doing.

Check yourself

Which of TTFT and output time grows when you paste a long document into the prompt?
Why does streaming not shorten the total time but still helps users?