What streaming changes: Streaming

Streaming does not make the model faster. It lets the reader start at the first token instead of the last.

Streaming

Instead of waiting for the whole reply, the server sends it in small pieces (often a few tokens each) while the model generates them. Your code receives a series of events. Most are text deltas (a delta is just the newest small piece of text), and the last one carries the token usage and the stop reason.

Level 1, module 6 split latency into the wait before the first token and the time to produce the rest. Streaming does not make the model faster. It lets the reader start when the first token arrives, instead of when the last one does.

The same 10.8 seconds of generation, but with streaming the reader begins at 0.8 s. Numbers computed by the example code for a 500-token answer, a 0.8 s wait and 20 ms per token.

The arithmetic: without streaming, the first and last text appear together at 0.8 + 500 x 0.02 = 10.8 seconds. With streaming, the first text appears at 0.8 s and the last at 10.8 s. The total is unchanged, and so is the cost.

Use streaming when a person is waiting and the answer is long. Skip it for short answers, for background jobs, and for output you must validate completely before using (module 3). You cannot validate a stream until it ends.

Check yourself

A reply has 800 tokens, a 1.2 s wait and 25 ms per token. When does the reader first see text with and without streaming? (Answers: 1.2 s with streaming, 21.2 s without. Both finish at 21.2 s.)