Part 3 of 7 · 2 min read
Consuming a stream
Cancel it, survive a dropped connection, and decide what to do with a partial answer.
result = consume(stream, on_delta=print, should_cancel=lambda text: stop_button_pressed)
result.text, result.finished, result.cancelled, result.error
The consume function does five things that every streaming client needs:
- Append each text piece to a buffer and pass it on (to the screen, a log or a websocket).
- Record the final event with usage, so you can log cost.
- Support cancellation. If the user presses stop, or you already have what you need, close the stream so the connection ends. A test checks that the underlying generator really is closed.
- Catch connection errors and return the partial text together with the error, instead of losing what arrived.
- Report whether it finished.
finishedis true only if the final event arrived.
Real output of the three cases for the text "Streaming shows text as soon as it exists.":
| Case | Text received | finished |
Other |
|---|---|---|---|
| Normal | the whole sentence (11 pieces) | True | usage taken from the final event |
| Cancelled once 12 characters had arrived | Streaming sh |
False | cancelled is True, 3 tokens counted |
| Connection dropped after 3 pieces | Streaming sh |
False | error is "connection dropped" |
Cancelling still costs money
Tokens generated before you cancel are normally billed, and some providers may keep generating for a moment after you disconnect. In the example, stopping at 12 characters still counts 3 tokens. That number is an estimate (about 4 characters per token) because a cancelled stream never sends its final usage event. A real bill will not match it exactly. Do not use cancel as a cost control after the fact. Set max_tokens (module 1) before the call.
A half-finished answer is not an answer
If the stream stops early (network drop, timeout, cancel), result.text is only a prefix of the answer. What you do with it depends on the use:
| Use | What to do with a partial reply |
|---|---|
| Chat shown to a person | Show it, mark it as interrupted, and offer retry or continue |
| JSON for code to read | Discard it and retry the call (module 5) |
| Tool call arguments | Never run a tool from partial arguments |
| Text saved to history | Save it flagged as incomplete, or do not save it |
Resuming a stream in the middle is rarely supported. The usual choices are to retry from the start, or to send the partial text back as an assistant message and ask the model to continue. The second needs care, so that the model does not repeat or contradict what the person has already read.
Check yourself
A stream fails before any text arrives. What should output_tokens_billed say in the example, and why is that still only an estimate? (It says 0, because nothing was received. The provider may still have started generating, so check the real invoice or usage report.)