Definitions

Definitions

The stream

Results

1. Stream progress, not just tokens, and give every event an id.
2. Resumption needs a durable event store, not memory.
3. Silence is a failure mode: heartbeat or the proxy will cut you.
4. Test through the real gateway: buffering and timeouts live there.

Design rules

• Default to SSE for server-to-client progress; use WebSockets when you truly need low-latency two-way interaction.
• Send typed events: delta, step, tool_call, approval_request, error, done, each with an id.
• Back the stream with a durable event store so any instance can resume it (module 3 and 1).
• Support Last-Event-ID and make clients idempotent to duplicate ids.
• Send heartbeats below the shortest idle timeout on the path; set retry for reconnect delay.
• Disable proxy buffering and compression on stream routes; flush after each event.
• Bound server buffers and choose a policy per event type: drop deltas, never drop approvals or final events.
• Provide cancel and approve as ordinary requests (module 1), not only messages on the stream.
• Authenticate per request and re-check authorisation on resume (module 7).
• Measure TTFT, time to first progress event, stall rate and reconnect rate.

Interview questions, with the follow-ups

Common mistakes

• Streaming tokens only, leaving tool waits silent.
• No event ids, so reconnect means restart or duplicates.
• Events held in server memory (lost on deploy or on another instance).
• One network chunk assumed to be one event in the parser.
• Proxy buffering left on, or compression that buffers.
• No heartbeats behind an idle-timeout proxy.
• Dropping approval events under backpressure.
• WebSockets chosen by default, adding session affinity for no need.
• Leaking sensitive intermediate content in progress events.

What the example does not do

• No real server, browser or proxy; buffering and timeouts are modelled, not observed.
• Infrastructure claims (session affinity, horizontal scaling, memory) come from general knowledge; I did not measure them.
• The parser covers the common fields only (no BOM handling or retry use).
• Event store is a list; durability, ordering across writers and retention are not modelled.
• Backpressure is a toy queue, with no real socket write timing.
• Optional real experiment (I did not run this): put your agent behind your real gateway, stream a run with a 90-second tool call, and record the first-byte time, the disconnect time with and without heartbeats, and the duplicate count on reconnect with and without Last-Event-ID.

For the curious A buffering proxy can hide the whole stream until it ends

With 40-byte events at 20 a second, a proxy that buffers 4 KiB delayed the first byte to 5.15 s, against 0.05 s unbuffered. A proxy that closes idle connections after 60 s closed a stream at 64 s while a tool ran silently, and a 15 s comment heartbeat kept it open. Those timeouts and sizes are my assumptions, so test through your real gateway.

What you must be able to say (one page)

• Stream progress, because agents are slow and silent request/response feels broken. In my simulation of a 60-second agent run, a silent response shows the user nothing for 60 s; streamed progress shows something at about 0.4 s. For agents the stream should carry progress events (step started, tool called, partial result), not only tokens.
• SSE or WebSockets? WebSockets are persistent two-way connections with state held in the server's memory (simple logic, but you need session affinity and the memory persists). Server-Sent Events (SSE) are one-way streaming over plain HTTP; with the context saved between interactions the app stays stateless, load-balances normally and can support approvals that wait for hours. A good default is SSE plus saved context, with WebSockets only for real-time two-way needs such as collaborative editing. The application design matters more than the protocol. In the SSE design, user input goes through a normal POST to the same or another endpoint.
• SSE framing is simple text. Fields id:, event:, data: (multiple data: lines join with newlines), retry:; an event ends at a blank line; lines starting with : are comments (use them as heartbeats). My parser is incremental: I fed every possible split point of a two-event stream (every chunk boundary, including mid-line) and it produced the same events. Parse by buffering until a newline, never assume one network chunk is one event.
• Make streams resumable. Give each event a monotonic id from a durable per-run event store (the journal of module 3 can double as it). On reconnect the browser sends Last-Event-ID and the server replays what was missed. In my test, a drop after 20 of 50 events: blind reconnect received 70 events (20 duplicates, so the UI would repeat text); Last-Event-ID received exactly 50, no duplicates, none missing.
• Heartbeats beat idle timeouts. With a proxy that closes idle connections after 60 s and a tool call that keeps the stream silent from 4 s to 95 s, the connection was closed at 64 s; a comment heartbeat every 15 s kept it open. The timeout and 15 s interval are my assumptions; find your real proxy's limits.
• Buffering proxies break streaming silently. With 40-byte events at 20 per second, a proxy that buffers 4 KiB delayed the first byte to 5.15 s (0.05 s unbuffered), and a 16 KiB buffer to 10 s, which here is the end of the stream because the whole stream fits in the buffer. Disable response buffering for stream routes, flush per event, and test through the real gateway.
• Backpressure for slow clients. With a stalled client and a queue of 10, three policies: drop old token deltas (89 dropped, the approval request kept; acceptable for text deltas, never for approval or final events), block the producer (89 pushes refused; fine when the run can pause), disconnect (the client resumes with Last-Event-ID). Choose per event type.
• Honest limits: no real HTTP, browser, proxy or network; assumed timeouts, rates and sizes.