CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 5 of 12 · 2 min read

Buffering and backpressure: the server's own slowdowns

Two separate ways a stream can stall even when the network is fine.

Think of a water pipe with an air pocket. Nothing comes out at the tap for a while even though water is flowing in. Then the pocket fills and bursts through all at once.

In plain words: a proxy that buffers data before forwarding it can hide a live stream behind a delay. A server that queues events for a slow client without limit can run out of memory. Both need an explicit limit.

A worked example (buffering). Events are small (40 bytes) and frequent (20 per second). With no buffering, the first one reaches the browser in about 0.05 seconds. A proxy that buffers 4 KiB waits for roughly 102 events before releasing anything, delaying the first byte to 5.15 seconds. A 16 KiB buffer delays it to 10 seconds — here, the whole stream, so nothing streams at all.

A worked example (backpressure). A client stalls while 99 events queue up behind a cap of 10. Dropping the oldest token-delta events keeps the queue bounded and still delivers the one approval request untouched (89 dropped, approval kept). Blocking the producer instead refuses the same 89 pushes. Disconnecting closes the connection outright, counting on Last-Event-ID to resume it later.

# from code/mini-stream/sse.py
def buffered_first_byte(bytes_per_event, events_per_s, buffer_bytes, total_events):
    needed = -(-buffer_bytes // bytes_per_event)           # ceil
    k = min(needed, total_events)
    return k / events_per_s

class BoundedQueue:
    def push(self, ev):
        if len(self.q) < self.cap: self.q.append(ev); return True
        if self.policy == "drop_oldest_delta":
            for i, e in enumerate(self.q):
                if e["type"] == "delta": del self.q[i]; self.dropped += 1; self.q.append(ev); return True
            self.q.append(ev); return True                 # never drop non-delta events
        if self.policy == "block": self.blocked += 1; return False
        self.closed = True; return False

Read it as: a buffering proxy only releases data once enough of it has piled up. A bounded queue has to pick, per event type, what happens once it is full. The two problems look similar from the outside, because nothing seems to be happening. But they have different fixes. For the proxy, disable response buffering. For the queue, choose an explicit drop or block policy on the server.

Backpressure

Slowing, limiting or dropping what a producer sends when the consumer reading it cannot keep up.

Example: BoundedQueue(10, "drop_oldest_delta") drops old text deltas first, but never an approval request, once the queue of 10 is full.