Part 3 of 8 · 3 min read
Load levelling and backlogs
A queue turns a spike in arrivals into a longer, steadier stream of work.
Think of a supermarket checkout. At noon fifty people arrive at once, but the cashier serves one person at a time. The line grows, then shrinks. Nobody is turned away, but some wait. The line absorbed the spike.
The UDS book names this load levelling. A channel is placed between clients and the service. The service no longer has load pushed at it. It pulls work from the channel at its own pace. The book says this suits clients that do not expect an answer within a short time, and that it handles short spikes well because the channel smooths them out.
Load levelling
Load levelling is using a queue so that a service processes work at its own steady rate, even when requests arrive in bursts.
Example: 1,000 sign-ups arrive in one minute, but the welcome-email service can send 400 per minute. The queue holds the extra 600 and the service works through them.
The book points out that levelling does not add capacity. It protects the service from being overloaded. To handle more load for a long time, the service must be scaled out, so levelling is usually combined with automatic scaling that adds consumers when the service runs hot.
A worked example (made up). A shop normally gets 200 orders per second. A sale pushes this to 1,000 per second for 60 seconds. Consumers can process 400 per second.
| Quantity | Calculation | Result |
|---|---|---|
| Spike arrivals | 1,000 x 60 | 60,000 messages |
| Work done during the spike | 400 x 60 | 24,000 messages |
| Backlog at the end of the spike | 60,000 - 24,000 | 36,000 messages |
| Net drain rate afterwards | 400 - 200 | 200 per second |
| Time to drain | 36,000 / 200 | 180 seconds |
So the queue absorbs a one-minute spike, and the system is back to normal after three more minutes. Without the queue, the service would have had to survive 1,000 per second directly. It could only do 400, so most requests would fail.
Backlog
A backlog is the number of messages in a queue that have not been processed yet.
Example: the backlog above peaks at 36,000 messages and falls to zero after 180 more seconds.
Two useful measures follow from this. The queue depth is the backlog size. The waiting time of a new message is roughly the backlog divided by the processing rate. With 36,000 waiting and 400 processed per second, a message added at the peak waits about 90 seconds (36,000 / 400), if no more arrive.
The DDIA book lists what a system can do when producers send faster than consumers can handle:
- Drop messages.
- Buffer them in a queue.
- Apply backpressure, which means blocking the producer from sending more.
Backpressure
Backpressure (also called flow control) is slowing or blocking the sender when the receiver cannot keep up.
Example: a Unix pipe has a small fixed buffer. When it is full, the sender is paused until the reader takes data out.
A queue is option two. Message brokers (servers that run queues) usually allow the queue to grow without a fixed limit, though this depends on the configuration. The book warns what happens if the queue grows large. It may no longer fit in memory, so the broker writes to disk, and then each message takes longer, and the overall throughput may drop. Most brokers assume that queues stay short.
Dropping is fine for some data. The book gives sensor readings: an occasional missing point is not important, because a new value follows soon. But if events are counted, every lost message makes the count wrong.
Common mistake
Treating a growing queue as harmless because nothing is failing. A backlog that never drains means the consumers are slower than the producers for good. The queue only hides the problem until memory or disk runs out or the delay becomes too long.
Check yourself:
- A queue has 6,000 messages and the consumers handle 100 per second with no new arrivals. How long until it is empty?
- Why does load levelling not replace scaling out?