Part 4 of 8 · 2 min read
Load levelling
A queue between the callers and the service.
When clients do not need an answer right away, there is a gentler option. Place a queue between clients and service. The service takes work when it is ready, and clients do not push it faster than it can go. UDS calls this load levelling. It smooths short spikes.
The SRE book names two more tools. Load shedding drops a share of the traffic as a server nears overload, for example by answering 503 when too many requests are in flight. Graceful degradation goes one step further and does less work per request, such as searching only a cached part of the data. The book warns that degraded mode runs rarely, so it may be broken when needed. It advises running a few servers near overload on purpose to exercise it, and alerting when many servers enter it. (Level 6 covered queues and backlogs.)
Load levelling
Load levelling means putting a queue in front of a service so that the service takes work at its own pace.
Example: orders go into a queue and the billing service handles 50 per second, even if 500 per second arrive for a few seconds.
A worked example (made up). Billing handles 50 orders per second. A flash sale sends 500 per second for 10 seconds, which is 5,000 orders. The queue holds the extra: it grows by 450 per second to a peak of 4,500 orders. After the sale, with 100 orders per second arriving, the queue drains at 50 per second and is empty after 90 seconds. Customers wait a little but nothing is lost.
UDS notes that shedding and levelling do not raise capacity. They protect the service. To handle more load, the service has to scale out, and these patterns are usually combined with auto-scaling, which adds instances when the service is running hot.
Check yourself: 1. When can you use load levelling and not load shedding? 2. In the example, how large does the queue get? 3. Do these two patterns increase capacity?