Part 5 of 8 · 2 min read
In the middle of one
Reduce load first, then restore slowly.
UDS says a cascade is very hard to bring under control once it has started. The books do not give a full incident procedure, so this section applies their ideas and marks the rest as common practice.
What follows from the books:
- Cut the feedback loop. Retries feed the loop. If clients can be told to stop or back off (a
Retry-Afterheader, page 4), do so. Open circuit breakers by hand: BM notes that a breaker can be blown manually to make callers fail fast while a service is offline. - Shed load. Reject low-priority work so the service can finish what it has (page 4). Drop optional features (graceful degradation).
- Do not remove a struggling node just because it is slow. DDIA's warning about declaring nodes dead applies. Moving its load to others can finish the job.
- Do not let the cure flood the patient. In UDS's story, when B comes back it is hit by the entire load and falls over again. A recovering instance should be given traffic in small steps.
Common practice, not from the books used here:
- Roll back first, investigate later. If a release started the cascade, roll it back.
- Add capacity only if the bottleneck is capacity. More servers do not help if the cause is a slow dependency or a lock. They may add more load on it.
- Restore in order. Bring back the dependencies first (database, cache), then the services that use them, then the traffic, in steps such as 10, 25, 50 and 100 percent.
- Write down the timeline as you go. It feeds the review afterwards.
A worked example (made up). A service has 4 instances at 70 percent load. A slow database makes calls take 10 times longer. Threads fill up, health checks fail and two instances are removed. The two left receive double traffic and fail. Response: (1) turn on shedding of the low-priority endpoints, (2) tell the gateway to cut retries, (3) open the breaker to the database for the non-critical reads, (4) when the database recovers, restore instances one at a time, each taking traffic in 25 percent steps.
Gradual recovery
Gradual recovery means bringing traffic back to a recovered service in small steps, and not all at once.
Example: the load balancer sends 10 percent of normal traffic to a restarted instance, then 25, then 50, then 100 as its error rate stays low.
Check yourself: 1. Why should retries be stopped early in a cascade? 2. Why is it wrong to add servers when the database is the bottleneck? 3. What does gradual recovery protect against?