Part 6 of 8 · 2 min read
Cascading failures
A failure that raises the chance of the next failure.
A cascading failure happens when a part of a system fails and, by doing so, raises the chance that other parts fail. UDS says failures spread "like cancer" from one process to the next until the whole system falls.
The book's story: clients query two database replicas, A and B, behind a load balancer. Each handles about 50 transactions per second (TPS). B loses its network link. The load balancer sees this and removes B. Now A takes all 100 TPS, twice its usual load. A struggles. Clients see errors and timeouts and retry the same requests, adding more load. A becomes too slow and the load balancer removes it too. Meanwhile B comes back, is put into the pool, and is hit by the whole flood at once, so it dies immediately. The cycle can repeat. UDS calls it a feedback loop of doom.
A worked example (made up to show the idea). Suppose each replica can handle at most 70 TPS.
| Moment | Load on A | Load on B | What happens |
|---|---|---|---|
| Normal | 50 | 50 | Both under 70 |
| B removed | 100 | 0 | A is over its limit of 70 |
| Clients retry up to 3 times | up to 300 | 0 | Even more load on a struggling A |
| A removed, B returns | 0 | all load | B is flooded and fails at once |
The book's advice is blunt: cascades are very hard to bring under control once begun, so the best defence is to prevent them. Page 5 returns to this with causes and a plan for the middle of one.
Cascading failure
A cascading failure is a chain reaction: one part fails, that raises the load or the risk on the next part, and the next part fails.
Example: one of two replicas dies, the other takes double load and dies too.
Check yourself: 1. In the story, why does bringing B back make things worse? 2. Which two actions by clients make A's load higher? 3. Why does removing a failed server from a pool sometimes cause the next failure?