CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 6 of 8 · 2 min read

Cascading failures

A failure that raises the chance of the next failure.

A cascading failure happens when a part of a system fails and, by doing so, raises the chance that other parts fail. UDS says failures spread "like cancer" from one process to the next until the whole system falls.

The book's story: clients query two database replicas, A and B, behind a load balancer. Each handles about 50 transactions per second (TPS). B loses its network link. The load balancer sees this and removes B. Now A takes all 100 TPS, twice its usual load. A struggles. Clients see errors and timeouts and retry the same requests, adding more load. A becomes too slow and the load balancer removes it too. Meanwhile B comes back, is put into the pool, and is hit by the whole flood at once, so it dies immediately. The cycle can repeat. UDS calls it a feedback loop of doom.

A worked example (made up to show the idea). Suppose each replica can handle at most 70 TPS.

Moment Load on A Load on B What happens
Normal 50 50 Both under 70
B removed 100 0 A is over its limit of 70
Clients retry up to 3 times up to 300 0 Even more load on a struggling A
A removed, B returns 0 all load B is flooded and fails at once

The book's advice is blunt: cascades are very hard to bring under control once begun, so the best defence is to prevent them. Page 5 returns to this with causes and a plan for the middle of one.

Cascading failure

A cascading failure is a chain reaction: one part fails, that raises the load or the risk on the next part, and the next part fails.

Example: one of two replicas dies, the other takes double load and dies too.

Check yourself: 1. In the story, why does bringing B back make things worse? 2. Which two actions by clients make A's load higher? 3. Why does removing a failed server from a pool sometimes cause the next failure?