Part 2 of 8 · 2 min read
Anatomy of a cascade
A trigger, an amplifier and a feedback loop.
Think of a row of dominoes where each one is a little heavier than the last. A light tap starts it, and the falling grows stronger. A cascade has three parts.
- A trigger: a small first failure. A network fault, a bad release, a slow dependency, a traffic spike.
- An amplifier: a mechanism that turns one failure into more load or more failures elsewhere.
- A feedback loop: the amplified problem makes the trigger worse or brings it back.
DDIA counts cascading failures among software faults: one component's small fault sets off a fault in a second part, which sets off more. It adds that the bugs behind them often lie dormant until unusual circumstances show that the software was assuming something about its environment that stopped being true.
The SRE book gives a worked picture (numbers from the book). A frontend cluster handles 1,000 requests per second. Its twin cluster fails, and the load rises to 1,200. The frontends cannot cope, so they crash or miss deadlines, and the work they finish falls below 1,000 per second. The load balancer then sends the extra load to other clusters, and the overload spreads. The book lists common triggers: process death, a change pushed to many tasks at once, a new rollout, and organic growth that nobody matched with capacity. Nygard adds that cascading failures are the main crack accelerator, and that any resource pool must limit how long a thread waits to borrow from it.
UDS's own story (page 1) has all three. The trigger is replica B losing its network link. The amplifiers are the load balancer, which removes B and gives A double load, and the clients, which retry. The feedback loop is that when B returns, it is flooded and dies, so the cycle repeats.
Feedback loop
A feedback loop is a cycle where the result of a change makes the cause of the change stronger.
Example: a server is slow, so clients retry, so the server gets more load, so it gets slower.
Check yourself: 1. Name the trigger, one amplifier and the loop in UDS's two-replica story. 2. Why do dormant bugs matter for cascades? 3. Give your own example of a feedback loop in a computer system.