Part 7 of 8 · 2 min read
The review
Turn the outage into a better design.
Reviewing a cascade
After a cascade, ask a standard set of questions.
- What was the trigger? Was it expected?
- Which amplifiers made it grow? Use the table in section 2.
- Which defence should have cut the chain? Was it present, configured, or tested?
- What was the blast radius, and could it be smaller?
- How long did detection, mitigation and recovery take?
BM adds a point on culture. He describes Netflix's blameless approach and the practice of deliberately causing failures in production (for example with its Chaos Monkey tool, which switches off random machines) so that developers must design for them. Learning from a failure, without blame, is part of resilience. Page 6 covers testing this.
Interview angle
Answer in your head first, then open it.
What is a cascading failure?
A failure that increases the load or risk on other parts, so they fail too. It often has a trigger, amplifiers such as retries and load rebalancing, and a feedback loop.
How do you prevent one?
Timeouts, backoff with jitter, retry at one level, circuit breakers, bulkheads, load shedding, rate limits, headroom and tests of the combination.
What do you do during one?
Stop the feedback (retries), shed load, degrade features, avoid flooding recovering nodes, and restore gradually.
Why can automatic failure detection make it worse?
A slow node is declared dead, its load moves to others, and they slow down in turn.
Check yourself: 1. Which question in the review checks the defences? 2. What did Netflix do on purpose to learn from failure? 3. Why does blameless review matter?
Words you will meet
| Word | Meaning |
|---|---|
| Trigger | The first small failure |
| Amplifier | A mechanism that spreads a failure |
| Feedback loop | A cycle where the result strengthens the cause |
| Headroom | Spare capacity above the normal load |
| Failure detector | A mechanism that decides whether a node is dead |
| Rebalancing | Moving data or work between nodes |
| Gradual recovery | Restoring traffic in small steps |
| Blameless review | Learning from a failure without blaming people |
| Chaos Monkey | Netflix's tool that switches off random machines |