CSE Roadmap

Common tier · about 3 hours · 8 parts

Part 7 of 8 · 2 min read

The review

Turn the outage into a better design.

Reviewing a cascade

After a cascade, ask a standard set of questions.

  1. What was the trigger? Was it expected?
  2. Which amplifiers made it grow? Use the table in section 2.
  3. Which defence should have cut the chain? Was it present, configured, or tested?
  4. What was the blast radius, and could it be smaller?
  5. How long did detection, mitigation and recovery take?

BM adds a point on culture. He describes Netflix's blameless approach and the practice of deliberately causing failures in production (for example with its Chaos Monkey tool, which switches off random machines) so that developers must design for them. Learning from a failure, without blame, is part of resilience. Page 6 covers testing this.

Interview angle

Answer in your head first, then open it.

What is a cascading failure?

A failure that increases the load or risk on other parts, so they fail too. It often has a trigger, amplifiers such as retries and load rebalancing, and a feedback loop.

How do you prevent one?

Timeouts, backoff with jitter, retry at one level, circuit breakers, bulkheads, load shedding, rate limits, headroom and tests of the combination.

What do you do during one?

Stop the feedback (retries), shed load, degrade features, avoid flooding recovering nodes, and restore gradually.

Why can automatic failure detection make it worse?

A slow node is declared dead, its load moves to others, and they slow down in turn.

Check yourself: 1. Which question in the review checks the defences? 2. What did Netflix do on purpose to learn from failure? 3. Why does blameless review matter?

Words you will meet

Word Meaning
Trigger The first small failure
Amplifier A mechanism that spreads a failure
Feedback loop A cycle where the result strengthens the cause
Headroom Spare capacity above the normal load
Failure detector A mechanism that decides whether a node is dead
Rebalancing Moving data or work between nodes
Gradual recovery Restoring traffic in small steps
Blameless review Learning from a failure without blaming people
Chaos Monkey Netflix's tool that switches off random machines