Part 8 of 8 · 2 min read
Exercises
Easy
Easy 1: Label the story (20 minutes)
Mark the trigger, the amplifiers and the feedback loop in BM's ad service story from page 3.
Done when: three labels with a quote of at most five words each.
Hint
The trigger started it. Amplifiers made it grow.
Easy 2: Headroom arithmetic (20 minutes)
Six servers each have a limit of 300 requests per second. What total load lets the system survive the loss of one? Of two?
Done when: two numbers with the working.
Hint
After losing one, the others must still be under the limit.
Easy 3: Match the defence (20 minutes)
For each amplifier in the section 2 table, name the tool that cuts it and the page where it appears.
Done when: a six-row table.
Hint
Reuse the prevention table.
Medium
Medium 1: Simulate a cascade (30 minutes)
Write a simple simulation of three replicas, a load balancer that removes failed ones and clients that retry. Show the cascade with and without a retry limit.
Done when: code and two output tables.
Hint
Make retries add to the arrival rate.
Medium 2: Failure detector trap (30 minutes)
A cluster of 5 nodes uses a 2 second timeout to declare a node dead. Describe how a 3 second pause (for example a garbage collection stop) in one node could cause a cascade, and give two changes.
Done when: a step list and two changes.
Hint
The pause looks like a crash from outside.
Medium 3: Recovery plan (30 minutes)
A database was down for 20 minutes. Write the order and the percentages for bringing back a cache, a service and the traffic.
Done when: a numbered plan with timings and checks.
Hint
Start with what others depend on.
Hard
Hard 1: Review a real outage (45 minutes)
Pick a public postmortem of a cascading outage. Answer the five review questions from section 6 using it.
Done when: five answers and a link to the source.
Hint
Public postmortems exist from many large providers.
Hard 2: Break the self-feeding loop (45 minutes)
Using section 5, design a mechanism that detects "load from retries exceeds capacity even though the cause is gone" and breaks the loop automatically.
Done when: a design with a signal, a threshold and an action.
Hint
Compare the arrival rate to the success rate.
Hard 3: Game day script (50 minutes)
Write a game day plan that tests three links of the chain in a staging system: what to break, what to measure, what you expect, and the stop rule.
Done when: a one-page plan with expected results.
Hint
A good stop rule limits harm to users.