Part 8 of 8 · 2 min read
Exercises
Easy
Easy 1: Nines to minutes (20 minutes)
Compute the downtime allowed per month (30 days) for 99 percent, 99.9 percent and 99.99 percent.
Done when: three numbers in minutes, with the working shown.
Hint
Minutes in a month is 30 times 24 times 60.
Easy 2: Find the SPOFs (20 minutes)
List ten components of a system you know. For each, say what happens if it fails and mark the single points of failure.
Done when: a table of ten rows with a SPOF column.
Hint
Ask "what happens if this is gone for an hour?"
Easy 3: Four silences (20 minutes)
For each of the four reasons a client hears nothing, say what the client can safely assume.
Done when: four rows, each with a safe assumption or "nothing".
Hint
The client cannot tell whether the work was done.
Medium
Medium 1: Leak arithmetic (30 minutes)
A service leaks 2 MB every 500 requests and has 16 GB of memory. Find how long it runs at 1 million and at 50 million requests a day before memory runs out.
Done when: two times in hours, with the working shown.
Hint
Leak per request is 2 MB divided by 500.
Medium 2: Trace a cascade (30 minutes)
Three replicas each have a limit of 120 TPS and a total load of 300 TPS. One fails. Trace the next five minutes, assuming clients retry twice.
Done when: a timeline table with loads and the point where the cascade starts or stops.
Hint
Remember that retries multiply the load.
Medium 3: Risk table (30 minutes)
Build a risk table with at least eight failures of a system you know. Score each and rank them.
Done when: a ranked table and a note on the top two.
Hint
Cheap fixes for high scores come first.
Hard
Hard 1: Pool exhaustion (45 minutes)
A service has a pool of 40 connections. A dependency starts hanging on 10 percent of calls and the service gets 200 calls per second. Estimate how fast the pool empties and what users see.
Done when: a calculation and a timeline marked as made up.
Hint
Calls stuck per second times the time they stay stuck.
Hard 2: Break the cascade (45 minutes)
Using your Medium 2 timeline, list three changes that would have stopped the cascade. Say which later page of this level covers each.
Done when: three changes, each with the new timeline.
Hint
Think of timeouts, limits and removing load.
Hard 3: Pre-mortem (50 minutes)
Pick a system and write a story of its worst outage in the first person, a year from now. List the failure causes from this page that appear in it.
Done when: a one-page story and a table tying each event to a cause.
Hint
Start from the end of the story and work back.