Part 8 of 8 · 2 min read
Exercises
Easy
Easy 1: Disk arithmetic (20 minutes)
A fleet of 4,000 disks has an MTTF of 20 years. How many failures per year and per week?
Done when: two numbers with the working.
Hint
Failures per year equal disks divided by MTTF in years.
Easy 2: Name the domain (20 minutes)
For each event, name the smallest failure domain it hits: a switch fails, a data centre loses cooling, a bad DNS change, a cloud account is locked.
Done when: four answers.
Hint
The smallest domain that explains all the lost parts.
Easy 3: RTO or RPO (20 minutes)
Decide whether each statement is about RTO or RPO: "back within 30 minutes", "lose at most 10 seconds of orders", "restore takes 3 hours", "replication lag is 2 seconds".
Done when: four labels.
Hint
Time down is RTO. Data lost is RPO.
Medium
Medium 1: Zone capacity (30 minutes)
A service has a peak of 9,000 requests per second and a safe per-server limit of 500. Compute servers needed to survive one zone loss with 2, 3 and 4 zones.
Done when: three numbers and a comment on the trend.
Hint
With N zones, the N−1 survivors must carry the peak.
Medium 2: Design from objectives (30 minutes)
Given an RTO of 20 minutes and an RPO of 1 minute, choose a design from the table in section 5 and justify it. Then do the same for an RTO of 1 day and an RPO of 1 hour.
Done when: two designs with reasons.
Hint
Fast RTO means the data is already at the second site.
Medium 3: Restore timing (30 minutes)
Estimate the restore time for a 5 TB database given your own made-up read speed, log replay time and DNS change time. Show where you could cut the time.
Done when: a time budget with three cuts.
Hint
Copy time is size divided by speed.
Hard
Hard 1: Two-region design (45 minutes)
Design a shop for two regions. Choose the routing, the data replication, the conflict plan, the failover trigger and the failback plan.
Done when: a design page with a failure walk-through.
Hint
Home each user to one region to avoid conflicts.
Hard 2: Corruption scenario (45 minutes)
A bug writes wrong totals for 3 hours before it is found. Using backups and the log, design the recovery and state the data you cannot recover.
Done when: a recovery plan with a loss statement.
Hint
Restore a clean backup, then replay only valid log entries.
Hard 3: Run a drill on paper (50 minutes)
Write a drill for losing a region, with a timeline of injects, expected actions, measures and a pass rule. Include the tools the team must reach from outside the failed region.
Done when: a drill script of about two pages.
Hint
Put the credentials and runbook somewhere that survives.