Part 7 of 8 · 2 min read
Practising disaster
A plan that has not been run is a guess.
Practising disaster
A plan on paper fails in ways nobody foresaw: a missing password, an out-of-date runbook, a dependency that exists only in the failed region. The practice comes from large operators. BM describes Google's annual DiRT (Disaster Recovery Test) exercises, which simulate large-scale disasters such as earthquakes, and Netflix, which runs failure-causing programs in production every day, including one that takes out a whole availability centre. He also stresses a blameless culture, so that people report and learn from mistakes.
A checklist for a disaster recovery drill (common practice):
- List what must work after the failure, in order. Dependencies first.
- Write the steps so someone who did not write them can follow them.
- Run the drill with a timer. Record the real RTO and RPO.
- Fix what broke, and repeat.
- Keep copies of the runbook, the access keys and the contacts outside the system that failed.
Common mistake
Keeping the recovery tools inside the thing being recovered. If the runbook, the passwords or the deployment system live only in the failed region, nobody can start the recovery.
Interview angle
Answer in your head first, then open it.
How do you make a system survive a data centre outage?
Spread over zones or regions, replicate data, route traffic away from the failed site, and know what data can be lost.
Active-passive or active-active?
Active-passive is simpler and cheaper but has a failover step. Active-active has little downtime but needs data conflict handling and enough capacity at each site.
Is replication a backup?
No. Replicas copy mistakes and corruption. A backup is a separate, older copy that has been tested.
What are RTO and RPO?
How long you can be down and how much data you can lose. They decide the design and its cost.
Check yourself: 1. What does a drill measure that a document cannot? 2. Why keep the runbook outside the main system? 3. What did Google's DiRT exercise practise?
Words you will meet
| Word | Meaning |
|---|---|
| Redundancy | More than one copy, so one can fail |
| Failure domain | A set of things that can fail for the same cause |
| Availability zone | A cloud provider's data-centre-like unit inside a region |
| Region | A distinct geographic area of a cloud |
| SLA | A promise about uptime, with limited remedies |
| Active-passive | One site serves, one waits |
| Active-active | Both sites serve at once |
| geoDNS | DNS that answers by the user's location |
| Homing region | Always sending a user's writes to one region |
| Backup | A separate, older copy used to rebuild |
| RTO | Longest accepted downtime |
| RPO | Most accepted data loss, as a time |