CSE Roadmap

Core tier · about 4 hours · 8 parts

Part 7 of 8 · 2 min read

Practising disaster

A plan that has not been run is a guess.

Practising disaster

A plan on paper fails in ways nobody foresaw: a missing password, an out-of-date runbook, a dependency that exists only in the failed region. The practice comes from large operators. BM describes Google's annual DiRT (Disaster Recovery Test) exercises, which simulate large-scale disasters such as earthquakes, and Netflix, which runs failure-causing programs in production every day, including one that takes out a whole availability centre. He also stresses a blameless culture, so that people report and learn from mistakes.

A checklist for a disaster recovery drill (common practice):

  1. List what must work after the failure, in order. Dependencies first.
  2. Write the steps so someone who did not write them can follow them.
  3. Run the drill with a timer. Record the real RTO and RPO.
  4. Fix what broke, and repeat.
  5. Keep copies of the runbook, the access keys and the contacts outside the system that failed.

Common mistake

Keeping the recovery tools inside the thing being recovered. If the runbook, the passwords or the deployment system live only in the failed region, nobody can start the recovery.

Interview angle

Answer in your head first, then open it.

How do you make a system survive a data centre outage?

Spread over zones or regions, replicate data, route traffic away from the failed site, and know what data can be lost.

Active-passive or active-active?

Active-passive is simpler and cheaper but has a failover step. Active-active has little downtime but needs data conflict handling and enough capacity at each site.

Is replication a backup?

No. Replicas copy mistakes and corruption. A backup is a separate, older copy that has been tested.

What are RTO and RPO?

How long you can be down and how much data you can lose. They decide the design and its cost.

Check yourself: 1. What does a drill measure that a document cannot? 2. Why keep the runbook outside the main system? 3. What did Google's DiRT exercise practise?

Words you will meet

Word Meaning
Redundancy More than one copy, so one can fail
Failure domain A set of things that can fail for the same cause
Availability zone A cloud provider's data-centre-like unit inside a region
Region A distinct geographic area of a cloud
SLA A promise about uptime, with limited remedies
Active-passive One site serves, one waits
Active-active Both sites serve at once
geoDNS DNS that answers by the user's location
Homing region Always sending a user's writes to one region
Backup A separate, older copy used to rebuild
RTO Longest accepted downtime
RPO Most accepted data loss, as a time