Part 5 of 8 · 2 min read
Backups
A copy that cannot be damaged by the same event.
Replication keeps live copies in step. That is its strength and its weakness. If a bad deployment deletes rows, or a bug writes wrong values, the replicas copy the mistake within seconds. DDIA notes this under durability: data on disk can slowly become corrupted without being noticed. If the damage has gone unseen for a while, the replicas and recent backups may be damaged too, and the only cure is an older backup. It also says perfect durability does not exist: if all disks and all backups are destroyed at once, nothing helps.
DDIA lists why none of the usual forms of durability is enough alone: a machine that dies makes data inaccessible even if it is not lost; a correlated fault, such as a power cut or a bug triggered by one input, can knock out all replicas at once; asynchronously replicated writes can be lost when the leader fails; SSDs and disk firmware can break their promises; and data can rot on disk.
So a backup is a separate, older copy. These rules are common practice, and not from the books used here:
- Keep several generations, not just the latest, so a slow corruption can be undone.
- Keep one copy away from the failure domain of the original (another region or provider) and out of reach of the same accounts, so one mistake or attacker cannot delete both.
- Test restores. A backup that has never been restored is a hope. Time the restore, since the time it takes decides your recovery time (next section).
Backup
A backup is a separate copy of data, kept for a time, that can be used to rebuild the system after loss or damage.
Example: a nightly snapshot of the database kept for 30 days in another region.
A worked example (made up). A faulty release corrupts prices at 14:00 and nobody notices until 16:30. Replicas already hold the bad data. Nightly backups from the last 30 days hold clean prices. The team restores yesterday's snapshot into a new database, replays the day's valid orders from the log, and fixes the prices. A system with replicas and no old backups would have no path to the clean data.
Check yourself: 1. Why does replication not protect against a bad deployment? 2. Why keep several generations of backups? 3. What is the only proof that a backup works?