CSE Roadmap

Core tier · about 4 hours · 8 parts

Part 2 of 8 · 2 min read

Why redundancy

At scale, a part is always breaking.

DDIA gives the arithmetic. A hard disk is rated at 10 to 50 years of mean time to failure (MTTF). That sounds safe. But in a storage cluster of 10,000 disks, you should expect on average one disk to die every day. The same maths holds for machines, power supplies and network links. (Page 1 called this the cruel maths.)

The first answer is redundancy at the level of parts: disks in a RAID setup, servers with two power supplies and hot-swappable CPUs, data centres with batteries and diesel generators. When one part dies, the spare takes its place while the broken one is replaced. DDIA says this does not prevent all hardware problems but is well understood and can keep a machine running for years.

DDIA notes the change of era. Once, part-level redundancy was enough, because a backup could be restored onto a new machine and the downtime was not catastrophic. Now applications use many machines, which raises the rate of faults in proportion. Cloud platforms such as AWS often let virtual machines vanish without warning, since they prefer flexibility and elasticity to the reliability of single machines. So whole-machine redundancy is now needed by many more systems.

Redundancy

Redundancy means keeping more than one copy of a part or of the data, so that one can fail without the system stopping.

Example: three web servers behind a load balancer, with room on the other two to carry the load of the third.

Redundancy is not free. It costs money, and it adds work: copies must be kept in step (Level 4), and the switch to the spare (failover) must itself work. Xu's first chapter lists the steps for a web system: a load balancer in front of several servers for failover, and database replication for redundancy, so that if a database server is destroyed, for example by a natural disaster, data is preserved.

Check yourself: 1. In a cluster of 10,000 disks with a 27-year MTTF, how many disks fail per year, roughly? 2. Why did machine-level redundancy become common only recently? 3. What does a load balancer add to a group of web servers?