Part 4 of 9 · 2 min read
Where faults come from
Hardware, software and human faults, with numbers that show why hardware faults are routine at scale.
Hardware faults. Disks crash, memory fails, power goes out, someone unplugs the wrong cable. The book reports that a hard disk has a mean time to failure of about 10 to 50 years. That sounds like forever. But at scale, it is not.
Mean time to failure
Mean time to failure, or MTTF, is the average time a part works before it fails. It describes many parts together. It does not predict when one disk will die.
Example: an MTTF of 30 years means that, among many disks, each one fails on average once in 30 years.
A worked example. The numbers are rounded.
- One year has about 365 days. 27 years is about 10,000 days.
- A disk with an MTTF of 27 years fails once in about 10,000 days on average.
- A cluster of 10,000 such disks has 10,000 chances per day for one failure each, with each disk failing once per 10,000 days.
- So the expected number of dead disks per day is 10,000 ÷ 10,000 = 1.
At that size, a failed disk is an ordinary daily event. Design for it. The usual answer is redundancy: keep spare copies, so a lost part is replaced from a copy.
Software faults. A bug can lie hidden for years and then show up under a rare input. A bug in one part can also spread, such as a process that uses all the memory on a machine and starves its neighbours. Software faults often hit many machines at once, because the same code runs on all of them. Careful testing, limits on resource use and checks that run while the system runs all help.
Human faults. People build and run systems, and people make mistakes. The book cites a study where configuration errors by operators were a leading cause of outages, while hardware faults played a much smaller part. Good designs make mistakes less likely and less harmful:
- Simple interfaces that make the right action the easy action.
- A safe practice area where people can try things without touching real data.
- Quick ways to undo a change, and a slow, careful way to roll out a risky one.
- Clear monitoring that shows the first signs of trouble.
Redundancy
Redundancy means keeping extra copies of a part or of data, so that if one fails another takes its place.
Example: two servers behind one front door instead of one.
Common mistake
Treating reliability as only a hardware topic. Hardware faults are loud, but software and human faults are often the larger source of real outages.
Check yourself:
- A cluster has 20,000 disks, each with an MTTF of 20 years (about 7,300 days). About how many fail per day?
- Name two ways to make a human mistake less harmful.