Part 3 of 9 · 1 min read
Faults and failures
The difference between a part going wrong and the system letting users down.
Think of a car with a spare tyre. A puncture is a problem with one part. If the driver changes the tyre and continues the trip, the journey did not fail. If the driver has no spare, it does.
Fault
A fault is one part of a system going wrong, such as a disk dying, a program having a bug, or a person making a typing mistake.
Example: one server loses power.
Fault tolerance
A system is fault tolerant when it keeps working even though some faults occur. Fault tolerance means preventing faults from turning into failures.
Example: the second server takes over, and users see no error.
A failure is when the whole system stops giving users what it promised. A fault can happen in any part at any moment. The aim is not zero faults, because that is impossible. The aim is to stop faults from becoming failures.
A useful habit is to cause faults on purpose in a safe setting, such as stopping a server and watching what happens. Teams do this to learn whether recovery really works. Many serious outages are caused by error-handling code that was never run until the day it was needed.
Check yourself:
- A single disk fails and the website keeps working. Is that a fault, a failure, or both?
- Why is aiming for "no faults at all" a poor design goal?