Part 2 of 6 · 2 min read
Partial failure
One computer works or breaks; many computers break in pieces.
Think of a single lamp and a string of fairy lights. The lamp is on or off. A string of lights can have a few bulbs out while the rest shine, and from far away it is hard to tell which.
The book explains that on one computer a fault usually becomes a total crash. This is a deliberate design choice: a crash is easier to handle than a wrong answer. Hardware hides the messy physical world and presents an idealised machine that behaves the same way every time.
With several computers joined by a network, that comfort is gone. Some parts may be broken in unpredictable ways while others work. The book calls this a partial failure. It is also nondeterministic [not the same each time]: an action involving several machines may work sometimes and fail at other times, and the sender may not even know which happened.
Partial failure
A partial failure is a state in which some parts of a distributed system are broken in unpredictable ways while other parts keep working.
Example: three of five servers answer normally, one answers slowly, and one has lost its network cable. The system as a whole is neither up nor down.
Why this matters for design, per the book:
- Supercomputers deal with partial failure by letting it become a total failure: they stop the whole job and restart from a saved checkpoint. Internet services cannot stop, since users expect an answer at any time.
- Cloud services use commodity machines, which fail more often. In a system with thousands of nodes, something is always broken.
- A system that survives failed nodes can be upgraded one node at a time (a rolling upgrade) with no downtime.
The goal is a reliable system built from unreliable parts, which is an old idea. TCP is the book's example: IP can drop, delay, duplicate and reorder packets, and TCP on top hides loss, duplicates and reordering. It cannot hide delay. A system can be more reliable than its parts, but only up to a limit.
The book advises suspicion: consider even unlikely faults, and create them on purpose in testing.
A worked example. Suppose each of 1,000 servers is down for a total of 0.1% of the time, independently (made up to show the idea). The chance that all 1,000 are up at one moment is 0.999^1000, which is about 0.37. So on average, more than six in ten moments have at least one broken server. Fault handling is a normal part of running the system, not a rare event.
Check yourself:
- Why is "either works or crashes" a comfort that a distributed system lacks?
- Why does a rolling upgrade need a system that tolerates failed nodes?