CSE Roadmap

Core tier · about 4 hours · 6 parts

Part 2 of 6 · 2 min read

Partial failure

One computer works or breaks; many computers break in pieces.

Think of a single lamp and a string of fairy lights. The lamp is on or off. A string of lights can have a few bulbs out while the rest shine, and from far away it is hard to tell which.

The book explains that on one computer a fault usually becomes a total crash. This is a deliberate design choice: a crash is easier to handle than a wrong answer. Hardware hides the messy physical world and presents an idealised machine that behaves the same way every time.

With several computers joined by a network, that comfort is gone. Some parts may be broken in unpredictable ways while others work. The book calls this a partial failure. It is also nondeterministic [not the same each time]: an action involving several machines may work sometimes and fail at other times, and the sender may not even know which happened.

Partial failure

A partial failure is a state in which some parts of a distributed system are broken in unpredictable ways while other parts keep working.

Example: three of five servers answer normally, one answers slowly, and one has lost its network cable. The system as a whole is neither up nor down.

Why this matters for design, per the book:

  • Supercomputers deal with partial failure by letting it become a total failure: they stop the whole job and restart from a saved checkpoint. Internet services cannot stop, since users expect an answer at any time.
  • Cloud services use commodity machines, which fail more often. In a system with thousands of nodes, something is always broken.
  • A system that survives failed nodes can be upgraded one node at a time (a rolling upgrade) with no downtime.

The goal is a reliable system built from unreliable parts, which is an old idea. TCP is the book's example: IP can drop, delay, duplicate and reorder packets, and TCP on top hides loss, duplicates and reordering. It cannot hide delay. A system can be more reliable than its parts, but only up to a limit.

The book advises suspicion: consider even unlikely faults, and create them on purpose in testing.

A worked example. Suppose each of 1,000 servers is down for a total of 0.1% of the time, independently (made up to show the idea). The chance that all 1,000 are up at one moment is 0.999^1000, which is about 0.37. So on average, more than six in ten moments have at least one broken server. Fault handling is a normal part of running the system, not a rare event.

Check yourself:

  • Why is "either works or crashes" a comfort that a distributed system lacks?
  • Why does a rolling upgrade need a system that tolerates failed nodes?