Part 2 of 5 · 1 min read
Partial failures
On one machine, a crash takes everything down together; across many machines, some can fail while others keep running normally.
Think of a single office building losing power entirely during an outage. Compare that to a city where one neighborhood loses power while every other neighborhood continues on as if nothing happened.
In plain words: a partial failure is a failure affecting only some parts of a distributed system while the rest continue operating normally, a situation that cannot occur on a single machine, where a crash typically takes the entire machine down at once. A distributed system must handle the possibility that any individual node, or any individual network link, might fail at any moment while the rest of the system keeps running, and it often cannot immediately tell whether a given node has actually failed or is simply slow to respond. This uncertainty, not knowing whether something failed or is merely delayed, is the single hardest problem a distributed system must contend with.
A worked example. A 10-node cluster loses one node to a hardware failure. The other 9 nodes continue serving reads and writes normally, but any client or node that was waiting on a response from the failed node must now decide, with no way to be fully certain, whether that node crashed or is simply slow.
The key lines. No new SQL; detecting and reacting to a partial failure is handled by the distributed database's own internal failure-detection and recovery mechanisms.
Partial failure
A partial failure is a failure affecting only some parts of a distributed system while the rest continue operating normally, a situation impossible on a single machine, where a crash typically affects the whole machine at once.
Example: one neighborhood losing power while the rest of a city keeps running normally is a partial failure.