Part 3 of 8 · 2 min read
One server to many
Why distribution is forced, and what it costs.
A key-value store on one machine is easy: a hash table in memory. The book offers two ways to fit more in one machine: compress the data, and keep only the often-used pairs in memory with the rest on disk. Even so, one machine soon runs out of room. For big data, the store must be spread across many servers, which makes it a distributed hash table.
Spreading data brings the CAP theorem. In a distributed system, three properties are wanted:
- Consistency: every client sees the same data at the same time, whichever node it asks.
- Availability: every request gets an answer, even when some nodes are down.
- Partition tolerance: the system keeps working when the network between nodes breaks.
The theorem says a system cannot provide all three at once. Networks do fail, so partition tolerance is not optional. The real choice is between consistency and availability when a partition happens. The book describes:
- CP systems choose consistency. They refuse some requests during a partition. A bank balance is the example, since an out-of-date balance is worse than an error.
- AP systems choose availability. They keep answering, possibly with stale data, and catch up after the partition heals.
- A "CA" system cannot exist in practice, since partitions cannot be avoided.
An example with three copies, n1, n2 and n3. The link to n3 breaks. A CP store blocks writes on n1 and n2 so the copies cannot disagree, and so is unavailable for writes. An AP store keeps accepting writes on n1 and n2 and sends them to n3 later, and meanwhile n3 may return old data.
The store designed here follows the Dynamo style: it is AP by default, with consistency tuned by settings.
Check yourself: 1. Why is partition tolerance not optional? 2. What does a CP store do during a partition? 3. Which choice suits a shopping-cart store, and which a bank balance?