Part 2 of 8 · 2 min read
Splitting the data
The idea of a partition, and how it differs from a copy.
Think of a printed encyclopedia in 12 volumes. No single volume holds everything. A reader who wants the entry for "Tiger" goes straight to the volume for T. Twelve readers can use twelve volumes at the same time. A set split like this is easy to carry, and easy to share.
The book "Designing Data-Intensive Applications" gives the main reason for partitioning as scalability. Different partitions can be put on different nodes in a shared-nothing cluster [a cluster in which each node has its own CPU, memory and disk]. A large data set then spreads over many disks, and the query load spreads over many processors. For queries that touch one partition, each node can answer on its own, so throughput grows as nodes are added.
Partition (shard)
A partition (also called a shard) is one piece of a data set. Each record belongs to exactly one partition. Partitioning (or sharding) is the act of splitting the data into such pieces and placing them on different nodes.
Example: a table of 100 million users is split into 10 partitions of about 10 million users each. Each lives on its own machine.
The word is easy to confuse. The book notes that partitioning here is nothing to do with a network partition [a fault that cuts the network between nodes].
| Replication | Partitioning | |
|---|---|---|
| What each node holds | A copy of everything | A different share |
| Helps with | Availability, read capacity | Data size, write capacity |
| Record lives on | Several nodes | One partition (which itself may be replicated) |
The two are used together. The book describes it: each record belongs to exactly one partition but is stored on several nodes for fault tolerance. A node may store more than one partition. With leaders and followers, each partition has its leader on one node and followers on other nodes. A node can be a leader for some partitions and a follower for others.
A worked example. A cluster has 6 nodes and 12 partitions. Each partition has 3 copies (1 leader and 2 followers). That makes 36 partition copies, 6 per node. Node 1 might lead partitions 1 and 7 and follow partitions 2, 3, 5 and 12. If node 1 dies, the 2 partitions it led get new leaders on other nodes, and the other 4 lose only a follower.
Everything said about replication in modules 1 to 3 applies to each partition. The choice of how to partition is mostly independent of the replication scheme, so the rest of this module ignores replication.
The aim: spread the data and the load evenly. If each of 10 nodes takes a fair share, then 10 nodes should handle 10 times the data and 10 times the throughput of one. The challenge is the word "fair".
Check yourself:
- A record is in how many partitions? On how many nodes?
- A cluster has 8 nodes and 16 partitions, each with 3 copies. How many partition copies does each node hold?