Part 3 of 7 · 3 min read
Topics and partitions
One log is limited to one machine, so a topic splits into many logs.
Think of a supermarket with one checkout. It handles only so many people per hour. Opening more checkouts handles more people, but now there is no single line, so the order in which people arrive at different checkouts is not defined.
A single log lives on one disk of one machine, so its speed is limited. DDIA explains how to go faster: split the log into partitions (the idea from Level 4, where data was split across machines). Different partitions can sit on different machines, and each is a separate log that can be read and written independently. A topic is a group of partitions that carry messages of the same type.
Topic
A topic is a named group of partitions that carry the same kind of message.
Example: a topic called orders holds all order events. It is stored as 6 partitions on 3 machines.
Partition
A partition is one append-only log that holds a share of a topic's messages. Each partition has its own offsets.
Example: partition 2 of orders holds messages with offsets 0, 1, 2 and so on, independent of the offsets in partition 3.
Within a partition, the broker gives every message a number that always increases: the offset. Since the partition is append-only, its messages are totally ordered. DDIA states the catch plainly: there is no ordering guarantee across different partitions.
The book names Apache Kafka, Amazon Kinesis Streams and Twitter's DistributedLog as log-based brokers of this kind. It notes that although they write every message to disk, they can reach millions of messages per second by spreading across machines, and they stay fault tolerant by replicating messages (replication as in Level 4).
Which partition gets a message? The producer usually decides. If a message has a key, such as a user id, the broker takes a hash of the key and picks the partition from it, as in hash partitioning in Level 4. All messages with the same key then go to the same partition and stay in order. Messages without a key are spread around evenly. The idea that a key controls the destination is general; DDIA also uses it in MapReduce, where the key acts like an address.
A worked example (made up). A topic clicks has 3 partitions. The rule is partition = user id mod 3. These messages arrive in this time order:
| Arrival | User | Partition (id mod 3) | Offset in that partition |
|---|---|---|---|
| 1 | 7 | 1 | 0 |
| 2 | 9 | 0 | 0 |
| 3 | 7 | 1 | 1 |
| 4 | 8 | 2 | 0 |
| 5 | 10 | 1 | 2 |
| 6 | 9 | 0 | 1 |
All of user 7's messages are in partition 1, at offsets 0 and 1, in the order they were sent. Between users, no order is promised: user 9's second message (arrival 6) may be read before or after user 10's (arrival 5), since they live in different partitions. For most uses, only the order of one user's events matters, so this is enough.
Read it as
Order is a promise per partition, not per topic. If two messages must be in order, they must share a key.
Hot partitions. The UDS book warns that a partition can become much hotter than the others. If one consumer cannot keep up with it, the topic must be repartitioned, which can temporarily slow the broker, since messages get reshuffled across all partitions. A key such as a country with huge traffic can cause this. Picking a key with many distinct values and even traffic helps.
Check yourself:
- In the table above, can the broker promise that user 8's message is read before user 10's? Why not?
- What decides which partition a keyed message goes to?