Part 3 of 8 · 2 min read
Main concepts
How messages are organised.
Topic. A named category of messages. Producers write to a topic and consumers read from it. In the publish and subscribe model, every subscriber to a topic gets every message. In the point-to-point model, each message goes to exactly one consumer.
Partition. One topic may hold more data than one server can store or serve. So the topic is split into partitions, each a small part of the topic. Each partition is a first-in, first-out line, so order holds inside a partition. Partitions are spread across the servers of the cluster, called brokers. More partitions mean more capacity.
Offset. The position of a message inside its partition, counting from 0.
Message key. An optional field, such as a user id. All messages with the same key go to the same partition, so they stay in order. If there is no key, a partition is chosen at random. A typical rule is hash(key) % number_of_partitions. A worked example (made up): with 4 partitions, a key whose hash is 22 goes to partition 22 % 4 = 2, every time.
Consumer group. A set of consumers that share the work of a topic. Each group keeps its own position, so two groups can read the same topic independently. Inside a group, each partition is read by only one consumer, which keeps order. If a group has more consumers than partitions, some consumers sit idle. A worked example (made up): a topic with 12 partitions and a group of 5 consumers gives two consumers 3 partitions each and three consumers 2 each, 3 + 3 + 2 + 2 + 2 = 12.
Both models are covered. Publish and subscribe is several groups reading one topic. Point-to-point is one group, where each message reaches one consumer of the group.
Partition
A partition is one ordered, append-only slice of a topic, stored on one broker (with copies on others).
Example: the topic "orders" has 4 partitions. Messages for user 17 always land in partition 1, in the order sent.
Check yourself: 1. Why does the key keep related messages in order? 2. How many consumers in a group can read one partition? 3. What happens if a group has 15 consumers and a topic has 12 partitions?