Part 5 of 8 · 2 min read
Flows
Writing and reading, with groups.
Producer. A producer must know which broker holds the right partition. One way is a routing layer between producer and broker, but it adds a network hop and blocks batching. The better way, from the book, is to build the routing and a buffer into the producer's client library. The producer finds the leader replica of the partition from metadata, buffers messages, and sends a batch directly. Followers pull data from the leader, and once enough of them have it, the leader confirms the write.
Consumer: pull, not push. A broker could push messages to consumers, or consumers could pull. Push gives low delay, but it can overwhelm a slow consumer, and the broker cannot know each consumer's speed. In pull, the consumer asks for messages from its offset and controls the pace. It suits batching and lets slow consumers catch up later. Its drawback is wasted requests when there is nothing new, which long polling fixes: a request waits for a set time for new data. Most queues choose pull.
Joining a group. The consumer finds its coordinator, a broker found by hashing the group's name. The coordinator tracks which consumers are alive through heartbeats (Level 8) and gives out partitions. The consumer fetches messages from its last committed offset, processes them, and commits the new offset.
Rebalancing reassigns partitions when a consumer joins, leaves or crashes. The coordinator notices a change, asks all members to rejoin, picks one as the group leader, and the leader creates a new plan that the coordinator sends to everyone. A crash is noticed when heartbeats stop.
A worked example (made up). A group has consumers A and B on 4 partitions: A has 1 and 3, B has 2 and 4. Consumer A stops sending heartbeats. After the timeout, the coordinator starts a rebalance, B rejoins, and B receives all four partitions. A new consumer C later joins and the partitions are shared again.
State. The queue stores each group's last committed offset per partition, so a replacement consumer can resume. Reads and writes of this state are frequent but small. A key-value store with strong consistency, such as ZooKeeper or etcd (Level 5), fits. The same service handles broker leader election and topic settings.
Common mistake
Committing the offset before the work is done. If the consumer crashes after the commit and before processing, the message is skipped for ever.
Check yourself: 1. Why build routing into the producer library? 2. What does the coordinator do? 3. What happens to partitions when a consumer stops sending heartbeats?