CSE Roadmap

Core tier · about 4 hours · 7 parts

Part 4 of 7 · 3 min read

Consumer groups

A group of consumers shares a topic by owning whole partitions.

Think of a team sorting several mailbags. Each person takes whole bags, not single letters. Within a bag the letters are handled in order. If there are more people than bags, the extra people have nothing to do.

DDIA explains how log-based brokers share work among consumers. Rather than giving single messages to different consumers (as a queue does), the broker assigns whole partitions to the members of a consumer group. Each consumer then reads all messages in its partitions, in order, one at a time.

Consumer group

A consumer group is a set of consumers that together read a topic, where each partition is read by exactly one member of the group at a time.

Example: a group of 3 consumers reads a topic of 6 partitions. Each consumer owns 2 partitions.

Different groups are independent. A billing group and an analytics group can read the same topic, and each sees all messages with its own offsets.

This coarse split has two costs, which DDIA spells out:

  • The number of consumers in a group that do useful work is at most the number of partitions. A topic with 6 partitions keeps at most 6 group members busy.
  • If one message is slow to process, it holds up all later messages in the same partition. This is called head-of-line blocking.

So the book gives a rule of choice. When messages are expensive to process, you want to parallelise per message, and order does not matter, the queue style of page 1 is better. When throughput is high, each message is fast to process and order matters, the log style works very well. If more parallelism is needed, use more partitions, rather than several threads in one consumer, because threads complicate the offset bookkeeping.

A worked example. A topic has 4 partitions, and each consumer can process 500 messages per second.

Group size Partitions per consumer Capacity of the group
1 4 500 per second
2 2 each 1,000 per second
4 1 each 2,000 per second
6 1 each for 4 members, 2 members idle 2,000 per second

The group of 6 gains nothing over the group of 4. To go beyond 2,000 per second, the topic needs more partitions.

When a consumer dies. If a consumer fails, another member of the group is given its partitions. It starts at the last recorded offset. Any messages that the dead consumer had processed but not yet checkpointed are handled again, which is the duplicate problem of page 2. The books also warn that the group has to rebalance (reassign partitions) when members join or leave, and that work pauses briefly while this happens. The brief pause during rebalancing is general knowledge about how such systems behave, and is not a figure from the books used here.

Bookkeeping is light. Because messages are read in order, a consumer's progress is one number per partition. The broker does not track an acknowledgment for every message. DDIA says this smaller bookkeeping, and the room it leaves for batching and pipelining, is part of why log systems reach high throughput.

Check yourself:

  • A topic has 8 partitions. How many consumers of one group can be busy? What happens to a ninth?
  • Why does one slow message delay everything after it in its partition?