Part 7 of 8 · 3 min read
Scaling and trade-offs
Brokers, partitions and follow-ups.
Scaling, failures and what to say
Producers and consumers. Producers are simple to scale, as they need no coordination. Consumers scale by adding members to a group, and rebalancing handles it.
Broker failure. Suppose broker 3 crashes. The controller sees it, and for each partition that lived there, picks a new leader from the in-sync followers and plans a new replica on another broker. The new replica copies from the leader until it catches up. Replicas of one partition must sit on different brokers, or one failure removes them all. Putting replicas in different data centres is safer, but costs delay.
Adding a broker. The controller temporarily allows more replicas than normal. The new broker copies data, and once it has caught up, the extra old replica is removed. No data is lost at any moment.
Adding partitions. Producers learn about it from any broker, and consumers rebalance. Old data stays in the old partitions, with no migration. New messages are spread across all. Allocate enough partitions at the start, so this is rare. Note that adding partitions changes hash(key) % n, so a key may move to a new partition.
| Choice | Gain | Cost |
|---|---|---|
| Append-only log | Fast, simple | No update in place |
| Pull model | Consumer sets the pace | Wasted polls, handled by long polling |
| ACK = all | No loss on leader failure | Higher latency |
| More partitions | More parallel consumers | More files, a longer rebalance |
| Exactly-once | Correct for money | Cost and complexity |
| Many replicas | Durability | Storage and copy traffic |
Extra features the book mentions are message filtering by tag, delayed messages that become visible later, and scheduled messages. A delayed message can be held in a separate structure and moved to the topic when due.
Interview angle
Answer in your head first, then open it.
Why partitions?
Capacity and parallel consumers, with order inside each.
Why a log on disk?
Sequential access is fast, and the operating system caches it.
Pull or push?
Pull, so consumers set their pace and batch.
How do you avoid losing messages?
Replicas, ACK = all, minimum ISR, commit offsets after processing.
Can you get exactly-once?
Practically, at-least-once with idempotent consumers.
One follow-up to try. "One consumer is slow and its partition falls far behind." A good reply: lag is the gap between the newest offset and the group's committed offset, and it should be watched. Add partitions and consumers if the work can be split by key. Otherwise speed up the consumer, or move heavy work to a second queue. The log keeps the data for two weeks, so the slow consumer loses nothing as long as it catches up inside that time.
Check yourself: 1. What happens to data when a partition is added? 2. Why must replicas of one partition be on different brokers? 3. What is consumer lag?
Words you will meet
| Word | Meaning |
|---|---|
| Topic | A named category of messages |
| Partition | One ordered slice of a topic |
| Broker | A server that stores partitions |
| Offset | A message's position in its partition |
| Consumer group | Consumers that share a topic's work |
| WAL | A file that is only appended to |
| ISR | Replicas that are in sync with the leader |
| Rebalancing | Reassigning partitions among consumers |