CSE Roadmap

Core tier · about 5 hours · 8 parts

Part 7 of 8 · 3 min read

Scaling and trade-offs

Brokers, partitions and follow-ups.

Scaling, failures and what to say

Producers and consumers. Producers are simple to scale, as they need no coordination. Consumers scale by adding members to a group, and rebalancing handles it.

Broker failure. Suppose broker 3 crashes. The controller sees it, and for each partition that lived there, picks a new leader from the in-sync followers and plans a new replica on another broker. The new replica copies from the leader until it catches up. Replicas of one partition must sit on different brokers, or one failure removes them all. Putting replicas in different data centres is safer, but costs delay.

Adding a broker. The controller temporarily allows more replicas than normal. The new broker copies data, and once it has caught up, the extra old replica is removed. No data is lost at any moment.

Adding partitions. Producers learn about it from any broker, and consumers rebalance. Old data stays in the old partitions, with no migration. New messages are spread across all. Allocate enough partitions at the start, so this is rare. Note that adding partitions changes hash(key) % n, so a key may move to a new partition.

Choice Gain Cost
Append-only log Fast, simple No update in place
Pull model Consumer sets the pace Wasted polls, handled by long polling
ACK = all No loss on leader failure Higher latency
More partitions More parallel consumers More files, a longer rebalance
Exactly-once Correct for money Cost and complexity
Many replicas Durability Storage and copy traffic

Extra features the book mentions are message filtering by tag, delayed messages that become visible later, and scheduled messages. A delayed message can be held in a separate structure and moved to the topic when due.

Interview angle

Answer in your head first, then open it.

Why partitions?

Capacity and parallel consumers, with order inside each.

Why a log on disk?

Sequential access is fast, and the operating system caches it.

Pull or push?

Pull, so consumers set their pace and batch.

How do you avoid losing messages?

Replicas, ACK = all, minimum ISR, commit offsets after processing.

Can you get exactly-once?

Practically, at-least-once with idempotent consumers.

One follow-up to try. "One consumer is slow and its partition falls far behind." A good reply: lag is the gap between the newest offset and the group's committed offset, and it should be watched. Add partitions and consumers if the work can be split by key. Otherwise speed up the consumer, or move heavy work to a second queue. The log keeps the data for two weeks, so the slow consumer loses nothing as long as it catches up inside that time.

Check yourself: 1. What happens to data when a partition is added? 2. Why must replicas of one partition be on different brokers? 3. What is consumer lag?

Words you will meet

Word Meaning
Topic A named category of messages
Partition One ordered slice of a topic
Broker A server that stores partitions
Offset A message's position in its partition
Consumer group Consumers that share a topic's work
WAL A file that is only appended to
ISR Replicas that are in sync with the leader
Rebalancing Reassigning partitions among consumers