CSE Roadmap

Core tier · about 4 hours · 7 parts

Part 2 of 7 · 2 min read

From a queue to a log

A log is a file that only grows at the end, and readers keep their own place in it.

Think of a diary. You only write on the next free line, and you never tear out pages. Anyone can read it from any page, any number of times. Compare that with a pile of notes in a tray: each is thrown away once read.

DDIA describes the two mindsets. A traditional message broker is built around a transient idea: receiving a message deletes it. Databases and file systems take the opposite view: whatever is written is kept until somebody deletes it. The book points out the impact. With an acknowledging broker, a consumer cannot be run again to get the same result, because the messages are gone. A new consumer sees only messages sent after it registered. Batch jobs, by contrast, can be rerun any number of times, because their input is read-only.

The book asks why there is no hybrid of database-style storage and low-latency notification. The answer is the log-based message broker.

Log

A log is an append-only sequence of records on disk. New records are written at the end. Old records are never changed.

Example: the Unix tool tail -f watches a file that only grows, and prints new lines as they are added. A log-based broker works the same way.

A producer appends a message to the end of the log. A consumer reads the log in order. When it reaches the end, it waits for a notification that something new was appended. The UDS book shows the same picture: producers append at the end, and the consumers read at their own pace.

What does the consumer remember? Only its position, called the offset: the number of the next message to read. The consumer saves ("checkpoints") this number from time to time. If it crashes and restarts, it reads the saved offset and carries on from there. DDIA compares this to database replication: the offset is much like the log sequence number that lets a follower resume after a disconnect. The broker acts like the leader, and the consumer like a follower.

Offset

An offset is the position number of a message in a log. Offsets only increase.

Example: a consumer has handled messages 0 to 41 and saved offset 42. After a restart it asks for message 42 next.

Because reading does not delete anything, many consumers can read the same log at once, and each has its own offset. That gives fan-out (page 1) for free.

Question Acknowledging queue (page 1) Log-based broker
What does reading do? Removes the message Nothing. Only the offset moves
Can a new consumer see old messages? No Yes, from the start or any offset
Can a consumer reread yesterday's messages? No Yes, by resetting its offset
What does the broker track? The state of each message One offset per consumer

Check yourself:

  • What does a log consumer remember between restarts?
  • Why can a log serve several independent consumers without extra work?