CSE Roadmap

Core tier · about 4 hours · 7 parts

Part 6 of 7 · 3 min read

Choosing between a queue and a log

Use the one whose trade-offs match the work.

Choosing between a queue and a log

Think of two jobs. In one, a list of independent tasks has to be shared among workers as fast as possible. In the other, a ledger of events has to be read in order by several departments, and re-read when something goes wrong. The first is a tray. The second is a diary.

DDIA ends its comparison with a summary, which can be set out as a table:

Point Queue (AMQP/JMS style) Log-based broker
Who gets a message? One consumer per message All messages of a partition go to one consumer of a group
Acknowledgment Per message One offset per partition
After delivery Deleted Retained until the retention limit
Order Lost with redelivery Kept inside a partition
Parallelism Add consumers freely At most the number of partitions
Slow message Others carry on Blocks the rest of its partition
Replay Not possible Possible
Best fit Task queues where order does not matter Streams of events, derived data, stream processing

Both can feed the same system. A shop may use a queue for "send this email" tasks and a log for the stream of order events that billing, search and analytics all read.

The UDS book describes the log as one form of messaging channel. It notes that other channels, such as plain queues, do not guarantee order at all. It also notes that a log is state machine replication in disguise: if each consumer applies the same entries in the same order, they all reach the same state. For example, a relational database and a search index can both follow the same product log and stay in step.

Interview angle

Answer in your head first, then open it.

Why is Kafka fast?

Partitioned append-only logs on disk, sequential writes, light bookkeeping with one offset per partition, and batching.

How does Kafka keep order?

Only within a partition. Messages with the same key go to the same partition.

How many consumers can read a topic in parallel?

Up to the number of partitions per consumer group.

A consumer is slow. What do you do?

Check lag, add partitions and consumers, fix head-of-line blocking, make processing faster. Retention gives time to react.

Queue or log?

Queue for independent tasks, log for ordered event streams that several systems read or that need replay.

Check yourself:

  • Name one task that fits a queue better than a log.
  • What does a log offer that makes it suitable as the source of truth for derived data?

Words you will meet

Word Plain meaning
Log An append-only sequence of records
Log-based broker A message broker that keeps messages in logs and does not delete them on reading
Topic A named group of partitions
Partition One append-only log inside a topic
Offset A message's position number in a partition
Key A value (such as a user id) used to choose the partition
Consumer group Consumers that share a topic, each partition read by one member
Consumer lag How far a consumer is behind the newest message
Retention How long messages are kept before deletion
Segment A piece of a log that can be deleted or archived as a unit
Head-of-line blocking One slow item holds back those behind it
Rebalance Reassigning partitions when group members change
Replay Reading old messages again by resetting the offset
Total order Every message has a fixed place relative to every other