Part 6 of 7 · 3 min read
Choosing between a queue and a log
Use the one whose trade-offs match the work.
Choosing between a queue and a log
Think of two jobs. In one, a list of independent tasks has to be shared among workers as fast as possible. In the other, a ledger of events has to be read in order by several departments, and re-read when something goes wrong. The first is a tray. The second is a diary.
DDIA ends its comparison with a summary, which can be set out as a table:
| Point | Queue (AMQP/JMS style) | Log-based broker |
|---|---|---|
| Who gets a message? | One consumer per message | All messages of a partition go to one consumer of a group |
| Acknowledgment | Per message | One offset per partition |
| After delivery | Deleted | Retained until the retention limit |
| Order | Lost with redelivery | Kept inside a partition |
| Parallelism | Add consumers freely | At most the number of partitions |
| Slow message | Others carry on | Blocks the rest of its partition |
| Replay | Not possible | Possible |
| Best fit | Task queues where order does not matter | Streams of events, derived data, stream processing |
Both can feed the same system. A shop may use a queue for "send this email" tasks and a log for the stream of order events that billing, search and analytics all read.
The UDS book describes the log as one form of messaging channel. It notes that other channels, such as plain queues, do not guarantee order at all. It also notes that a log is state machine replication in disguise: if each consumer applies the same entries in the same order, they all reach the same state. For example, a relational database and a search index can both follow the same product log and stay in step.
Interview angle
Answer in your head first, then open it.
Why is Kafka fast?
Partitioned append-only logs on disk, sequential writes, light bookkeeping with one offset per partition, and batching.
How does Kafka keep order?
Only within a partition. Messages with the same key go to the same partition.
How many consumers can read a topic in parallel?
Up to the number of partitions per consumer group.
A consumer is slow. What do you do?
Check lag, add partitions and consumers, fix head-of-line blocking, make processing faster. Retention gives time to react.
Queue or log?
Queue for independent tasks, log for ordered event streams that several systems read or that need replay.
Check yourself:
- Name one task that fits a queue better than a log.
- What does a log offer that makes it suitable as the source of truth for derived data?
Words you will meet
| Word | Plain meaning |
|---|---|
| Log | An append-only sequence of records |
| Log-based broker | A message broker that keeps messages in logs and does not delete them on reading |
| Topic | A named group of partitions |
| Partition | One append-only log inside a topic |
| Offset | A message's position number in a partition |
| Key | A value (such as a user id) used to choose the partition |
| Consumer group | Consumers that share a topic, each partition read by one member |
| Consumer lag | How far a consumer is behind the newest message |
| Retention | How long messages are kept before deletion |
| Segment | A piece of a log that can be deleted or archived as a unit |
| Head-of-line blocking | One slow item holds back those behind it |
| Rebalance | Reassigning partitions when group members change |
| Replay | Reading old messages again by resetting the offset |
| Total order | Every message has a fixed place relative to every other |