Part 4 of 6 · 2 min read
Log-based brokers
Think of a log-based broker as a long paper ledger that is only ever written at the bottom. Readers can walk back to any earlier line and read forward from there, and the line number itself tells them exactly how far they have gotten.
In plain words: A traditional message queue typically deletes a message once it has been delivered, so a new consumer only sees messages sent after it joined. A log-based broker instead appends every message to a durable, append-only log, and a consumer reads that log sequentially from wherever it left off, which also lets a new consumer start from the very beginning if needed. To scale past what one machine's disk can handle, the log is split into partitions, each hosted on its own machine, grouped together under one topic. Every message within a partition gets a monotonically increasing offset, so a consumer only needs to remember one number, its current offset, to know exactly which messages it has already processed.
A worked example. The bookstore's "orders" topic has four partitions spread across four machines. A new analytics consumer that starts up today can still read every order ever placed, starting from offset 0 in each partition, because the log was never deleted.
The key lines.
# Each partition is read sequentially from its last known offset
offset = consumer.get_offset(partition=2)
for message in broker.read(topic="orders", partition=2, from_offset=offset):
process(message)
consumer.commit_offset(partition=2, offset=message.offset)
Partition (streaming), offset
A partition is one append-only shard of a topic's log, hosted on one machine, that can be read independently of the topic's other partitions. An offset is the monotonically increasing position of a message within its partition, which a consumer records to track exactly how far it has read.
Example: offset for B.0 = 4 means a consumer reading partition 0 of topic B has processed messages up through position 4 and will resume at position 5.