Part 3 of 6 · 1 min read
Classes
A log, a consumer and a broker.
There are two ways to build this.
- A queue per subscriber. When a message arrives, copy it into every subscriber's queue. Reading removes it. This is simple, but it copies each message many times, it cannot replay, and a new subscriber sees nothing old.
- One log per topic, one offset per subscriber. The message is stored once. Each subscriber reads from its own position, and moving the position forward is how it "removes" a message. Replay is moving the position back. A new subscriber starts wherever the design says.
The second design is the one used by log-based brokers, and the one built here.
| Class | Job in one sentence |
|---|---|
Message (record) |
Offset, topic, key and body, fixed once made |
Topic |
An append-only list of messages. Offers append and readFrom(offset, max) |
Subscriber (interface) |
A name and an onMessage method that may throw |
Consumer |
Binds one subscriber to one topic, with a position, a failure count and a dead list |
Broker |
Finds or creates topics, registers consumers, and publishes |
The Topic is the only place that decides offsets, and its append is synchronized, so two publishers never get the same offset. Reading takes a copy of a slice, so a reader holds no lock while it runs the subscriber's code.
The subscriber is called by the consumer, and not the other way round. This is a pull design: the consumer asks the topic for messages when it is ready. A slow subscriber falls behind in its own position and does not slow anyone else. A push design would call each subscriber from the publisher's thread, and one slow subscriber would stall publishers.
Check yourself: 1. Why store each message once? 2. Which class assigns offsets? 3. Why does a slow subscriber not slow the others?