CSE Roadmap

Core tier · about 4 hours · 6 parts

Part 3 of 6 · 1 min read

Classes

A log, a consumer and a broker.

There are two ways to build this.

  • A queue per subscriber. When a message arrives, copy it into every subscriber's queue. Reading removes it. This is simple, but it copies each message many times, it cannot replay, and a new subscriber sees nothing old.
  • One log per topic, one offset per subscriber. The message is stored once. Each subscriber reads from its own position, and moving the position forward is how it "removes" a message. Replay is moving the position back. A new subscriber starts wherever the design says.

The second design is the one used by log-based brokers, and the one built here.

Class Job in one sentence
Message (record) Offset, topic, key and body, fixed once made
Topic An append-only list of messages. Offers append and readFrom(offset, max)
Subscriber (interface) A name and an onMessage method that may throw
Consumer Binds one subscriber to one topic, with a position, a failure count and a dead list
Broker Finds or creates topics, registers consumers, and publishes

The Topic is the only place that decides offsets, and its append is synchronized, so two publishers never get the same offset. Reading takes a copy of a slice, so a reader holds no lock while it runs the subscriber's code.

The subscriber is called by the consumer, and not the other way round. This is a pull design: the consumer asks the topic for messages when it is ready. A slow subscriber falls behind in its own position and does not slow anyone else. A push design would call each subscriber from the publisher's thread, and one slow subscriber would stall publishers.

Check yourself: 1. Why store each message once? 2. Which class assigns offsets? 3. Why does a slow subscriber not slow the others?