CSE Roadmap

Core tier · about 4 hours · 6 parts

Part 2 of 6 · 2 min read

Requirements

Publish once, deliver to many.

Think of a notice board in a school corridor. A teacher pins a notice under the heading "Sports". Students who care about sports read that heading when they pass. The teacher does not know who reads it. Each student remembers which notices they have already seen. A student who was away for a week can read the whole week's notices on return.

The board is the broker. The heading is the topic. The teacher is the publisher. The students are subscribers. Each student's memory of what they have seen is the offset.

Questions and the answers assumed here:

  1. What is published? A message with a key and a body, under a topic name.
  2. Who receives it? Every subscriber of that topic gets every message.
  3. In what order? Messages in one topic keep their order for each subscriber.
  4. Do subscribers share progress? No. Each has its own position.
  5. What if a subscriber fails on a message? Retry a few times, then set the message aside.
  6. Can a subscriber replay old messages? Yes, by moving its position back.
  7. Out of scope: disk storage, several brokers, and splitting a topic into partitions.

The numbered requirements:

  • R1. A publisher appends a message to a topic. The topic is created if it does not exist.
  • R2. Each message in a topic gets a unique, increasing offset.
  • R3. Each subscriber has its own next offset in each topic it follows.
  • R4. A subscriber receives messages in offset order.
  • R5. A failing message is retried up to a set number of times, then put in a dead list, and the subscriber moves on.
  • R6. A subscriber can seek to an earlier offset and read again.
  • R7. Publishers and subscribers may run on many threads.

Offset

An offset is the position of a message in a topic's log, counted from zero in the order the messages arrived.

Example: the third message published to a topic has offset 2.

Check yourself: 1. What does each subscriber remember? 2. What happens to a message that fails every retry? 3. What is the offset of the first message?