CSE Roadmap

Core tier · about 5 hours · 8 parts

Part 2 of 8 · 2 min read

Scope and numbers

A queue that remembers.

Think of a post office with a long shelf for each street. Letters are added at the end of the shelf and never moved. Each household keeps a bookmark showing the last letter it read. Anyone can re-read from an earlier bookmark, and old letters are thrown away after two weeks.

That is the design here. A message queue lets one part of a system, the producer, hand messages to another part, the consumer, without waiting for it. The book builds one with extra features usually found in event streaming platforms such as Kafka.

Questions and the answers assumed here (from the book's sample talk).

  1. Message format and size? Text, a few KB.
  2. Can messages be read again? Yes, by different consumers.
  3. Are messages consumed in the order they were produced? Yes.
  4. Is data kept? Yes, for two weeks.
  5. Delivery guarantee? At-least-once at least, and ideally configurable.
  6. Throughput and delay? High throughput for log collection, and low delay for classic queue use.

Requirements.

  • R1. Producers send, consumers receive.
  • R2. Messages can be read many times or once.
  • R3. Order is kept within a stream of related messages.
  • R4. Delivery guarantee is configurable.
  • R5. Throughput and latency are tunable.
  • R6. Data is stored on disk and copied to several machines.
  • R7. The system scales out.

A traditional queue such as RabbitMQ keeps a message only until it is delivered, and does not promise order. Those differences make the traditional design much simpler. This page builds the harder version, and notes where it could be simpler.

Numbers (made up to show the idea). A topic receives 50,000 messages per second, each 1 KB. That is 50 MB per second. Per day: 50 × 86,400 = 4,320,000 MB, which is 4.32 TB. Kept for 14 days: 60.48 TB. With 3 copies of everything, 181.44 TB. This is why the storage layer matters.

Producer and consumer

A producer is a client that sends messages to the queue. A consumer is a client that reads them. The queue is the server between them.

Example: an order service produces "order placed" messages, and the billing service consumes them.

Check yourself: 1. What two features make this queue harder than a traditional one? 2. Why is storage the main cost? 3. What does the queue let producers avoid?