CSE Roadmap

Core tier · about 4 hours · 8 parts

Part 4 of 8 · 3 min read

Brokers, and how they differ from databases

A broker is a server that stores messages briefly and hands them out.

Think of a post office. Senders drop letters at the counter. Recipients collect them when they come. The post office holds the letters in between, and does not keep a copy after delivery.

A message broker (also called a message queue in DDIA) is a server that does this for programs. DDIA describes it as a kind of database that is optimised for handling message streams. Producers and consumers connect to it as clients. Because the data sits in the broker, the system can tolerate clients that come and go, or crash. The question of durability, meaning whether messages survive a crash, moves to the broker. Some brokers keep messages only in memory. Others write them to disk, and more safely, copy them to other machines. This costs speed. The book notes that if a system can afford to lose a message now and then, it can get higher throughput and lower latency from the same hardware.

Message broker

A message broker is a server that receives messages from producers, stores them, and delivers them to consumers.

Example: RabbitMQ, Amazon SQS and Google Cloud Pub/Sub are brokers.

The UDS book describes what a message holds. It has a header with metadata, such as a unique message ID, and a body with the content. A message is usually one of two kinds:

  • A command tells the receiver to do something. Example: "send this email".
  • An event tells the receiver that something happened. Example: "order 501 was paid".

DDIA lists differences between a broker and a database:

Question Typical database Typical message broker
How long is data kept? Until explicitly deleted Deleted once delivered to the consumers
How big is the working set? Can be very large Assumed small, so queues are short
How is data found? Indexes and queries Subscribe to a topic or pattern
What does the client learn? A snapshot at query time A notice when new messages arrive

The first row has a big consequence. A traditional broker is not a place to keep data for the long term. Pages 3 and 4 return to this when they look at logs, which keep messages after delivery.

Brokers can also be avoided. DDIA lists direct options: UDP multicast, brokerless libraries such as ZeroMQ, and webhooks, where one service calls a web address of another whenever an event happens. They work, but the application must handle lost messages itself, and they assume the consumer is online. If a consumer is offline, it misses the messages. If the producer crashes while it holds messages to retry, they are lost too. A broker removes these gaps by being the one durable place in the middle.

The UDS book also notes size limits. Some queues limit a message to 64 KB, and AWS Kinesis (a streaming service) to 1 MB. For a large item, such as a video, the usual pattern is to put the item in blob storage and send a message holding only its address. The book calls this queue plus blob.

Check yourself:

  • Name two differences between a message broker and a database.
  • What is the difference between a command and an event?