CSE Roadmap

Core tier · about 4 hours · 8 parts

Part 6 of 8 · 3 min read

Failure: poison messages and fault isolation

A bad message can loop forever, so queues need retry limits and a place to park failures.

Think of a parcel that no courier can deliver because the address is wrong. If each courier brings it back and puts it on the pile, the pile never shrinks. Somebody has to set it aside and look at it.

The Building Microservices book tells a true story of this. A system put pricing requests on a queue, and workers took them. A bug appeared, so a certain type of request made a worker crash. The queue was transacted: when the worker died, its lock on the request timed out and the request went back to the queue. Another worker took it and died too. The workers kept dying. The author calls this a catastrophic failover.

The team made two fixes. They fixed the bug, and they set a maximum retry limit. They also built a way to look at, and replay, the bad messages. They called it a message hospital, now more commonly a dead letter queue.

Poison message

A poison message is a message that makes its consumer fail every time it is processed, so it keeps being redelivered.

Example: a request with a missing field causes an error in the worker. After each crash, the broker puts the request back.

Dead letter queue

A dead letter queue is a separate queue where messages are moved after they have failed more than a set number of times. People or tools can inspect them later and replay them.

Example: after 5 failed tries, request 9917 moves to the dead letter queue. An engineer fixes the bug, then replays the message.

A worked example. A queue has a retry limit of 3. Message M is poison.

Attempt What happens
1 Worker takes M, crashes, the timeout returns M to the queue
2 Another worker takes M, crashes
3 A third worker takes M, crashes
4 The retry count is above 3, so the broker moves M to the dead letter queue

Three workers crashed, not an endless number. The other messages carry on. An alert on "dead letter queue not empty" tells the team to look.

Fault isolation. The System Design Interview book shows queues also containing failures. In its notification system, each third-party service (for example SMS, email or push) has its own message queue. An outage at one provider then does not affect the others. Workers pull events from their queue and send them. If a send fails, the worker puts the event back for a retry, and if the problem persists, an alert goes to developers.

The same idea applies to your own services. If the order service sends to the email service directly, a slow email service slows orders. If the order service only writes to a queue, a slow email service only grows its own backlog.

The Building Microservices book adds a caution: asynchronous designs are harder to follow. It advises good monitoring and correlation IDs, which are identifiers attached to a request and copied into every message that results from it, so that its journey across processes can be traced.

Correlation ID

A correlation ID is a unique tag created at the start of a request and passed along with every call and message it causes.

Example: the web server tags a checkout with id "c-7712". The log lines of the order, payment and email services all carry "c-7712", so one search finds the full story.

Check yourself:

  • Why does a queue with no retry limit make a poison message dangerous?
  • How does one queue per third-party service isolate failures?