Part 5 of 12 · 1 min read
Poison messages: the job that always fails
Why retrying forever can starve every healthy job behind it, and the dead-letter fix.
Think of one jammed photocopier in an office with only two machines. If everyone keeps feeding it the same stuck page, it holds up both machines, and nobody's good document gets copied either.
In plain words: a job that fails every single time will be retried forever unless something takes it out of rotation. While it is retried, it can occupy a worker that a healthy job needed.
A worked example. 4 poison jobs and 40 good jobs, 2 workers. If poison jobs are retried forever, they sit at the front of the queue and keep being handed back to workers: 0 of the 40 good jobs finish after 2,000 seconds. If a job is instead parked after 3 failed attempts, the 2 workers stop wasting time on it. All 40 good jobs finish by 26 seconds, and the 4 bad ones sit aside for a human to look at.
# from code/mini-queue/jobqueue.py
if x.attempts >= self.max_attempts: # too many deliveries: park it
x.state = "dead"; continue
Read it as: before offering a job to a worker, check whether it has already failed too many times. If so, set it aside instead of trying again. The parked job keeps its payload and error history, so someone can look at why it failed.
Dead-letter queue (DLQ)
Where jobs go once they have failed more times than the limit allows, instead of being retried forever.
Example: the 4 poison jobs above, marked "dead" after 3 attempts each, with their last error kept for a person to read.