Part 3 of 12 · 1 min read
The lease: how a dead worker's job comes back
A library due date for a message, so a crashed worker's job is not stuck forever.
Think of a library book with a due date. If it is not returned by then, the library assumes it is lost or forgotten and lets someone else borrow a copy, rather than waiting forever.
In plain words: when a worker takes a job, the queue hides it from everyone else only until a deadline (the lease); if the worker has not acked by then, the job becomes available again.
The key lines. This is the check that runs every time a worker asks for a new job:
# from code/mini-queue/jobqueue.py
if x.state == "leased" and x.lease_until <= now: # lease expired: worker presumed dead
x.state = "ready"
Read it as: before handing out jobs, first look for any "leased" job whose deadline has passed, and put it back in the pool. A worker that is killed mid-job never gets the chance to ack or nack. The lease notices on its behalf.
The lease length is a trade-off. Set it too short, and a worker still genuinely working on a slow job will have that job handed to someone else. That causes the same duplicate-effect problem as section 1. Set it too long, and a job stuck behind a truly dead worker waits that whole length before anyone else can try it. The fix for a job that is simply slow, not dead, is a heartbeat. The worker periodically extends its own lease while it is still alive.
Lease (visibility timeout)
How long a received job stays hidden from other workers before it becomes available again.
Example: a lease of 30 seconds on a job that normally takes 5: a worker killed at second 3 leaves the job invisible until second 30, then another worker can take it.