CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 3 of 12 · 1 min read

The lease: how a dead worker's job comes back

A library due date for a message, so a crashed worker's job is not stuck forever.

Think of a library book with a due date. If it is not returned by then, the library assumes it is lost or forgotten and lets someone else borrow a copy, rather than waiting forever.

In plain words: when a worker takes a job, the queue hides it from everyone else only until a deadline (the lease); if the worker has not acked by then, the job becomes available again.

The key lines. This is the check that runs every time a worker asks for a new job:

# from code/mini-queue/jobqueue.py
if x.state == "leased" and x.lease_until <= now:      # lease expired: worker presumed dead
    x.state = "ready"

Read it as: before handing out jobs, first look for any "leased" job whose deadline has passed, and put it back in the pool. A worker that is killed mid-job never gets the chance to ack or nack. The lease notices on its behalf.

The lease length is a trade-off. Set it too short, and a worker still genuinely working on a slow job will have that job handed to someone else. That causes the same duplicate-effect problem as section 1. Set it too long, and a job stuck behind a truly dead worker waits that whole length before anyone else can try it. The fix for a job that is simply slow, not dead, is a heartbeat. The worker periodically extends its own lease while it is still alive.

Lease (visibility timeout)

How long a received job stays hidden from other workers before it becomes available again.

Example: a lease of 30 seconds on a job that normally takes 5: a worker killed at second 3 leaves the job invisible until second 30, then another worker can take it.