Part 7 of 12 · 1 min read
How to build it so it holds up
- Ack after the work, with an idempotent handler keyed by job id.
- Set the lease above p99 run time and heartbeat to extend it for long runs.
- Retry transient errors with backoff and jitter; do not retry permanent ones (bad input, failed validation).
- Cap attempts and dead-letter, keeping payload, error and attempt history; alert on DLQ depth and age.
- Schedule per tenant with round-robin or weights, and per-tenant concurrency caps.
- Bound queue depth and apply backpressure; scale workers on queue age.
- Respect downstream rate limits with a shared limiter, not per-worker guesses.
- Separate queues by job class (fast chat versus slow research) so slow jobs do not block fast ones.
- Make jobs small and resumable (module 3) so a lease expiry loses little work.
- Expose queue metrics: depth, age of oldest, attempts, DLQ count, time in queue versus run time.