Part 9 of 12 · 2 min read
Mistakes that lose jobs or starve good ones
Common mistakes, and what the example does not do.
Mistakes that lose jobs or starve good ones
- Ack before work, losing jobs on crashes.
- Non-idempotent handlers behind an at-least-once queue.
- Lease shorter than the run time with no heartbeat.
- Retrying permanent errors and rate-limit responses immediately.
- No jitter, so every retry lands together.
- No DLQ, or a DLQ nobody watches.
- One shared FIFO for all tenants and job types.
- Scaling workers on queue depth only, ignoring age.
- Unbounded queues, turning an outage into a cost event.
What the example does not do
About this example What the example does not do
- One process, tick-based, no real broker, network, persistence or clock drift.
- Assumed parameters: crash rate and timing, capacity, job durations, tenant sizes.
- The retry-storm model is simplified: all clients start failed together and clients that are rejected retry once more per attempt.
- Poison results depend on FIFO order and retry delay; a longer delay or per-attempt demotion would let good jobs through earlier.
- No exactly-once claims, no partitioned ordering, no transactional outbox.
- Optional real experiment: run a real queue (SQS, Redis or Postgres
SKIP LOCKED), kill workers at random while jobs run, and count lost jobs and duplicate effects for ack-first and ack-last.
Check yourself
A worker crashes after sending an email but before acknowledging the job. What happens next, and what stops a second email?
Show answer
The lease expires and the job is delivered again. An idempotent effect, keyed by job and step, stops the duplicate; in the simulation at-least-once delivery with an idempotent effect gave 0 lost and 0 duplicates.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Ack after the work, make the effect idempotent | At most once loses jobs; at least once duplicates them. |
| Add jitter to retry backoff | Full jitter cut the retry clearing time from 962 s to 69 s. |
| Dead-letter after a few attempts | And alert on the dead-letter queue. |
| Schedule fairly per tenant | Cap concurrency per tenant and per downstream. |