CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that lose jobs or starve good ones

Common mistakes, and what the example does not do.

Mistakes that lose jobs or starve good ones

  • Ack before work, losing jobs on crashes.
  • Non-idempotent handlers behind an at-least-once queue.
  • Lease shorter than the run time with no heartbeat.
  • Retrying permanent errors and rate-limit responses immediately.
  • No jitter, so every retry lands together.
  • No DLQ, or a DLQ nobody watches.
  • One shared FIFO for all tenants and job types.
  • Scaling workers on queue depth only, ignoring age.
  • Unbounded queues, turning an outage into a cost event.

What the example does not do

About this example What the example does not do
  • One process, tick-based, no real broker, network, persistence or clock drift.
  • Assumed parameters: crash rate and timing, capacity, job durations, tenant sizes.
  • The retry-storm model is simplified: all clients start failed together and clients that are rejected retry once more per attempt.
  • Poison results depend on FIFO order and retry delay; a longer delay or per-attempt demotion would let good jobs through earlier.
  • No exactly-once claims, no partitioned ordering, no transactional outbox.
  • Optional real experiment: run a real queue (SQS, Redis or Postgres SKIP LOCKED), kill workers at random while jobs run, and count lost jobs and duplicate effects for ack-first and ack-last.

Check yourself

A worker crashes after sending an email but before acknowledging the job. What happens next, and what stops a second email?

Show answer

The lease expires and the job is delivered again. An idempotent effect, keyed by job and step, stops the duplicate; in the simulation at-least-once delivery with an idempotent effect gave 0 lost and 0 duplicates.

What you will use daily at work

Practice What it looks like
Ack after the work, make the effect idempotent At most once loses jobs; at least once duplicates them.
Add jitter to retry backoff Full jitter cut the retry clearing time from 962 s to 69 s.
Dead-letter after a few attempts And alert on the dead-letter queue.
Schedule fairly per tenant Cap concurrency per tenant and per downstream.