Part 8 of 12 · 1 min read
Answering queue questions in an interview
At-most-once versus at-least-once: which do you choose?
"At-least-once with idempotent handlers. In the example's simulation acking first lost 99 of 2000 jobs. Acking last lost none but ran 65 side effects twice; an idempotency check removed the duplicates. Exactly-once is built from at-least-once plus dedupe." Follow-up: "How do you make an email send idempotent?" Record the job id before sending, or pass an idempotency key to the provider, and skip if present. A crash between send and record can still duplicate, so prefer provider-side keys.
What is a dead-letter queue for?
"To stop poison messages from consuming workers forever. After a maximum number of attempts the job is parked with its error for investigation. In the example's test, retrying forever left 40 good jobs undone. Dead-lettering finished them by 26 seconds instead." Follow-up: "What do you do with DLQ items?" Alert, inspect, fix the cause or data, then redrive. Add the case to the regression suite.
Why add jitter to retries?
"Without it clients that failed together retry together. With 1000 clients and 50 requests per second capacity, no jitter rejected 9,500 requests and took 962 seconds to clear; full jitter rejected 2,283 and cleared in 69 seconds. It does not fix the first spike, so I also use a wider initial window or admission control." Follow-up: "What else helps?" Circuit breakers and shared rate limiters (module 6).
One customer floods the queue. What happens to the others?
"With FIFO they wait behind the backlog: in the example's simulation the small tenant's p95 was 101 seconds. Round-robin per tenant brought it to 4 seconds without hurting the big tenant. I add per-tenant concurrency limits and weights." Follow-up: "How do you stop one tenant using all workers?" Concurrency caps per tenant and per job class.