CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Ack after the work, with an idempotent handler keyed by job id.
  • Set the lease above p99 run time and heartbeat to extend it for long runs.
  • Retry transient errors with backoff and jitter; do not retry permanent ones (bad input, failed validation).
  • Cap attempts and dead-letter, keeping payload, error and attempt history; alert on DLQ depth and age.
  • Schedule per tenant with round-robin or weights, and per-tenant concurrency caps.
  • Bound queue depth and apply backpressure; scale workers on queue age.
  • Respect downstream rate limits with a shared limiter, not per-worker guesses.
  • Separate queues by job class (fast chat versus slow research) so slow jobs do not block fast ones.
  • Make jobs small and resumable (module 3) so a lease expiry loses little work.
  • Expose queue metrics: depth, age of oldest, attempts, DLQ count, time in queue versus run time.