CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that bill twice or leak ids

Common mistakes, and what the example does not do.

Mistakes that bill twice or leak ids

  • Holding the HTTP connection for the whole run.
  • No idempotency on submit, so retries bill twice.
  • Guessable or sequential job ids and no owner check.
  • 403 on other owners' ids, which confirms existence.
  • Webhooks without signatures or without a replay window.
  • Processing events twice because the receiver does not dedupe.
  • Cancel that only flips a flag while the worker keeps spending.
  • No maximum run time or budget per job.

What the example does not do

About this example What the example does not do
  • No real network, framework, storage or queue; the clock is a parameter.
  • Exponential run time and fixed loss rates are assumptions.
  • Retry timing is not simulated (no sleeping), only attempt counts.
  • Cancel is a state change; it does not stop a real worker (module 3 covers that).
  • No authentication layer; the owner is a string.
  • Optional real experiment: put a 2-minute proxy in front of a real 5-minute agent endpoint, call it with and without an idempotency key through a flaky network, and count duplicate runs and bills.

Check yourself

A client calls an agent endpoint and the proxy times out after 2 minutes, but the run takes 3. The client retries. What do you pay for?

Show answer

Both runs. In the simulation, retrying on timeout answered 715 of 1000 requests but ran 1541 jobs, 541 of them duplicates. A job API returns 202 with a job id at once, and an idempotency key makes a retried submit return the same job.

For the curious A webhook can apply twice even when you acknowledge nearly all of them

With 30 percent lost acknowledgements and up to 6 attempts, 999 of 1000 events were acknowledged, but the receiver applied all 1000, and a lost ack means the sender tries again. Sign each event, check the timestamp, and keep a record of event ids so a second delivery is acknowledged without being applied again. Durations and loss rates here are assumed; there is no real HTTP server.

What you will use daily at work

Practice What it looks like
Return a job id, never hold the connection POST returns 202; the client polls or gets a webhook.
Send an idempotency key with every submit The same key and body gives the same job; a different body gives 409.
Return 404 for another owner job So ids cannot be probed.
Sign webhooks and dedupe by event id Delivery is at least once.