Set a maximum run time and budget per job and report how it ended (Level 9 budgets).
Version the API and keep old job ids valid.
Interview questions, with the follow-ups
How would you expose a 10-minute agent over HTTP?
"Not as one blocking request. POST returns 202 and a job id; the client polls a status endpoint with Retry-After or gets a signed webhook; DELETE cancels. In my simulation a 2-minute proxy timeout answered 459 of 1000 sync requests, and retries added 541 duplicate runs." Follow-up: "Why not WebSockets?" For progress streaming they are good (module 4), but you still need a durable job id so the run survives the connection.
What is an idempotency key and how do you implement it?
"A client-chosen token sent with the submit. The server stores (owner, key) with the body hash and job id; a repeat with the same body returns the original job, a different body returns 409. With 5 percent lost responses it removed all 47 duplicate jobs in my test." Follow-up: "How long do you keep keys?" Long enough to cover client retry windows, such as 24 hours; document it.
How do you make webhooks reliable and safe?
"At-least-once delivery with exponential backoff, a stable event id so receivers can dedupe, and an HMAC over timestamp and body so they can verify origin and reject replays. A lost ack means the receiver may have processed an event the sender thinks failed." Follow-up: "Exactly-once?" Not on the wire; you get effectively-once by dedupe on the receiver.
How do you stop clients hammering the status endpoint?
"Retry-After, a documented backoff, rate limits, and webhooks as the alternative. Polling every 2 seconds cost 30,000 requests for 100 ten-minute jobs; backoff cut it to 1,400." Follow-up: "What goes in the status?" State, progress, cost so far, and the result link when terminal.
Common mistakes
Holding the HTTP connection for the whole run.
No idempotency on submit, so retries bill twice.
Guessable or sequential job ids and no owner check.
403 on other owners' ids, which confirms existence.
Webhooks without signatures or without a replay window.
Processing events twice because the receiver does not dedupe.
Cancel that only flips a flag while the worker keeps spending.
No maximum run time or budget per job.
What the example does not do
No real network, framework, storage or queue; the clock is a parameter.
Exponential run time and fixed loss rates are assumptions.
Retry timing is not simulated (no sleeping), only attempt counts.
Cancel is a state change; it does not stop a real worker (module 3 covers that).
No authentication layer; the owner is a string.
Optional real experiment (I did not run this): put a 2-minute proxy in front of a real 5-minute agent endpoint, call it with and without an idempotency key through a flaky network, and count duplicate runs and bills.
Check yourself
A client calls an agent endpoint and the proxy times out after 2 minutes, but the run takes 3. The client retries. What do you pay for? (Both runs. In the simulation, retrying on timeout answered 715 of 1000 requests but ran 1541 jobs, 541 of them duplicates. A job API returns 202 with a job id at once, and an idempotency key makes a retried submit return the same job.)
For the curious A webhook can apply twice even when you acknowledge nearly all of them
With 30 percent lost acknowledgements and up to 6 attempts, 999 of 1000 events were acknowledged, but the receiver applied all 1000, and a lost ack means the sender tries again. Sign each event, check the timestamp, and keep a record of event ids so a second delivery is acknowledged without being applied again. Durations and loss rates here are assumed; there is no real HTTP server.
What you will use daily at work
Practice
What it looks like
Return a job id, never hold the connection
POST returns 202; the client polls or gets a webhook.
Send an idempotency key with every submit
The same key and body gives the same job; a different body gives 409.
Return 404 for another owner job
So ids cannot be probed.
Sign webhooks and dedupe by event id
Delivery is at least once.
Draft: confirm the timeouts of your own gateway and load balancer, and your provider webhook retry rules.
What you must be able to say (one page)
An agent run can take minutes, so a single blocking HTTP request is the wrong interface. Proxies, gateways and browsers time out, clients retry, and the server keeps working on abandoned requests. The standard fix is a job API: POST /runs returns 202 Accepted with a job id immediately; the client polls GET /runs/{id} or receives a webhook; DELETE /runs/{id} cancels.
Why sync fails, in my simulation (assumed numbers): 1000 requests, mean run 3 minutes, proxy timeout 2 minutes. With no retry only 459 were answered. When clients retried on timeout, 715 were answered but the server ran 1541 jobs, 541 duplicates you pay for.
Idempotency keys stop duplicate jobs. If the response to a submit is lost, the client retries. With 5 percent of responses lost, 1000 submits created 47 duplicate jobs without a key and 0 with an Idempotency-Key. Rules my code enforces: same owner and key and same body returns the original job (200); same key with a different body returns 409; keys are scoped per owner.
The job is a state machine:queued -> running -> succeeded | failed | cancelled; queued -> cancelled is also legal. Terminal states never change and illegal transitions raise. Cancellation is idempotent and returns 200 on an already-finished job.
Status endpoint: return state and, while not terminal, a Retry-After hint so clients do not hammer you. Return 404 for another owner's job id, not 403, so ids cannot be probed.
Polling versus webhooks (requests to learn that 100 ten-minute jobs finished): poll every 2 s = 30,000; every 10 s = 6,000; backoff from 2 s doubling to 60 s = 1,400; webhook = 100. Webhooks are cheaper but need the client to host an endpoint, so offer both.
Webhooks are at-least-once, so sign and dedupe. Sign HMAC(secret, timestamp.body), check the timestamp within a window (replay protection), compare in constant time, and dedupe by a stable event id. In my simulation with 30 percent lost acks and up to 6 attempts, 999 of 1000 events were acknowledged (mean 1.43 attempts) but the receiver applied all 1000, because one was applied and never acknowledged. The sender cannot tell "not delivered" from "delivered, ack lost".
User control: users must be able to interrupt, pause and cancel long-running work, usually with a cancellation token the worker checks between steps. The API's cancel endpoint is the network version of that.
Honest limits: in-memory, logical clock, no real HTTP server, no persistence, assumed durations and loss rates.