Part 6 of 8 · 2 min read
A budget in practice
A one-page budget that catches runaway agents before the invoice does.
Before shipping, write a one-page budget:
- Average steps per task (measure it, do not guess), and average input and output per step.
- Cost per task from the formulas above, with and without caching.
- Tasks per day times that cost, plus 30% for retries and peaks. (The 30% is a common safety margin, not a rule.)
- A hard cap per task (maximum steps, maximum tokens) and a daily alert on spend. An agent stuck in a loop is the most common way to burn money quickly.
For the curious Why caching helps most for long runs
Each call repeats almost the whole previous call, with one round added. With caching, the repeated part is billed at the low price, so mostly only the new round is billed at the full input price. In the 40-step demo, 793,650 of the 833,000 input tokens are billed at the cached price. That is why the cost curve with caching is much flatter.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Cost per task in the logs | Every run records steps, input tokens, output tokens and the dollars they cost (Level 0, module 6). |
| Caching-friendly prompts | Stable content first, variable content last, no edits to earlier messages. |
| Hard caps | A maximum number of steps, tokens and dollars per run, with a clear stop reason. |
| Retry helpers | A shared function for backoff with jitter and a retry-after hint, reused by every API call. |
| Concurrency limits | A semaphore or bucket sized from arrival rate times latency, and below the provider's limit. |
| Spend alerts | A daily alert when spend or token use jumps. |
Draft: to be checked against current provider documentation (prompt caching rules, batch discounts, rate-limit headers) before this page goes live.
Interview angle
Answer in your head first, then open it.
Why does an agent cost so much more than a chatbot?
Each step re-sends the whole history, so total input grows roughly with the square of the number of steps.
How would you cut the cost of a long-running agent?
Cache the stable prefix, trim or summarise history, shrink tool results, use a cheaper model for easy steps, and cap steps.
What is TTFT and what affects it?
The wait for the first token; it depends mainly on input length because of prefill.
How do you handle a 429 rate-limit error?
Back off with jitter, respect any retry-after hint, limit concurrency with a bucket or semaphore, and only retry idempotent operations.
How do you size concurrency?
Arrival rate times latency (Little's law).