Part 9 of 12 · 2 min read
Mistakes that waste a budget
Common mistakes, and what the example does not do.
Mistakes that waste a budget
- Optimising cost per call instead of per resolved task.
- Switching to a cheaper model with no quality floor or eval.
- No per-run cap, so one loop eats a day's budget.
- Volatile content early in the prompt, destroying cache hits.
- Semantic cache with a loose threshold, or shared across tenants.
- Router with no fallback, or a verifier nobody evaluated.
- Ignoring tail latency of escalations.
- Hard-coding prices and routes.
- Counting only output tokens; agents pay mostly for input.
What the example does not do
About this example What the example does not do
- All prices, success curves, distributions and rates are assumed; conclusions are directional, not measurements of any real model or provider.
- No real tokenisation, no real calls, no real caching behaviour (providers differ on cache lifetimes, minimum sizes and write prices).
- The verifier and classifier are random processes with fixed accuracy, not models with correlated errors.
- Task difficulty is a single number; real tasks differ in kind.
- Latency is a simple formula; no queuing, rate limits or network.
- The exact-cache figure depends entirely on the assumed repeat distribution.
- Optional real experiment: replay 500 logged tasks against two real models and a cheap verifier, and compute cost, success and p95 latency for large-only, small-only, a router and a cascade.
Check yourself
A cheap model costs one tenth as much per call. Why is cheaper per call not the right measure?
Show answer
Cost per resolved task is. The small model cost $0.0013 per task but succeeded only 64.4 percent of the time, and a failed task still has to be handled.
For the curious Routers and caches buy savings with errors
A classifier router cost $0.0059 per task but succeeded only 77.2 percent of the time, because hard tasks sent to the small model fail. A cascade (small model, cheap verifier, escalate) reached 90.9 percent at about 47 percent of the large-only cost, with a worse p95 latency (14.3 s against 10.8 s). A semantic cache at threshold 0.75 gave 100 percent hits and 66 percent wrong answers. Prices and curves are assumed.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Track cost per resolved task | Not per call or per token. |
| Cap each run in dollars | A cap must end the run in a defined state, not a crash. |
| Keep the prompt prefix stable | Put variable content last so the cache can hit. |
| Budget latency as well as cost | Time to first token and time per output token. |