CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that waste a budget

Common mistakes, and what the example does not do.

Mistakes that waste a budget

  • Optimising cost per call instead of per resolved task.
  • Switching to a cheaper model with no quality floor or eval.
  • No per-run cap, so one loop eats a day's budget.
  • Volatile content early in the prompt, destroying cache hits.
  • Semantic cache with a loose threshold, or shared across tenants.
  • Router with no fallback, or a verifier nobody evaluated.
  • Ignoring tail latency of escalations.
  • Hard-coding prices and routes.
  • Counting only output tokens; agents pay mostly for input.

What the example does not do

About this example What the example does not do
  • All prices, success curves, distributions and rates are assumed; conclusions are directional, not measurements of any real model or provider.
  • No real tokenisation, no real calls, no real caching behaviour (providers differ on cache lifetimes, minimum sizes and write prices).
  • The verifier and classifier are random processes with fixed accuracy, not models with correlated errors.
  • Task difficulty is a single number; real tasks differ in kind.
  • Latency is a simple formula; no queuing, rate limits or network.
  • The exact-cache figure depends entirely on the assumed repeat distribution.
  • Optional real experiment: replay 500 logged tasks against two real models and a cheap verifier, and compute cost, success and p95 latency for large-only, small-only, a router and a cascade.

Check yourself

A cheap model costs one tenth as much per call. Why is cheaper per call not the right measure?

Show answer

Cost per resolved task is. The small model cost $0.0013 per task but succeeded only 64.4 percent of the time, and a failed task still has to be handled.

For the curious Routers and caches buy savings with errors

A classifier router cost $0.0059 per task but succeeded only 77.2 percent of the time, because hard tasks sent to the small model fail. A cascade (small model, cheap verifier, escalate) reached 90.9 percent at about 47 percent of the large-only cost, with a worse p95 latency (14.3 s against 10.8 s). A semantic cache at threshold 0.75 gave 100 percent hits and 66 percent wrong answers. Prices and curves are assumed.

What you will use daily at work

Practice What it looks like
Track cost per resolved task Not per call or per token.
Cap each run in dollars A cap must end the run in a defined state, not a crash.
Keep the prompt prefix stable Put variable content last so the cache can hit.
Budget latency as well as cost Time to first token and time per output token.