Definitions

Definitions

The levers

Results

1. The cascade matched large-model quality at roughly half the cost, paying in tail latency.
2. The router was cheap but lost quality because misrouted hard tasks fail with no second chance.
3. A cap turned a $12 run into a $0.48 one.
4. Caching helps most on long runs and stable prefixes.

Design rules

• Track cost per resolved task with traces (Level 9), split by step, model and tenant.
• Set a quality floor first, then minimise cost above it.
• Cap every run (dollars, steps, wall time) and define what happens at the cap.
• Cache the stable prefix: system prompt and tools first, volatile data last; measure hit rate.
• Use exact caches for deterministic lookups; treat semantic caches as risky and keep thresholds high, scoped per tenant and excluded for sensitive queries.
• Prefer cascades over blind routers where you can verify answers cheaply; evaluate the verifier.
• Compress context: summarise, truncate tool output, drop stale steps (Level 7).
• Batch non-urgent work and keep interactive traffic separate.
• Budget latency too: TTFT, time to first progress event (module 4), total.
• Re-evaluate routing when models or prices change; keep the policy in config, not code.

Interview questions, with the follow-ups

Common mistakes

• Optimising cost per call instead of per resolved task.
• Switching to a cheaper model with no quality floor or eval.
• No per-run cap, so one loop eats a day's budget.
• Volatile content early in the prompt, destroying cache hits.
• Semantic cache with a loose threshold, or shared across tenants.
• Router with no fallback, or a verifier nobody evaluated.
• Ignoring tail latency of escalations.
• Hard-coding prices and routes.
• Counting only output tokens; agents pay mostly for input.

What the example does not do

• All prices, success curves, distributions and rates are assumed; conclusions are directional, not measurements of any real model or provider.
• No real tokenisation, no real calls, no real caching behaviour (providers differ on cache lifetimes, minimum sizes and write prices).
• The verifier and classifier are random processes with fixed accuracy, not models with correlated errors.
• Task difficulty is a single number; real tasks differ in kind.
• Latency is a simple formula; no queuing, rate limits or network.
• The exact-cache figure depends entirely on the assumed repeat distribution.
• Optional real experiment (I did not run this): replay 500 logged tasks against two real models and a cheap verifier, and compute cost, success and p95 latency for large-only, small-only, a router and a cascade.

For the curious Routers and caches buy savings with errors

A classifier router cost $0.0059 per task but succeeded only 77.2 percent of the time, because hard tasks sent to the small model fail. A cascade (small model, cheap verifier, escalate) reached 90.9 percent at about 47 percent of the large-only cost, with a worse p95 latency (14.3 s against 10.8 s). A semantic cache at threshold 0.75 gave 100 percent hits and 66 percent wrong answers. Prices and curves are assumed.

What you must be able to say (one page)

• Measure cost per resolved task, not per call or per token. A cheap model that fails is expensive per success. In my simulation (20,000 tasks, assumed prices and difficulty curves) the large model cost $0.0150 per task at 91.1 percent success; the small model $0.0013 at 64.4 percent; per resolved task that is $0.0165 versus $0.0019, but a 64 percent success rate is a product failure, so the target is the cheapest policy above a quality floor. Every price and curve here is invented for the lab: use your own measurements.
• Agents cost more than their step count suggests. Each step re-sends the history, so input cost grows roughly with the square of the steps: for a large model with a 2,000-token system prompt and 850 tokens added per step, a 5-step run cost $0.083, 10 steps $0.23, 20 steps $0.72, 40 steps $2.45. Fewer steps and shorter context are the first savings.
• Prompt caching reuses an unchanged prefix (system prompt, tools, earlier turns). In my simulation (cached reads at a tenth of the input price, assumed) the same runs cost $0.04, $0.08, $0.18 and $0.45 at 5, 10, 20 and 40 steps: savings grow with run length. Keep the prefix stable (put variable content last) or the cache never hits.
• Exact and semantic caches. An exact cache returns a stored answer only for an identical request. A semantic cache matches similar queries through embeddings, and its value is more doubtful, because embeddings, vector search and similarity thresholds can all fail. My simulation shows the mechanism: with assumed similarity distributions, threshold 0.95 gave 16 percent useful hits and 1 percent wrong answers; 0.85 gave 84 percent and 16 percent; 0.75 gave 100 percent and 66 percent. A looser threshold buys hits with wrong answers. Do not semantic-cache anything user-specific, time-sensitive or high-stakes. The exact-cache hit rate in my run (92 percent) is an artefact of a heavily repeating assumed query stream; a real agent workload often repeats far less.
• Model routing. Use a powerful model for complex queries and a lightweight one for routine tasks; dynamic routing evaluates each request and picks a model by complexity or urgency. In my simulation a classifier router (85 percent accurate, threshold 0.45) cost $0.0059 per task but succeeded only 77.2 percent of the time, because hard tasks sent to the small model fail; its cost per resolved task fell from $0.0097 to $0.0068 as classifier accuracy rose from 0.6 to 0.95. A cascade (small model first, a cheap verifier, escalate to the large model on a failed check) cost $0.0070 per task at 90.9 percent success, about 47 percent of the large-only cost for the same success, with a worse p95 latency (14.3 s versus 10.8 s) because escalated tasks pay for both. The verifier quality (90 percent true-positive, 15 percent false-positive, assumed) drives the result; a verifier that wrongly passes bad answers silently lowers quality.
• Per-task budget caps bound the tail. With 2 percent of runs looping for 400 steps at $0.03, the uncapped mean was $0.49 per run and the worst run $12.00; a $0.50 cap cut the mean to $0.25, the worst run to $0.48, and stopped 105 of 5000 runs. A cap must end the run in a defined state (partial result, escalation), not just crash (Level 9 budgets).
• Latency is a separate budget. Time to first token (TTFT) and time per output token (TPOT) combine as total latency = TTFT + TPOT x output tokens. Escalation adds latency, caching lowers TTFT, streaming (module 4) lowers the perceived wait, and batch APIs suit non-urgent work, so separate online and batch traffic.
• Honest limits: simulated tasks, curves and prices; no real model calls; the router and verifier are statistical stand-ins.