Part 6 of 12 · 2 min read
What the simulation measured
Policy comparison, caching, the semantic-cache trade-off and a budget cap, each with a reading guide.
How to read the policy table: all four rows face the same 20,000 tasks; compare the "Per resolved" column, not "Cost per task", to see the real price of each policy.
| Policy (20,000 tasks, assumed) | Cost per task | Success | Per resolved | p50 / p95 latency |
|---|---|---|---|---|
| Large only | $0.0150 | 91.1% | $0.0165 | 10.8 / 10.8 s |
| Small only | $0.0013 | 64.4% | $0.0019 | 3.5 / 3.5 s |
| Router (85% accurate) | $0.0059 | 77.2% | $0.0076 | 3.5 / 10.8 s |
| Cascade | $0.0070 | 90.9% | $0.0077 | 3.5 / 14.3 s |
What to notice: the cascade nearly matches large-only on success and per-resolved cost. The router's cheap-looking per-call price hides a worse per-resolved price than either large-only or the cascade.
How to read the caching table: both columns run the identical agent; only whether the repeated history is cached differs.
| Steps in a run | Without cache | With prefix cache |
|---|---|---|
| 5 | $0.083 | $0.039 |
| 10 | $0.230 | $0.078 |
| 20 | $0.716 | $0.177 |
| 40 | $2.451 | $0.450 |
What to notice: the saving ratio grows with run length. Caching roughly halves the cost at 5 steps but cuts it to about a fifth at 40, because longer runs have more repeated history to cache.
How to read the semantic-cache table: read top to bottom as the threshold loosens; both columns move together.
| Semantic threshold | Useful hits | Wrong answers served |
|---|---|---|
| 0.95 | 16% | 1% |
| 0.90 | 50% | 4% |
| 0.85 | 84% | 16% |
| 0.80 | 98% | 38% |
| 0.75 | 100% | 66% |
What to notice: there is no threshold that gives both high hits and low wrong answers. Every gain in coverage is paid for in wrong answers, which is why semantic caching is a poor fit for anything user-specific or high-stakes.
A per-run budget cap bounds the other tail of the cost problem. That tail is the one run that loops, instead of the many runs that merely cost a bit more. Take 5,000 simulated runs, with 2 percent looping for 400 steps. The uncapped mean cost was $0.49 and the worst run $12.00. A $0.50 cap cut the mean to $0.25 and the worst run to $0.48. It stopped 105 of the 5,000 runs before they ran away.
Budget cap
A hard limit on spend, tokens or steps for one task, enforced while the run is still in progress.
Example: Budget(0.50) raises once a run's spend would exceed $0.50, instead of letting a looping run reach $12.
{{svg:levers|Measure, cap, cache, route, trim.}} {{svg:results|Policies, caching, budget caps and semantic-cache trade-offs.}}