CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 6 of 12 · 2 min read

What the simulation measured

Policy comparison, caching, the semantic-cache trade-off and a budget cap, each with a reading guide.

How to read the policy table: all four rows face the same 20,000 tasks; compare the "Per resolved" column, not "Cost per task", to see the real price of each policy.

Policy (20,000 tasks, assumed) Cost per task Success Per resolved p50 / p95 latency
Large only $0.0150 91.1% $0.0165 10.8 / 10.8 s
Small only $0.0013 64.4% $0.0019 3.5 / 3.5 s
Router (85% accurate) $0.0059 77.2% $0.0076 3.5 / 10.8 s
Cascade $0.0070 90.9% $0.0077 3.5 / 14.3 s

What to notice: the cascade nearly matches large-only on success and per-resolved cost. The router's cheap-looking per-call price hides a worse per-resolved price than either large-only or the cascade.

How to read the caching table: both columns run the identical agent; only whether the repeated history is cached differs.

Steps in a run Without cache With prefix cache
5 $0.083 $0.039
10 $0.230 $0.078
20 $0.716 $0.177
40 $2.451 $0.450

What to notice: the saving ratio grows with run length. Caching roughly halves the cost at 5 steps but cuts it to about a fifth at 40, because longer runs have more repeated history to cache.

How to read the semantic-cache table: read top to bottom as the threshold loosens; both columns move together.

Semantic threshold Useful hits Wrong answers served
0.95 16% 1%
0.90 50% 4%
0.85 84% 16%
0.80 98% 38%
0.75 100% 66%

What to notice: there is no threshold that gives both high hits and low wrong answers. Every gain in coverage is paid for in wrong answers, which is why semantic caching is a poor fit for anything user-specific or high-stakes.

A per-run budget cap bounds the other tail of the cost problem. That tail is the one run that loops, instead of the many runs that merely cost a bit more. Take 5,000 simulated runs, with 2 percent looping for 400 steps. The uncapped mean cost was $0.49 and the worst run $12.00. A $0.50 cap cut the mean to $0.25 and the worst run to $0.48. It stopped 105 of the 5,000 runs before they ran away.

Budget cap

A hard limit on spend, tokens or steps for one task, enforced while the run is still in progress.

Example: Budget(0.50) raises once a run's spend would exceed $0.50, instead of letting a looping run reach $12.

{{svg:levers|Measure, cap, cache, route, trim.}} {{svg:results|Policies, caching, budget caps and semantic-cache trade-offs.}}