CSE Roadmap

Core tier · about 6 hours · 13 parts

Part 6 of 13 · 3 min read

What the example found

The build-up table, naive-sum comparison, retry curve, sensitivity ranking, fixes, and the assumption sweeps.

How to read the build-up table: each row adds one more box to the architecture, in the order the module builds it up.

Step added p50 (s) p95 (s) p99 (s) Cost per request Availability Assumed quality Misses (p95 8 s, $0.02, 99.5%, q 0.80)
Model only 2.0 4.6 6.7 $0.0080 100.0% 0.84 none
+ context 2.3 5.0 7.0 $0.0085 100.0% 0.84 none
+ guardrails 2.5 5.1 7.2 $0.0090 99.89% 0.84 none
+ router 1.7 4.2 6.1 $0.0052 99.88% 0.78 quality
+ cache 1.5 4.0 5.8 $0.0042 99.91% 0.78 quality
+ agent loop 4.7 10.4 13.1 $0.0125 99.91% 0.78 latency, quality

What to notice: the router is the point where the design first misses a target, and it misses on quality specifically, not on the latency or cost it was added to improve. That's a reminder that every box's benefit has to be checked against every SLO, not just the one it was meant to fix.

How to read the retry table: each row allows one more retry per LLM call on the full design.

Retries per LLM call Availability p99 (s)
0 98.61% 12.8
1 99.91% 13.2
2 99.91% 13.2
3 99.91% 13.2

What to notice: all the gain from retries happens going from 0 to 1. Retries 2 and 3 buy nothing more in this independent-failure model, while p99 creeps up slightly, because a failed attempt still costs its timeout before the retry fires.

How to read the sensitivity table: each row halves one stage's latency; the number is the resulting saving in request p95.

Halve one stage p95 saved (s)
planner LLM 2.69
CRM tool 0.85
answer LLM 0.70
small answer model 0.20
retrieval 0.16

What to notice: the planner LLM alone accounts for more p95 saving than the next three stages combined. It's the single clearest lever in the whole design, found by measurement rather than guessing.

How to read the fixes table: each row is a candidate fix for the agent-loop design's latency miss, against the same 8-second target.

Fix to the latency miss p95 (s) Cost Misses
As designed 10.4 $0.0125 latency
Cap loop at 2 rounds 9.1 $0.0115 latency
Small planner model 7.7 $0.0063 none (planner quality not modelled)
Both 7.0 $0.0061 none (planner quality not modelled)

What to notice: swapping to a small planner model clears the latency target on its own, cheaper than capping the loop does. But the table's own caveat is the important part: planning quality for the smaller model was never measured. So "no misses" here means "nothing modeled failed," not "this is safe to ship."

How to read the assumption-sweep table: each row varies one assumed input (cache hit rate or router share) across the full design.

Assumption sweep (full design) p50 (s) p95 (s) Cost
Cache hit rate 0% 5.5 10.8 $0.0156
10% 5.1 10.6 $0.0140
20% 4.7 10.4 $0.0125
40% 3.5 9.8 $0.0094
Router share 0% 11.3 $0.0156
90% 9.8 $0.0109

What to notice: the median (p50) moves much more than the p95 as the cache hit rate rises. A cache helps the typical request a lot. But a cache miss still has to take the full slow path, so the tail barely benefits by comparison.

{{svg:buildup|Adding boxes one at a time: p95, cost and assumed quality.}} {{svg:tail|Sum of percentiles vs simulation, sensitivity, and three fixes.}}