Part 6 of 13 · 3 min read
What the example found
The build-up table, naive-sum comparison, retry curve, sensitivity ranking, fixes, and the assumption sweeps.
How to read the build-up table: each row adds one more box to the architecture, in the order the module builds it up.
| Step added | p50 (s) | p95 (s) | p99 (s) | Cost per request | Availability | Assumed quality | Misses (p95 8 s, $0.02, 99.5%, q 0.80) |
|---|---|---|---|---|---|---|---|
| Model only | 2.0 | 4.6 | 6.7 | $0.0080 | 100.0% | 0.84 | none |
| + context | 2.3 | 5.0 | 7.0 | $0.0085 | 100.0% | 0.84 | none |
| + guardrails | 2.5 | 5.1 | 7.2 | $0.0090 | 99.89% | 0.84 | none |
| + router | 1.7 | 4.2 | 6.1 | $0.0052 | 99.88% | 0.78 | quality |
| + cache | 1.5 | 4.0 | 5.8 | $0.0042 | 99.91% | 0.78 | quality |
| + agent loop | 4.7 | 10.4 | 13.1 | $0.0125 | 99.91% | 0.78 | latency, quality |
What to notice: the router is the point where the design first misses a target, and it misses on quality specifically, not on the latency or cost it was added to improve. That's a reminder that every box's benefit has to be checked against every SLO, not just the one it was meant to fix.
How to read the retry table: each row allows one more retry per LLM call on the full design.
| Retries per LLM call | Availability | p99 (s) |
|---|---|---|
| 0 | 98.61% | 12.8 |
| 1 | 99.91% | 13.2 |
| 2 | 99.91% | 13.2 |
| 3 | 99.91% | 13.2 |
What to notice: all the gain from retries happens going from 0 to 1. Retries 2 and 3 buy nothing more in this independent-failure model, while p99 creeps up slightly, because a failed attempt still costs its timeout before the retry fires.
How to read the sensitivity table: each row halves one stage's latency; the number is the resulting saving in request p95.
| Halve one stage | p95 saved (s) |
|---|---|
| planner LLM | 2.69 |
| CRM tool | 0.85 |
| answer LLM | 0.70 |
| small answer model | 0.20 |
| retrieval | 0.16 |
What to notice: the planner LLM alone accounts for more p95 saving than the next three stages combined. It's the single clearest lever in the whole design, found by measurement rather than guessing.
How to read the fixes table: each row is a candidate fix for the agent-loop design's latency miss, against the same 8-second target.
| Fix to the latency miss | p95 (s) | Cost | Misses |
|---|---|---|---|
| As designed | 10.4 | $0.0125 | latency |
| Cap loop at 2 rounds | 9.1 | $0.0115 | latency |
| Small planner model | 7.7 | $0.0063 | none (planner quality not modelled) |
| Both | 7.0 | $0.0061 | none (planner quality not modelled) |
What to notice: swapping to a small planner model clears the latency target on its own, cheaper than capping the loop does. But the table's own caveat is the important part: planning quality for the smaller model was never measured. So "no misses" here means "nothing modeled failed," not "this is safe to ship."
How to read the assumption-sweep table: each row varies one assumed input (cache hit rate or router share) across the full design.
| Assumption sweep (full design) | p50 (s) | p95 (s) | Cost |
|---|---|---|---|
| Cache hit rate 0% | 5.5 | 10.8 | $0.0156 |
| 10% | 5.1 | 10.6 | $0.0140 |
| 20% | 4.7 | 10.4 | $0.0125 |
| 40% | 3.5 | 9.8 | $0.0094 |
| Router share 0% | 11.3 | $0.0156 | |
| 90% | 9.8 | $0.0109 |
What to notice: the median (p50) moves much more than the p95 as the cache hit rate rises. A cache helps the typical request a lot. But a cache miss still has to take the full slow path, so the tail barely benefits by comparison.
{{svg:buildup|Adding boxes one at a time: p95, cost and assumed quality.}} {{svg:tail|Sum of percentiles vs simulation, sensitivity, and three fixes.}}