CSE Roadmap

Core tier · about 6 hours · 13 parts

Part 5 of 13 · 2 min read

Find the bottleneck by perturbation, then weigh what a router or cache costs in quality

Guessing which stage is slow is a coin flip; halving each stage one at a time and measuring the effect is not.

Think of tracking down which bulb is dimming a whole string of fairy lights. Instead of guessing, you swap each bulb one at a time and watch the string's total brightness. The one swap that changes the reading the most is your answer.

In plain words: sensitivity analysis means changing one input at a time, with the same random numbers held fixed elsewhere, and measuring the change in the output. The stage whose change moves the result most is the real bottleneck. This same build-and-measure habit is what lets you judge whether a router (sending requests to a cheaper model) or a cache (reusing an answer) is actually worth its hidden cost in quality.

A worked example. Halving each stage's latency one at a time, the planner LLM call saves 2.69 seconds of p95, by far the largest single lever, followed by the CRM tool (0.85s) and the answer LLM (0.70s). Every guardrail stage saves under 0.2 seconds. Separately, adding a router that sends 60% of requests to a small model cuts cost from $0.0090 to $0.0052 and p95 from 5.1s to 4.2s, but drops assumed quality from 0.84 to 0.78, below the 0.80 target. A cache set to a 20% hit rate shifts the median latency down 37% at a 40% hit rate, but the p95 only 9%. That's because a cache miss still takes the full slow path; the tail is whatever the cache can't catch.

The key lines. From code/mini-sysdesign/budget.py:

# from code/mini-sysdesign/budget.py
def sensitivity(node, factor=0.5, n=100_000, seed=0):
    """p95 saving from making each distinct stage `factor` times as slow (same random numbers)."""
    base = simulate(node, n, seed)["p95"]
    out = {}
    for nm in sorted({s.name for s in stages_of(node)}):
        out[nm] = base - simulate(scale_stage(node, nm, factor), n, seed)["p95"]
    return out

Read it as: the same seed is passed to every simulation, both the baseline and every perturbed version. Holding the random numbers fixed is what makes the comparison fair. Any difference in the resulting p95 comes from the one stage that changed, not from new random noise, which is exactly how a controlled experiment is supposed to work.

Sensitivity analysis, router and cache

Sensitivity analysis changes one input (halving one stage's latency) with the same random numbers and measures the change in the output. A router sends each request to a cheaper or specialised model; an exact cache reuses an answer for an identical request, a semantic cache for a similar one.

Example: the planner LLM topping the sensitivity ranking at 2.69 seconds is the number that should drive a design decision. Adding a router or cache without first running this check risks optimizing a stage that was never the bottleneck at all.