Part 3 of 13 · 2 min read
Percentiles do not add; simulate the whole request instead
Summing up each stage's worst-case latency assumes every stage has its bad day on the same request, which almost never happens.
Think of a relay race where each runner has a personal-best and a personal-worst lap time. The team's worst-case total time is not four runners simultaneously having their worst lap, because that's an extremely unlikely coincidence.
In plain words: the tail latency (p95 or p99) of a multi-stage request is not the sum of each stage's own tail latency, because not every stage is slow on the same request at once. A Monte Carlo simulation that samples each stage's latency from its own distribution and adds the draws gives a realistic request-level percentile. Adding the percentiles directly, instead, systematically overstates it.
A worked example. For the full ten-stage design, summing each stage's own p95 gives 12.6 seconds. Simulating the actual request — sampling each stage's latency independently per simulated request and summing those draws — gives a request p95 of 10.4 seconds. The naive sum overstates the real tail by roughly 20%. In the other direction, for the simplest model-only design, the simulated p95 (4.6s) came out slightly above the listed component p95 (4.5s). That's because retry timeouts on rare failures add a little extra tail the naive number doesn't capture at all.
The key lines. From code/mini-sysdesign/budget.py:
# from code/mini-sysdesign/budget.py
def simulate(node, n=100_000, seed=0):
"""Summary of the request: percentiles are over successful requests only."""
rng = np.random.default_rng(seed)
lat, cost, ok = run(node, rng, n)
L = lat[ok]
return {"availability": float(ok.mean()),
"p50": float(np.percentile(L, 50)), "p95": float(np.percentile(L, 95)),
"p99": float(np.percentile(L, 99)), "mean_cost": float(cost[ok].mean())}
Read it as: the percentile is computed once, over the full simulated array L of n complete requests. Each simulated request draws its own random latency for every stage it passes through. So the p95 reflects how often several stages are simultaneously slow on the same request, which is a much rarer event than any single stage being slow on its own.
Tail latency
p95 / p99 is the latency that 95% / 99% of requests stay under. The tail is what users remember.
Example: the 12.6-versus-10.4-second gap above is the tail-latency arithmetic error in miniature. It's exactly the mistake an interviewer is listening for when they ask you to estimate end-to-end latency from a list of component numbers.