Part 5 of 6 · 2 min read
Why high percentiles matter
Why high percentiles matter
Think of a relay race where the whole team's time is decided by its slowest runner, not its average runner. One bad leg ruins the result no matter how fast the others ran.
In plain words: High percentiles matter because they directly shape the experience of specific real users, often the same users repeatedly, such as customers who make many requests and are therefore more likely to eventually hit a slow one. A slow response is frequently caused by queueing: a server can only handle a limited number of requests at once, so a few slow requests ahead in line delay every request behind them, an effect called head-of-line blocking. The problem compounds when one end-user request depends on several backend calls, even made in parallel, since the overall request can only finish once its single slowest backend call finishes, which means the chance of hitting at least one slow call rises with every additional backend dependency added.
A worked example. The bookstore's order page calls three backend services in parallel: inventory, pricing, and recommendations. Each has a p99 of 200 ms individually, but because the page must wait for the slowest of the three, the page's own overall p99 is measurably worse than any single service's p99 alone.
The key lines. No new SQL; this is a latency-composition pattern across services.
inventory call: p99 = 200ms
pricing call: p99 = 200ms
recommendations call: p99 = 200ms
page response = max(all three) -> page's own p99 is worse than any single call's p99
Head-of-line blocking, tail latency
Head-of-line blocking is the delay a request experiences because it is stuck waiting behind other requests already being processed, even though the request itself is fast to handle once it starts. Tail latency refers to the response times at the high percentiles, the small share of unusually slow requests.
Example: the order page's overall slowness above comes from needing every one of three parallel calls to finish, so its tail latency is worse than any one call's tail latency alone.
Quick notes
- Load is described with a few chosen load parameters, picked to match the system's actual bottleneck, such as fan-out for a system where one write triggers many downstream operations.
- Throughput measures work completed per unit time, the natural fit for batch jobs; response time measures what a client experiences, the natural fit for online systems.
- Latency is specifically the waiting portion of a response; response time is the full total including latency, service time, and network delay.
- The average response time hides how many users actually experienced a delay; percentiles, especially the median and high percentiles like p95, p99, and p999, describe it far better.
- High percentiles matter because they often repeatedly affect the same users, and because head-of-line blocking and multi-call fan-out both make tail latency worse than it looks from any single component.