Part 3 of 8 · 3 min read
Percentiles and tails
The average hides the slow requests that matter.
DDIA says response time is not a single number but a distribution. Even the same request twice gives different times, from context switches, lost packets, garbage collection pauses, page faults and more. The mean says little about the typical experience. DDIA recommends percentiles. Sort the times from fastest to slowest: the median (p50) is the halfway point. Higher percentiles (p95, p99, p999) show how bad the outliers are.
Percentile
The p99 response time is the value that 99 percent of requests beat. One request in a hundred is slower.
Example: a p99 of 1.2 seconds means 1 in 100 requests takes 1.2 seconds or more.
Why the tail matters. Amazon, DDIA reports, states internal response time targets at the 99.9th percentile. The slowest requests tend to come from the customers with the most data, who are often the most valuable. DDIA also cites that a 100 ms rise in response time cut Amazon's sales by 1 percent. On the other side, optimising the 99.99th percentile was judged too costly for the benefit.
Latency and response time are not the same. Response time is the whole wait the client sees: service time, network delays and time spent queueing. Latency, in DDIA's usage, is the time a request waits to be handled. A server can process only a few requests in parallel, so a few slow requests hold up the ones behind them. This is head-of-line blocking. Queueing is often the main part of high-percentile response time. Therefore measure on the client side.
Tail latency amplification. If one user request needs many backend calls, it must wait for the slowest. The chance that at least one is slow grows with the number of calls. A worked example (made up): suppose 1 backend call in 100 is slow (p99). A request that makes 1 call is slow 1 percent of the time. With 10 calls, the chance that all 10 are fast is 0.99 to the power 10, which is 0.904, so 9.6 percent of requests are slow. With 100 calls it is 1 − 0.99^100 = 1 − 0.366 = 63.4 percent. A tail that looks small at one service becomes the typical experience for the user.
Aggregating. DDIA warns that an average of percentiles taken on several machines has no mathematical meaning. Merge the histograms instead. UDS gives a related tip for dashboards: a long range with daily resolution suits capacity planning, while a short range with a one-minute resolution suits a live incident.
Targets. Percentiles appear in service level objectives (SLOs) and agreements (SLAs). DDIA's sample: a median under 200 ms and a p99 under 1 second, up at least 99.9 percent of the time. UDS advises starting with comfortable targets and tightening them. Work back from what users care about. It also notes that anything above three nines is very costly and gives diminishing returns.
Check yourself: 1. Why is the mean a poor measure of the typical user's wait? 2. In the amplification example, why does a small tail become common? 3. Why must response time be measured on the client side?