CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 5 of 8 · 1 min read

Averages and tails

The mean hides the people who suffer.

Response times are a distribution, usually right-skewed with a long tail. UDS gives a case: 100 requests, 99 take 1 second and one takes 10 minutes (600 seconds). The average is (99 + 600) ÷ 100 = 6.99 seconds, nearly 7 times what 99 percent of users saw. One outlier moved it.

Percentiles are better. The 99th percentile (p99) is the value below which 99 percent of requests fall. The top percentiles, such as p99 and p99.9, are the long-tail latencies. (Page 6 of Level 8 and DDIA covered them.) UDS adds two reasons the tail matters:

  • It hits your most profitable users. They make the most requests, so they have the best chance of meeting a slow one. UDS cites studies that a 100 ms delay in load time can hurt conversion by 7 percent.
  • It costs capacity. UDS uses Little's law: a service with 2,000 threads serving 10,000 requests per second has an average thread time of 200 ms (2,000 ÷ 10,000). A network switch gets congested, and 1 percent of requests, which is 100 per second, now take 20 seconds. Those slow requests alone need 100 × 20 = 2,000 threads. The thread count must double to handle a slowness that touches only 1 percent of users.

UDS's conclusion: keeping the tail in check makes users happier, improves resilience and lowers operating cost. Guarding against the worst case tends to improve the average case too.

Check yourself: 1. In the 99-and-1 example, why is the average 6.99 seconds and not 1? 2. Why do profitable users meet tail latency more often? 3. In the thread example, how many extra threads do the slow requests need?