Part 6 of 9 · 3 min read
Describing performance
Throughput, response time, and why the average hides the slow requests.
Once load is described, ask what happens to performance when load grows. Two measures are common.
Throughput
Throughput is how much work a system finishes in a given time, such as requests per second or records processed per hour.
Example: a batch job that handles 2 million records in one hour has a throughput of about 555 records a second.
Response time
Response time is the time between a user sending a request and getting the answer. It includes the work itself and all waiting: in queues, on the network and for other requests.
Example: a user taps "buy" and the confirmation appears after 180 milliseconds.
Response time is not one number. It varies, even for identical requests, because of random delays: a busy network, a pause in a program, a disk that took longer. So it is better to think of a distribution, many values, with a shape.
A worked example. Ten requests to one service take these times in milliseconds: 80, 85, 90, 90, 95, 100, 105, 110, 120, 1,800.
- The average is the sum divided by 10. The sum is 2,675, so the average is 267.5 ms.
- Most users, nine out of ten, waited 120 ms or less. The average of 267 ms does not describe any real request.
- The median is the middle value. Sorted, the fifth and sixth are 95 and 100, so the median is about 97.5 ms. Half of the requests are faster, half are slower.
Percentile
A percentile is a threshold below which a given share of requests fall. The 95th percentile, written p95, is the time that 95 out of 100 requests beat. The median is the 50th percentile, or p50.
Example: if p95 is 1.5 seconds, then 95 out of 100 requests take under 1.5 seconds and 5 take 1.5 seconds or longer.
High percentiles, such as p99 and p999, show how bad the slowest requests are. They are called tail latencies, because they sit on the long tail of the distribution.
Why care about the slow ones? The book gives two reasons. First, the slowest requests often come from the most valuable customers, because they have the most data, for example the people with the most purchases. Second, speed affects money: a 100 millisecond delay was reported to reduce sales by one percent at a large store.
But there is a limit. Chasing the very last percentile can cost far more than it earns, because random events outside the designer's control dominate there.
Service level objective
A service level objective, or SLO, is a target for a measure such as response time or availability. It is often written with a percentile. Level 9 explains how teams use it.
Example: "p99 response time under 1 second, and up 99.9 percent of the time."
Common mistake
Reporting only the average response time. A single slow request can push the average up, and many slow requests can hide behind many fast ones. Report the median and at least one high percentile.
Check yourself:
- Using the ten values above, what is the highest value that p90 could be?
- Why is the median a better "typical" number than the average here?