Part 7 of 9 · 2 min read
Slow requests multiply
Tail latency amplification: why one slow call in a hundred slows most pages.
Think of a dinner party where the host serves everyone at the same moment. If one dish is late, everyone waits for it. The meal is only as fast as the slowest dish.
Many user requests are like this. A single page may need calls to ten or a hundred services. Even if the calls run side by side, the page waits for the slowest. This is called tail latency amplification.
Tail latency amplification
Tail latency amplification is the effect that a request needing many backend calls is slow if any one of them is slow, so the share of slow user requests is larger than the share of slow calls.
Example: a page that needs 100 calls is slow whenever at least one of the 100 is slow.
A worked calculation. Suppose 1 in 100 backend calls is slow, so 99 out of 100 are fast.
- The chance that one call is fast is 0.99.
- A page with 100 independent calls is fast only if all 100 are fast: 0.99 multiplied by itself 100 times, which is about 0.37.
- So the page is fast only 37 percent of the time. The page is slow about 63 percent of the time.
A slow rate of 1 percent per call became a slow rate of 63 percent per page. With 10 calls the figure is smaller: 0.99 to the power 10 is about 0.90, so about 10 percent of pages are slow. The more calls, the worse the effect.
This is why large systems care about high percentiles of every internal part, and why they measure response time at the client. A server that looks fast may still make the user wait, because requests queue before the server sees them.
There is a second effect, called head-of-line blocking. A server can handle only a few requests at once. A few slow requests at the front of the line hold up everything behind them, even requests that are quick to process.
Interview angle
Answer in your head first, then open it.
One page calls many services. How does that change its speed?
When a design has a request that touches many services, say out loud: "tail latency adds up across calls, so I want timeouts and the option to answer without the slowest part." That one sentence shows knowledge of how large systems behave.
Check yourself:
- A page makes 50 calls and 1 in 200 is slow. Roughly how often is the page slow?
- Why measure response time at the client and not only at the server?