Part 4 of 8 · 2 min read
Load testing
Push until it bends, and measure honestly.
A load test sends generated traffic at a system to learn how it behaves as load rises. The aim is to find the capacity: the load at which targets are no longer met.
DDIA gives one key rule. The load generator must send requests on its own clock, whatever the response time is. If each virtual client waits for its previous request to finish before sending the next, then a slow system receives less traffic, which keeps the queues short and flatters the results. Real users do not wait for each other. This is often called open-loop load. (The name is common practice, not DDIA's.)
Load test
A load test applies generated traffic to a system, with a rising rate, to find the load at which it stops meeting its targets.
Example: raise requests from 100 to 1,500 per second over 15 minutes, and watch for the point where p99 passes 1 second.
A worked example (made up). A service with 20 workers takes 50 ms per request, so it can finish 20 ÷ 0.05 = 400 requests per second. A closed-loop test with 20 virtual users each waiting for a reply never exceeds 400 per second, and shows low latency throughout. An open-loop test at 450 per second shows the truth: 50 requests per second pile up in the queue, so the wait grows each second, and after 10 seconds there are 500 queued. The first test reports "fine". The second reports the failure that real traffic would cause.
What to record at each step of the test:
- Throughput and goodput (successes per second).
- Response time at p50, p99 and p999, on the client side.
- Error rate and timeouts.
- Resource use on each tier: CPU, memory, connections, queue lengths.
The knee is the point where response time starts to climb steeply. Capacity is at or just below it. (The word "knee" is common practice.)
Where to test. UDS describes pre-production environments: lacking the realism of production, but able to catch hard failures early. It advises that a pre-production environment use the same metrics and alerts as production. Production-like data size and shape matter too. A test with a small database gives numbers that do not carry over. (Both realism points are common practice.)
Check yourself: 1. Why does a closed-loop test flatter the system? 2. In the example, how fast does the queue grow at 450 per second? 3. Which numbers do you record at each load step?