Part 5 of 7 · 2 min read
Trade-offs
In plain words: Two numbers describe a server. Throughput is the number of requests finished per second. Latency is the time one request or client waits from its start to its end. A design may be good at one and poor at the other. From the 32-client tests (320 requests in all):
| Design | Total time | Throughput | Slowest client |
|---|---|---|---|
| Thread per connection | 0.20 s | about 1,600 requests/s | 0.20 s |
| Pool of 16 | 0.41 s | about 780 requests/s | 0.41 s |
| Pool of 4 | 1.62 s | about 200 requests/s | 1.62 s |
The throughput figures come from 320 divided by the total time. In this workload, where threads mostly sleep, the throughput is set by the concurrency limit, not by the two processors.
The machine sets upper limits, whatever the design:
- Open files. Each connection uses one file descriptor, and a process may open a limited number.
ulimit -nprinted 20,000 here, and many systems use 1,024 by default. - Threads. The kernel limits the total (
/proc/sys/kernel/threads-maxshowed 64,036) and the per-user count (ulimit -ushowed 32,018). - Memory. Every thread's stack takes address space and some real memory.
- Scheduling cost. Thousands of runnable threads make the scheduler (Level 3) work harder, and the context switches of Level 1, page 5 add up.
A third design avoids the per-client thread: one thread (or one per processor) uses poll or epoll to watch all connections and serve whichever is ready, as in page 5. It scales to many idle connections cheaply, and it needs code written as small steps that never block, which is harder to write. A common mix is one event-loop thread per processor, each handling thousands of connections.
Throughput and latency
Throughput is the number of requests a system completes per unit of time. Latency is the time one request takes from its arrival to its reply.
Example: 320 requests finished in 0.20 s is a throughput of 1,600 requests per second, while the slowest client waiting 1.62 s is a latency of 1.62 s.
Check yourself: 1. Compute the throughput of 320 requests in 0.41 s. 2. Name two limits that stop a thread-per-connection server from growing without bound. 3. Why is an event loop cheaper for many idle connections?