Part 6 of 7 · 3 min read
Choosing
Choosing and hardening
No design wins everywhere. The table gives the usual choice:
| Workload | Good fit |
|---|---|
| A few hundred clients, long connections, simple code wanted | Thread per connection |
| Many short requests, work uses the processor | Fixed pool (size about the number of processors) |
| Many requests with waits, queue per request | Pool with a larger size, or a pool that takes requests, not connections |
| Tens of thousands of mostly idle connections | Event loop, perhaps with one thread per processor |
In plain words: The pool of section 3 served whole connections. The fix for its limit is to put requests in the queue, not connections, which means that the threads need to know which connection a request came from and that the event loop or the readers must hand them out. That is the design of many real servers, and it combines page 5's pool with an event loop that reads the requests.
Real servers also need care that the test left out:
- Check every return value (
accept,read,write,pthread_create). A failedpthread_createunder load must not crash the server. - Limit the queue and the connection count, so that a flood cannot use all memory.
- Protect shared state: the test counted answered requests under a mutex, and the queue is shared by the mutex of page 4.
- Use timeouts so that a client that stops sending does not hold a worker forever.
- Block
SIGPIPEor useMSG_NOSIGNAL, because writing to a closed connection otherwise kills the process (Level 1, page 6).
Common mistake
Sizing a pool for processor-bound work when the requests mostly wait, or the reverse. With waits, the pool must be large (page 5 gave workers about processors × (1 + wait/compute)). The test showed the result: a pool of 4 took 8 times longer than a thread per connection.
Interview angle
Answer in your head first, then open it.
How does a threaded server handle many clients?
One thread per connection, a fixed pool, or an event loop, each with different limits.
What is the drawback of thread per connection?
Memory per thread, kernel limits, and no bound on load.
Why can a pool be slow though the machine is idle?
If workers hold whole connections and sleep, the pool size is the concurrency limit.
What is the C10K problem?
Serving ten thousand connections at once, which led to event-driven designs.
How is a server shut down safely?
Stop accepting, let workers finish, then join them.
Check yourself: 1. Which design fits 20,000 mostly idle connections? 2. Why must the server ignore or block SIGPIPE? 3. What is the change that lifts the limit of section 3?
Words you will meet
| Word | Meaning |
|---|---|
| Socket | One end of a communication channel, used as a file |
| Port | A number that selects the program for an incoming connection |
| Loopback address | An address that leads back to the same machine |
| Listening socket | The socket that only accepts new connections |
| Concurrency limit | The most clients served at the same moment |
| Throughput | Requests finished per second |
| Latency | The time one request waits for its reply |
Quick notes
- A socket is an operating-system object that represents one end of a communication channel, identified to the program by a file descriptor, and used with
readandwritelike a file. - Thread per connection is a server design in which the program creates a new thread for each accepted connection and lets that thread serve the client until the connection closes.
- The concurrency limit of a server is the greatest number of clients it can serve at the same moment.
- Throughput is the number of requests a system completes per unit of time.