CSE Roadmap

Core tier · about 5 hours · 7 parts

Part 6 of 7 · 3 min read

Choosing

Choosing and hardening

No design wins everywhere. The table gives the usual choice:

Workload Good fit
A few hundred clients, long connections, simple code wanted Thread per connection
Many short requests, work uses the processor Fixed pool (size about the number of processors)
Many requests with waits, queue per request Pool with a larger size, or a pool that takes requests, not connections
Tens of thousands of mostly idle connections Event loop, perhaps with one thread per processor

In plain words: The pool of section 3 served whole connections. The fix for its limit is to put requests in the queue, not connections, which means that the threads need to know which connection a request came from and that the event loop or the readers must hand them out. That is the design of many real servers, and it combines page 5's pool with an event loop that reads the requests.

Real servers also need care that the test left out:

  1. Check every return value (accept, read, write, pthread_create). A failed pthread_create under load must not crash the server.
  2. Limit the queue and the connection count, so that a flood cannot use all memory.
  3. Protect shared state: the test counted answered requests under a mutex, and the queue is shared by the mutex of page 4.
  4. Use timeouts so that a client that stops sending does not hold a worker forever.
  5. Block SIGPIPE or use MSG_NOSIGNAL, because writing to a closed connection otherwise kills the process (Level 1, page 6).

Common mistake

Sizing a pool for processor-bound work when the requests mostly wait, or the reverse. With waits, the pool must be large (page 5 gave workers about processors × (1 + wait/compute)). The test showed the result: a pool of 4 took 8 times longer than a thread per connection.

Interview angle

Answer in your head first, then open it.

How does a threaded server handle many clients?

One thread per connection, a fixed pool, or an event loop, each with different limits.

What is the drawback of thread per connection?

Memory per thread, kernel limits, and no bound on load.

Why can a pool be slow though the machine is idle?

If workers hold whole connections and sleep, the pool size is the concurrency limit.

What is the C10K problem?

Serving ten thousand connections at once, which led to event-driven designs.

How is a server shut down safely?

Stop accepting, let workers finish, then join them.

Check yourself: 1. Which design fits 20,000 mostly idle connections? 2. Why must the server ignore or block SIGPIPE? 3. What is the change that lifts the limit of section 3?

Words you will meet

Word Meaning
Socket One end of a communication channel, used as a file
Port A number that selects the program for an incoming connection
Loopback address An address that leads back to the same machine
Listening socket The socket that only accepts new connections
Concurrency limit The most clients served at the same moment
Throughput Requests finished per second
Latency The time one request waits for its reply

Quick notes

  • A socket is an operating-system object that represents one end of a communication channel, identified to the program by a file descriptor, and used with read and write like a file.
  • Thread per connection is a server design in which the program creates a new thread for each accepted connection and lets that thread serve the client until the connection closes.
  • The concurrency limit of a server is the greatest number of clients it can serve at the same moment.
  • Throughput is the number of requests a system completes per unit of time.