Part 4 of 7 · 3 min read
A fixed pool
In plain words: The second design applies page 5. The program starts a fixed number of worker threads at the beginning. The main thread puts each accepted socket into a queue (protected by a mutex, with condition variables for "not empty" and "not full"), and a worker takes one out and serves it:
static void *worker(void *a) {
(void)a;
for (;;) {
pthread_mutex_lock(&qm);
while (qn == 0) pthread_cond_wait(&qne, &qm); /* while, not if (page 5) */
int fd = q[qh]; qh = (qh + 1) % QMAX; qn--;
pthread_cond_signal(&qnf);
pthread_mutex_unlock(&qm);
serve(fd); /* the whole connection */
}
}
The number of threads is now fixed. With 8 workers, the server showed 9 threads (8 workers and the main thread) in /proc/<pid>/status, however many clients came.
There is a catch, and the test shows it. A worker serves one whole connection before it takes the next one. A connection that lasts 10 requests holds its worker for 10 × 20 ms = 200 ms. With a pool of W workers, only W clients can be served at the same time. The others wait in the queue.
pool of 4: 8 clients -> 0.41 s 32 clients -> 1.62 s
pool of 8: 8 clients -> 0.20 s 16 clients -> 0.41 s 24 clients -> 0.61 s
pool of 16: 8 clients -> 0.20 s 32 clients -> 0.41 s
The times match a simple rule. The clients are served in waves of W, and each wave takes 0.20 s. So the total is about 0.20 × (number of clients ÷ W, rounded up). Check it: 32 clients on 4 workers make 8 waves, 8 × 0.20 = 1.6 s, and 1.62 s was measured. 24 clients on 8 workers make 3 waves, 0.60 s, and 0.61 s was measured.
In every run, the first client finished after 0.20 s, but the last one waited up to 1.62 s. A client that arrives late and waits for a free worker sees poor service although the server is idle most of the time, since each worker spends nearly all its time asleep.
This is the hidden limit of the design: the concurrency limit, the number of requests or clients that can be in progress at one time, equals the pool size. A pool sized for processor-bound work (page 5, about the number of processors) is too small for work with long waits.
Concurrency limit
The concurrency limit of a server is the greatest number of clients it can serve at the same moment. In a design where each worker serves one whole connection, it equals the number of workers.
Example: a pool of 4 workers serves 4 clients at once, so 32 clients with 200 ms of work each need 8 waves, or about 1.6 s.
Check yourself: 1. What was the total time for 24 clients on a pool of 8, and why? 2. Why did the pool of 16 serve 32 clients in 0.41 s? 3. What would help the late clients most?