Part 5 of 7 · 2 min read
Pool size
In plain words: A pool that is too small leaves processors idle, and one that is too large adds switching cost and uses memory. The right size depends on whether the tasks mostly compute or mostly wait.
A CPU-bound task spends nearly all its time computing. More workers than processors cannot make the work finish sooner, since each processor can run only one at a time. A good size is about the number of processors (nproc).
An I/O-bound task spends much of its time waiting for a disk or a network. While one worker waits, another can use the processor, so more workers than processors pay off. A common rule of thumb, not an exact law, is:
workers ≈ processors × (1 + wait time ÷ compute time)
For example, with 4 processors and tasks that compute for 10 ms and wait for 90 ms, the ratio of wait to compute is 9, so the estimate is 4 × (1 + 9) = 40 workers. With 40 workers, there are about 4 tasks computing at any moment and 36 waiting. For pure computing the wait is 0, and the formula gives 4, the number of processors.
The rule is a start. The way to decide is to measure the throughput with a few sizes and pick the knee of the curve.
The saving over thread-per-request can be measured. The test below does 20,000 tiny tasks, each a loop of 100 multiplications. One version creates and joins a thread per task. The other uses a pool of 4 workers:
20000 tiny tasks: thread per task 653 ms, pool of 4 14 ms
20000 tiny tasks: thread per task 730 ms, pool of 4 16 ms
20000 tiny tasks: thread per task 642 ms, pool of 4 12 ms
On average, 675 ms against 14 ms, about 48 times faster with the pool. The tasks are so small that nearly all of the thread-per-task time goes to creating and ending threads. With longer tasks the gap shrinks, as the creation cost matters less, but it never reverses.
CPU-bound and I/O-bound
A CPU-bound task is one whose run time is set by the processor's speed, because it spends almost all its time computing. An I/O-bound task is one whose run time is set by waiting for input or output, such as the disk or network, because it spends most of its time waiting.
Example: compressing a file is CPU-bound, and downloading it over a slow link is I/O-bound.
Check yourself: 1. How many workers suit a CPU-bound pool on 8 processors? 2. Estimate the pool size for 8 processors and tasks with 5 ms of computing and 45 ms of waiting. 3. Why does the saving shrink for longer tasks?