Part 5 of 7 · 3 min read
Disk-bound case
Think of a clerk who must stamp each page, then wait for a courier to carry it to the archive before stamping the next. The clerk's hands are free most of the time but the work does not advance. The analogy stops in one place: a courier works by foot, and a disk waits for a device that may be fast or slow.
In plain words: The program syncer writes 4 KiB and calls fsync, which waits until the data is on the device (Level 8, Level 9), then repeats.
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char **argv) {
alarm(argc > 1 ? atoi(argv[1]) : 10);
int fd = open("/tmp/l8/l12/sync.dat", O_WRONLY | O_CREAT | O_TRUNC, 0644);
char buf[4096] = {1};
for (;;) {
write(fd, buf, sizeof buf);
fsync(fd); /* wait until the data is on the device */
}
}
During the run the numbers were:
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 0 7780304 2152 271164 0 0 0 47844 25336 43854 0 24 45 32 0
1 0 0 7767908 2164 286576 0 0 12 44992 24235 41174 1 22 45 33 1
PID %CPU STAT WCHAN COMMAND
978 24.5 D submit_bio_wait syncer
A worked example. Utilisation: us is about 0, and id is 45 per cent, so the processors are not busy with user work. Saturation: b is 1 (a process blocked in uninterruptible sleep) and wa is 32 to 33 per cent: time in which a core was idle only because a process waited for I/O. The process is in state D and waits in the kernel function submit_bio_wait (page 1: the wchan column), which means it waits for a block request to complete. Throughput: bo is 47,844 blocks per second, close to 47 MB/s with the machine's 1 KiB blocks, which is about 11,900 writes of 4 KiB per second. The system share sy of 22 to 24 and the interrupt and context-switch counts (25,336 and 43,854 per second) are high, since each sync causes a request, an interrupt and a wake-up. The load average rises, even though the CPU is half idle, because a process in state D counts in the load (Level 3).
The key lines. The signature of a disk-bound workload is non-zero b and wa, a process in state D, and a wchan that names a block or file-system function. A faster processor would not help. A faster device, fewer syncs, or batching of writes would (the group commit of Level 8).
I/O wait
I/O wait (wa in vmstat) is the share of time in which a processor was idle while at least one process was blocked waiting for disk I/O to finish. It is a sign that the workload is limited by the disk, not by the processor.
Example: wa of 33 per cent in the syncer test means a third of the time went to waiting for the device.
Check yourself:
1. What do b and wa show in the disk-bound test?
2. In what state, and where, did the process wait?
3. Why does the load average rise although the processors are half idle?