Part 5 of 7 · 2 min read
Testing with dd
Think of checking a road by driving a lorry along it once, then driving a motorbike with a small load a hundred times. The two runs answer different questions. The analogy stops in one place: a road does not change between runs, while a cache between a program and a disk can.
In plain words: The dd command copies data and prints the rate. With iflag=direct it skips the page cache, so it tests the device and not memory. With a large block size it tests throughput, and with a small block size it tests the rate of requests. It sends one request at a time, so it cannot show how a disk behaves at a high queue depth. A more complete tool such as fio can set the queue depth, the mix of reads and writes and the access pattern.
A worked example. On the test machine:
$ dd if=big of=/dev/null bs=1M iflag=direct count=512
536870912 bytes (537 MB, 512 MiB) copied, 0.293388 s, 1.8 GB/s
$ dd if=big of=/dev/null bs=4k iflag=direct count=20000
81920000 bytes (82 MB, 78 MiB) copied, 0.592486 s, 138 MB/s
The large requests ran at 1.8 GB/s and the small ones at 138 MB/s, a ratio of 13. In the small test the disk completed 20,000 / 0.592 = 33,800 reads per second, so 138 MB/s is IOPS times 4 KiB. The gap shows that each request has a fixed cost, as in page 1, even though this disk has no arm.
The key lines. A single dd run is a rough tool. It does not repeat, and the first run of a fresh file may be slow because of a host cache (see page 5). Run a test at least three times, discard a first run that is far from the rest, and note the block size and flags with the result.
Queue depth
Queue depth is the number of I/O requests that are submitted to a device and not yet finished at one moment. Flash disks reach their highest IOPS only at a depth of several or many requests. Hard disks gain less, since they have one arm.
Example: a program with 32 threads that each wait for one read has a queue depth of up to 32.
Interview angle
A common question is "How would you find out whether the disk is the bottleneck?" A good answer reads utilisation and latency from iostat -x or /proc/diskstats, compares the queue depth with what the device can handle, and checks with dd or fio what the device can give on its own.
Check yourself:
1. Why does dd with iflag=direct test the device and not memory?
2. How many reads per second does the second dd run show?
3. Why can dd not show the best IOPS of a flash disk?