Part 4 of 7 · 5 min read
A small iostat
Think of a speed camera that takes two photos of a car a known time apart. From the distance between the photos it gets the speed. The analogy stops in one place: a camera sees one car, and the counters add up all the processes that use the disk.
In plain words: The program below samples the counters of one device, starts a child that makes random 4 KiB reads with O_DIRECT for two seconds, samples the counters again and prints the rates. It is a tiny version of iostat -x.
A worked example. The test machine printed these lines in two runs.
reads/s 29685, read MB/s 121.6, avg read time 0.028 ms, busy 66%
reads/s 29992, read MB/s 122.8, avg read time 0.028 ms, busy 68%
The checks follow. 29,685 reads of 4,096 bytes per second is 121.6 MB/s. Little's law gives 29,685 x 0.028 ms = 0.83 requests in flight on average. This is below 1 because the program has one thread, and the busy share of 66% to 68% agrees: the device is busy most of the time, not all of it. The time for one read as the program sees it is 1 / 29,685 = 33.7 microseconds, and the device accounts for 28 of them.
The key lines.
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/wait.h>
#include <time.h>
#include <unistd.h>
struct st { unsigned long long rd, rd_sec, rd_ms, wr, wr_sec, wr_ms, busy_ms; };
static int sample(const char *dev, struct st *s) { /* one line of /proc/diskstats */
FILE *f = fopen("/proc/diskstats", "r");
char name[64]; unsigned long long v[11]; int maj, min;
if (!f) return -1;
while (fscanf(f, "%d %d %63s", &maj, &min, name) == 3) {
for (int i = 0; i < 11; i++) if (fscanf(f, "%llu", &v[i]) != 1) v[i] = 0;
char rest[256]; if (!fgets(rest, sizeof rest, f)) rest[0] = 0; /* newer kernels add fields */
if (strcmp(name, dev) == 0) {
s->rd = v[0]; s->rd_sec = v[2]; s->rd_ms = v[3];
s->wr = v[4]; s->wr_sec = v[6]; s->wr_ms = v[7]; s->busy_ms = v[9];
fclose(f); return 0;
}
}
fclose(f); return -1;
}
static void load(const char *path, double seconds) { /* child: random 4 KiB O_DIRECT reads */
int fd = open(path, O_RDONLY | O_DIRECT);
char *buf; if (fd < 0 || posix_memalign((void **)&buf, 4096, 4096)) exit(1);
long blocks = lseek(fd, 0, SEEK_END) / 4096;
struct timespec a, b; clock_gettime(CLOCK_MONOTONIC, &a);
do {
if (pread(fd, buf, 4096, (rand() % blocks) * 4096L) < 0) exit(1);
clock_gettime(CLOCK_MONOTONIC, &b);
} while ((b.tv_sec - a.tv_sec) + (b.tv_nsec - a.tv_nsec) / 1e9 < seconds);
exit(0);
}
int main(int argc, char **argv) {
if (argc < 3) return 1;
struct st s0, s1;
if (sample(argv[1], &s0)) { fprintf(stderr, "no such device\n"); return 1; }
if (fork() == 0) load(argv[2], 2.0);
wait(NULL);
sample(argv[1], &s1);
double secs = 2.0, n = (double)(s1.rd - s0.rd);
printf("reads/s %.0f, read MB/s %.1f, avg read time %.3f ms, busy %.0f%%\n",
n / secs, (s1.rd_sec - s0.rd_sec) * 512.0 / secs / 1e6,
n ? (double)(s1.rd_ms - s0.rd_ms) / n : 0.0,
(s1.busy_ms - s0.busy_ms) / (secs * 10.0));
return 0;
}
Compile it with gcc -Wall -Wextra mini.c -o mini and run it as ./mini vda FILE, where FILE is a file of at least a few hundred megabytes.
Utilisation
The utilisation of a device is the share of a time interval in which at least one request was in progress. It is computed from the io_ticks counter. A utilisation of 100% does not mean that the device is out of capacity if it can serve many requests in parallel, as flash disks can.
Example: a rise of 1,300 ms in io_ticks over 2,000 ms is a utilisation of 65%.
Common mistake
On a flash disk, a busy share of 100% does not prove that the disk is the limit. Flash serves many requests at once, so a disk that is "100% busy" with a queue depth of 1 can still serve many more requests per second when the queue depth is raised.
Check yourself: 1. Why does the program use two samples? 2. What does the average read time of 0.028 ms include? 3. Why is 0.83 requests in flight possible for a program with one thread?