Part 3 of 7 · 4 min read
Measure first
Think of counting both the parcels a courier is given and the trips the courier makes. If the courier makes one trip for each parcel, the trips are the cost, not the parcels. The analogy stops in one place: a trip carries many parcels at no extra cost, and a device write also carries a fixed overhead.
In plain words: The program writes 5,000 records of 4 KiB and makes the data safe with fsync, and it does so at four different intervals: after every record, after every 10, after every 100, and only at the end. It also reads the writes-completed counter of the disk from /proc/diskstats before and after each run, so that the writes that reach the device can be counted.
A worked example.
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
static double now(void) {
struct timespec t; clock_gettime(CLOCK_MONOTONIC, &t);
return t.tv_sec + t.tv_nsec / 1e9;
}
static unsigned long long dev_writes(const char *dev) { /* writes completed, from /proc/diskstats */
FILE *f = fopen("/proc/diskstats", "r");
char name[64], rest[512]; int maj, min; unsigned long long v[5];
while (fscanf(f, "%d %d %63s %llu %llu %llu %llu %llu", &maj, &min, name,
&v[0], &v[1], &v[2], &v[3], &v[4]) == 8) {
if (strcmp(name, dev) == 0) { fclose(f); return v[4]; }
if (!fgets(rest, sizeof rest, f)) break;
}
fclose(f); return 0;
}
int main(int argc, char **argv) {
if (argc < 3) return 1;
const char *path = argv[1], *dev = argv[2];
int n = 5000, every[] = {1, 10, 100, 0};
char rec[4096]; memset(rec, 'x', sizeof rec);
for (int m = 0; m < 4; m++) {
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0644);
unsigned long long w0 = dev_writes(dev);
double t0 = now();
for (int i = 1; i <= n; i++) {
if (write(fd, rec, sizeof rec) != sizeof rec) return 1;
if (every[m] && i % every[m] == 0) fsync(fd);
}
fsync(fd);
double t = now() - t0;
unsigned long long w1 = dev_writes(dev);
printf("fsync every %3d: %6.3f s, %6.0f records/s, %5llu device writes, %.2f records per device write\n",
every[m], t, n / t, w1 - w0, (double)n / (double)(w1 - w0));
close(fd); unlink(path);
}
return 0;
}
Output of the first run on the test machine:
fsync every 1: 0.689 s, 7262 records/s, 15002 device writes, 0.33 records per device write
fsync every 10: 0.108 s, 46090 records/s, 1502 device writes, 3.33 records per device write
fsync every 100: 0.059 s, 84963 records/s, 152 device writes, 32.89 records per device write
fsync every 0: 0.034 s, 148897 records/s, 22 device writes, 227.27 records per device write
The first row is the service as it is: 7,262 records per second, which is 0.689 s / 5,000 = 138 microseconds per record. The disk did 15,002 writes for 5,000 records, which is 3 for each record. That is 15,002 / 0.689 s = 21,774 device writes per second, far from the limit of a flash disk, so the disk is not saturated.
The key lines. The number of device writes is almost exactly 3 for each fsync in the first three rows: 15,002 / 5,000 = 3.0, 1,502 / 500 = 3.0 and 152 / 50 = 3.04. Each fsync is one data write plus about two more, most likely the file system's journal records. Level 9 explains the journal. The time of a run follows the number of fsync calls, not the number of records.
fsync
fsync(fd) is a system call that forces all changed data and the needed metadata of one open file to reach the storage device, and it returns only when the device reports that this is done. Without it, written data may stay in the page cache for many seconds.
Example: a program that calls fsync after every record is safe against a crash after each record, and it waits for the device each time.
Check yourself:
1. How many device writes does one fsync cause in the output?
2. Why is the disk not saturated at 21,774 writes per second?
3. How many records per device write does the last row show, and why is it so large?