Part 6 of 7 · 4 min read
Counting interrupts
Seeing interrupts on a real system
Think of a postman who rings the bell once for each parcel or once for a whole bag. The number of bell rings tells you how the parcels were packed. The analogy stops in one place: a postman chooses how to pack, and in an OS the packing is the request size and the batching of the driver.
In plain words: Linux counts all interrupts in /proc/stat (the intr line) and in /proc/interrupts for each source and each processor. Reading a large file with O_DIRECT in requests of different sizes shows how the number of interrupts changes with the work. Large requests give the driver fewer, larger commands, and so fewer interrupts for the same bytes.
A worked example. The program below reads 256 MiB with the request size from its command line, and prints the change in the total interrupt count. The count includes timer and other interrupts, so it is an upper bound for the disk.
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
static unsigned long long total_intr(void) { /* first number after "intr" in /proc/stat */
FILE *f = fopen("/proc/stat", "r");
char line[8192]; unsigned long long v = 0;
while (fgets(line, sizeof line, f))
if (sscanf(line, "intr %llu", &v) == 1) break;
fclose(f);
return v;
}
int main(int argc, char **argv) {
if (argc < 3) return 1;
size_t req = (size_t)atol(argv[2]); /* request size in bytes */
int fd = open(argv[1], O_RDONLY | O_DIRECT);
if (fd < 0) { perror("open"); return 1; }
char *buf; if (posix_memalign((void **)&buf, 4096, req)) return 1;
unsigned long long before = total_intr();
long reads = 0; long long bytes = 0; ssize_t n;
while (bytes < (256LL << 20) && (n = read(fd, buf, req)) > 0) { bytes += n; reads++; }
unsigned long long after = total_intr();
printf("request %7zu B: %6ld reads, %lld MiB, interrupts %llu, per read %.2f\n",
req, reads, bytes >> 20, after - before, reads ? (double)(after - before) / reads : 0.0);
return 0;
}
On the test machine (virtual disk):
request 4096 B: 65536 reads, 256 MiB, interrupts 109920, per read 1.68
request 65536 B: 4096 reads, 256 MiB, interrupts 5127, per read 1.25
request 1048576 B: 256 reads, 256 MiB, interrupts 704, per read 2.75
The bytes moved for each interrupt grow from 268,435,456 / 109,920 = 2,442 bytes (4 KiB requests) to 268,435,456 / 704 = 381,300 bytes (1 MiB requests). The total falls about 156 times. The numbers vary from run to run, since the count contains unrelated interrupts, but the trend holds.
The key lines. /proc/interrupts has one row for each interrupt source, with one count for each processor. Run cat /proc/interrupts before and after a large copy and look for the row that grew.
Interrupt rate
The interrupt rate is the number of interrupts that a processor or a device causes in each second. The cost is the rate times the cost of one interrupt. Large requests, batching and polling at high load all reduce it.
Example: 704 interrupts for a 256 MiB read of 1 MiB requests, against 109,920 for the same data in 4 KiB requests.
Check yourself:
1. Which line of /proc/stat holds the total interrupt count?
2. Why are the counts of the program only an upper bound for the disk?
3. What happens to the interrupts per byte when the request size grows?
Words you will meet
| Word | Meaning |
|---|---|
| Controller | The circuit that the processor talks to for a device. |
| Register | A small named storage cell in a controller. |
| Memory-mapped I/O | Device registers placed in the ordinary address space. |
| Polling | Asking the device again and again if it is ready. |
| DMA | Moving data between device and memory without the processor. |
Quick notes
- A device is driven through its controller's status, command and data registers.
- Port-mapped I/O uses special instructions. Memory-mapped I/O uses ordinary loads and stores.
- Polling is simple and fast to react, and it wastes the processor for the whole wait.
- An interrupt costs a fixed amount, and it is cheap for slow devices and costly at a high rate.
- DMA moves the data itself and raises one interrupt per transfer. The buffer needs a physical address and must stay pinned.
- Larger requests mean fewer interrupts for the same data.