Part 6 of 7 · 3 min read
Measuring the benefit
Do they make the program faster
In plain words: The pointer-chasing test of Level 6 page 5 reads one pointer in each page, in random order, so that each read needs a different TLB entry. The same test was run with 4 KiB pages and with huge pages (madvise):
| Region size | 4 KiB pages | Huge pages |
|---|---|---|
| 2 MiB (512 pages) | 4.3 to 5.8 ns per access | 1.8 ns |
| 16 MiB (4,096 pages) | 17 to 19 ns | 33 to 34 ns |
| 128 MiB (32,768 pages) | 44.5 ns | 34.8 ns (one run each) |
| 256 MiB (65,536 pages) | 54 to 57 ns | 68 to 81 ns |
The first row is the clean case. A 2 MiB region fits in the cache, so the only difference is translation. The huge page needs one TLB entry, and the access time fell from about 4.3 ns to 1.8 ns, the same as the smallest test. The other rows do not follow the textbook. At 16 MiB and at 256 MiB, huge pages were slower, and only at 128 MiB were they faster. The data no longer fits in the cache there, so most of the time goes to waiting for main memory. The way cache lines of a huge page spread over the cache sets may also differ from that of scattered 4 KiB frames. This test, run on a virtual machine, does not show which effect is at work. It does show that the benefit of huge pages is not guaranteed, and that it depends on the program, the cache and the machine.
The rule is to measure the real workload. Count TLB misses with a tool such as perf stat -e dTLB-load-misses where the hardware allows it, compare the run time with THP on and off, and watch the resident size.
madvise
madvise is a system call by which a program tells the kernel how it expects to use a range of its memory. Examples are MADV_HUGEPAGE (use huge pages if you can), MADV_NOHUGEPAGE (do not) and MADV_DONTNEED (the contents are no longer needed).
Example: madvise(buf, 64 << 20, MADV_HUGEPAGE) asks the kernel to back the 64 MiB region with huge pages.
Interview angle
"What are huge pages, and when would you use them?" A good answer: they are 2 MiB or 1 GiB pages that cut TLB misses, page table size and fault counts for large regions. The cost is internal waste and some latency. The answer names the explicit pool and THP, and ends with "measure first".
Check yourself:
1. In the table, in which row did huge pages show the clean benefit, and why?
2. What advice follows from the other rows?
3. What does MADV_NOHUGEPAGE ask for?
Words you will meet
| Word | Meaning |
|---|---|
| Huge page | A page of 2 MiB or 1 GiB instead of 4 KiB |
| Base page | The ordinary page, 4 KiB on x86-64 |
| TLB reach | Entries times page size |
| Explicit huge pages | A reserved pool that programs request by name |
MAP_HUGETLB |
The flag that takes pages from the pool |
| Transparent huge pages (THP) | Huge pages chosen by the kernel |
AnonHugePages |
The amount of anonymous memory held in huge pages |
| Compaction | Moving pages together to make a contiguous range |
| Memory bloat | Resident memory larger than the data it holds |
madvise |
A hint to the kernel about how memory will be used |
Quick notes
- A huge page is a page much larger than the base page, such as 2 MiB or 1 GiB on x86-64 against 4 KiB.
- The explicit huge page pool is a number of huge pages that the administrator reserves in advance.
- Transparent huge pages (THP) are a Linux feature in which the kernel backs suitable anonymous regions with 2 MiB pages, with no change to the program.
- Memory bloat is the growth of a process's resident memory beyond what it uses.
madviseis a system call by which a program tells the kernel how it expects to use a range of its memory.