CSE Roadmap

Common tier · about 4 hours · 7 parts

Part 5 of 7 · 2 min read

Memory bloat

In plain words: The price of a bigger page is the one the shelf analogy gave. A program that touches only one byte of a 2 MiB region, as a sparse data structure does, is given the whole page. The test measured it:

Region RSS growth after writing one byte in a new 2 MiB region
MADV_NOHUGEPAGE 4 kB
MADV_HUGEPAGE 2,048 kB

The resident size grew by 512 times as much. A program with many sparse regions, such as a process with thousands of small heaps, can use far more memory with THP than without. This is called memory bloat, and it is the main reason that many systems leave THP on madvise and not always. Some database vendors recommend turning THP off for their products for this reason and for the next two.

  • Latency. Zeroing a 2 MiB page takes longer than zeroing 4 KiB, and a fault that must compact memory may stall the program for a long time. The measured first touch was faster in total, and a single fault is slower.
  • Copy-on-write. After fork (page 6 of Level 6), a write to a shared huge page must copy 2 MiB (or the kernel first splits the page into 512 small ones), not 4 KiB.

Memory bloat

Memory bloat is the growth of a process's resident memory beyond what it uses. It is caused by pages larger than the data placed in them, as when one byte in a 2 MiB region makes the kernel give a whole huge page.

Example: writing one byte into a fresh 2 MiB region raised the resident size by 2,048 kB with MADV_HUGEPAGE and by 4 kB without it.

Common mistake

"Huge pages are always a free speed-up" is false. They save translation work and faults, and they cost memory and sometimes latency. Turn them on for large, dense, long-lived regions, and measure before and after.

Check yourself: 1. Why does a sparse data structure suffer under THP? 2. Why can a single fault be slower with a huge page? 3. What does fork do to a shared huge page when a child writes to it?