CSE Roadmap

Core tier · about 5 hours · 7 parts

Part 6 of 7 · 4 min read

Tracing a leak

From a growing number to a line of code

A report says that a service grows. The procedure narrows the cause in steps, and each step uses one of the tools above.

  1. Is it growing? Sample VmRSS over time. The leaking request handler of page 2 gave 27,304, 53,192, 78,892 and 104,592 kB after 25, 50, 75 and 100 requests: a steady rise of about 1,030 kB per request. A rise that stops at a level is a cache or a start-up cost, and not a leak.
  2. Which kind? Watch the three parts. If RssAnon grows, the program allocates. If RssFile grows, it maps more files. If RssShmem grows, a shared object grows.
  3. Which mapping? Compare pmap -x before and after. A growing [heap] line points to malloc of small blocks. A growing number of anonymous lines points to large blocks, or to mmap without munmap.
  4. Is it a leak or fragmentation? Compare arena and uordblks from mallinfo2. If the live bytes are small and the arena is big, the heap is fragmented. If the live bytes are big, the program holds the data.
  5. Which line? Run the program under a heap profiler. Valgrind's Massif tool, run on the leaking handler (valgrind --tool=massif ./leak), drew the rising curve to 100 MB, and ms_print named the place: 99.99% of the heap at handle_request (leak.c:5), called from main (leak.c:7). For a quick check of leaked blocks, use Memcheck or ASan (page 2).
Symptom Likely cause Tool
RssAnon rises steadily with work leak Massif, ASan, Memcheck
RssAnon rises and then levels cache or start-up cost mallinfo2, the program's own counters
Large arena, small uordblks heap fragmentation mallinfo2, a different allocator or arena
RssAnon jumps with huge pages on memory bloat (page 6) AnonHugePages in smaps_rollup
RssFile rises files mapped, not unmapped pmap -x, lsof
Large VmSize, small VmRSS reserved, unused (stacks, big arrays) pmap -x

Heap profiler

A heap profiler is a tool that records every allocation of a running program with the call stack that made it, and reports which places hold the most memory at the peak, such as Valgrind's Massif.

Example: Massif reported 99.99% of the heap at handle_request (leak.c:5), which is the malloc that was never freed.

Interview angle

"A server's memory grows over a week. How do you investigate?" A good answer follows the five steps: confirm the growth over time, split it into anonymous, file and shared parts, find the mapping, decide leak or fragmentation or cache, and use a profiler for the allocation site. It adds that RSS and PSS differ for forked servers.

Check yourself: 1. How can you tell a cache from a leak by sampling VmRSS? 2. Which tool found the line leak.c:5? 3. What does a growing RssFile suggest?

Words you will meet

Word Meaning
VmSize The total size of all mappings
VmRSS The resident set size
RssAnon, RssFile, RssShmem The three parts of the resident set
PSS Resident memory with shared pages divided among the sharers
smaps The per-mapping memory file of a process
pmap A tool that prints the mappings
Dirty page A page changed and not yet written out
Arena The heap of brk as the allocator sees it
Heap profiler A tool that records where memory is allocated
Massif The heap profiler of Valgrind

Quick notes

  • The resident set size (RSS) of a process is the amount of its memory that is held in physical frames now, shown as VmRSS in /proc/PID/status and as RSS or.
  • RssAnon, RssFile and RssShmem are the three parts of a process's resident set in /proc/PID/status: anonymous pages with no file, pages of mapped files, and pages of shared memory objects.
  • The proportional set size (PSS) of a process is its resident memory in which each page shared with other processes is counted as a fraction, the page size divided by.
  • /proc/PID/smaps is a file that lists every mapping of a process with its size, resident size, proportional size, dirty and clean pages, and the pages in huge pages, and pmap -x and /proc/PID/smaps_rollup show the same data in compact forms.
  • A heap profiler is a tool that records every allocation of a running program with the call stack that made it, and reports which places hold the most memory at the peak, such as Valgrind's Massif.