Part 6 of 7 · 4 min read
Tracing a leak
From a growing number to a line of code
A report says that a service grows. The procedure narrows the cause in steps, and each step uses one of the tools above.
- Is it growing? Sample
VmRSSover time. The leaking request handler of page 2 gave 27,304, 53,192, 78,892 and 104,592 kB after 25, 50, 75 and 100 requests: a steady rise of about 1,030 kB per request. A rise that stops at a level is a cache or a start-up cost, and not a leak. - Which kind? Watch the three parts. If
RssAnongrows, the program allocates. IfRssFilegrows, it maps more files. IfRssShmemgrows, a shared object grows. - Which mapping? Compare
pmap -xbefore and after. A growing[heap]line points tomallocof small blocks. A growing number of anonymous lines points to large blocks, or tommapwithoutmunmap. - Is it a leak or fragmentation? Compare
arenaanduordblksfrommallinfo2. If the live bytes are small and the arena is big, the heap is fragmented. If the live bytes are big, the program holds the data. - Which line? Run the program under a heap profiler. Valgrind's Massif tool, run on the leaking handler (
valgrind --tool=massif ./leak), drew the rising curve to 100 MB, andms_printnamed the place: 99.99% of the heap athandle_request (leak.c:5), called frommain (leak.c:7). For a quick check of leaked blocks, use Memcheck or ASan (page 2).
| Symptom | Likely cause | Tool |
|---|---|---|
RssAnon rises steadily with work |
leak | Massif, ASan, Memcheck |
RssAnon rises and then levels |
cache or start-up cost | mallinfo2, the program's own counters |
Large arena, small uordblks |
heap fragmentation | mallinfo2, a different allocator or arena |
RssAnon jumps with huge pages on |
memory bloat (page 6) | AnonHugePages in smaps_rollup |
RssFile rises |
files mapped, not unmapped | pmap -x, lsof |
Large VmSize, small VmRSS |
reserved, unused (stacks, big arrays) | pmap -x |
Heap profiler
A heap profiler is a tool that records every allocation of a running program with the call stack that made it, and reports which places hold the most memory at the peak, such as Valgrind's Massif.
Example: Massif reported 99.99% of the heap at handle_request (leak.c:5), which is the malloc that was never freed.
Interview angle
"A server's memory grows over a week. How do you investigate?" A good answer follows the five steps: confirm the growth over time, split it into anonymous, file and shared parts, find the mapping, decide leak or fragmentation or cache, and use a profiler for the allocation site. It adds that RSS and PSS differ for forked servers.
Check yourself:
1. How can you tell a cache from a leak by sampling VmRSS?
2. Which tool found the line leak.c:5?
3. What does a growing RssFile suggest?
Words you will meet
| Word | Meaning |
|---|---|
VmSize |
The total size of all mappings |
VmRSS |
The resident set size |
RssAnon, RssFile, RssShmem |
The three parts of the resident set |
| PSS | Resident memory with shared pages divided among the sharers |
smaps |
The per-mapping memory file of a process |
pmap |
A tool that prints the mappings |
| Dirty page | A page changed and not yet written out |
| Arena | The heap of brk as the allocator sees it |
| Heap profiler | A tool that records where memory is allocated |
| Massif | The heap profiler of Valgrind |
Quick notes
- The resident set size (RSS) of a process is the amount of its memory that is held in physical frames now, shown as
VmRSSin/proc/PID/statusand as RSS or. RssAnon,RssFileandRssShmemare the three parts of a process's resident set in/proc/PID/status: anonymous pages with no file, pages of mapped files, and pages of shared memory objects.- The proportional set size (PSS) of a process is its resident memory in which each page shared with other processes is counted as a fraction, the page size divided by.
/proc/PID/smapsis a file that lists every mapping of a process with its size, resident size, proportional size, dirty and clean pages, and the pages in huge pages, andpmap -xand/proc/PID/smaps_rollupshow the same data in compact forms.- A heap profiler is a tool that records every allocation of a running program with the call stack that made it, and reports which places hold the most memory at the peak, such as Valgrind's Massif.