Part 4 of 7 · 2 min read
The OOM killer
In plain words: Suppose the programs do touch more than the machine has. File pages in the cache are dropped first, and with swap, idle anonymous pages are written out. If these run out and a page fault still needs a frame, the kernel has two choices: stop the faulting process for ever, or free memory by ending another process. Linux does the second. Its out-of-memory (OOM) killer picks a process and sends it SIGKILL.
The test was safe to run, since a control group (cgroup) limited it to 200 MiB. The program allocates 10 MiB at a time and touches every page:
| Event | Result |
|---|---|
| Limit set | memory.limit_in_bytes = 209,715,200 |
| Program output | the last line was touched 190 MiB |
| Exit status in the shell | 137 |
memory.oom_control |
oom_kill 1 |
The process was ended during its next 10 MiB, and no further message came from the program. The exit status 137 is 128 plus 9, the sign of death by signal 9 (SIGKILL). The process had no chance to run a handler or to free anything. The kernel also writes a report to its log, which dmesg shows (with lines that begin "Out of memory: Killed process").
The kill is wide in effect. The program was not wrong to ask for memory, and any process that holds the pages can be chosen, whether it caused the shortage or not. A web server may be killed because a batch job grew. The kernel does it because the alternative is a machine that makes no progress.
OOM killer
The OOM killer is the part of the Linux kernel that, when memory is exhausted and no frame can be freed, chooses a process and ends it with SIGKILL so that its frames can be reused.
Example: in the test, the program was killed after it touched 190 MiB inside a 200 MiB control group, and the shell saw exit status 137.
Check yourself: 1. What does the kernel try before it kills a process? 2. What does exit status 137 mean? 3. Why can the killed process be an innocent one?