CSE Roadmap

Core tier · about 5 hours · 7 parts

Part 5 of 7 · 2 min read

Victim selection

In plain words: The kernel gives each process a badness score. The main part is the memory the process holds: its resident pages, its swap and its page tables, as a share of the total. A big process scores high. An administrator can shift the score with the file /proc/PID/oom_score_adj, which holds a number from −1000 to +1000. The value is added to the score. The value −1000 makes a process exempt, and +1000 makes it the first choice.

The files are readable in any process. In a test, the score of a process rose from 666 to 752 after it touched 1 GiB, as expected. The starting value is high because the test ran inside a sandbox, and only the change is meaningful. An attempt to write −1000 was refused: lowering the adjustment normally needs privilege (the capability CAP_SYS_RESOURCE), so that an ordinary user cannot protect his own program at the cost of others. Raising it is always allowed.

The practical uses are:

  • Protect the critical process (a database, the system manager) with a low oom_score_adj.
  • Sacrifice the expendable one (a cache, a batch job) with a high one.
  • Run services in control groups with memory limits, so that a kill stays inside the group, as in section 3.

A control group (cgroup) limit also limits the effect of the shortage: the test above killed the program inside its group while the rest of the machine was untouched. This is how containers get their memory limit.

oom_score_adj

oom_score_adj is a number from −1000 to +1000, set per process in /proc/PID/oom_score_adj, that is added to the process's OOM score, so that a high value makes the OOM killer choose it sooner and −1000 makes it exempt.

Example: a batch job started with oom_score_adj 500 is killed before a database that has the default 0.

Interview angle

"What is the OOM killer and how would you protect a critical service from it?" A strong answer says that it ends a process when memory is exhausted, that it scores processes by memory use and the adjustment value, and that a service can be protected with a low oom_score_adj and by running others in control groups with memory limits. It adds that the better fix is to size memory correctly.

Check yourself: 1. What is the biggest part of the OOM score? 2. Why is lowering oom_score_adj restricted? 3. How does a control group limit the damage of an OOM kill?