CSE Roadmap

Core tier · about 3 hours · 7 parts

Part 4 of 7 · 3 min read

Memory limit

Think of a guest at a buffet who may take only one tray. The guest fills the tray, goes back for more, and the staff stops the guest at the tray limit. The analogy stops in one place: a buffet guest is asked to stop, and a process over its group limit is killed without any question.

In plain words: The program below allocates memory one mebibyte at a time and writes to every page, since only touched pages count (Level 7). It reports every 10 MiB, up to 100 MiB.

A worked example.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(void) {
    for (int mib = 1; mib <= 100; mib++) {
        char *p = malloc(1 << 20);
        if (!p) { puts("malloc failed"); return 1; }
        memset(p, 1, 1 << 20);                 /* touch every page */
        if (mib % 10 == 0) { fprintf(stderr, "touched %d MiB\n", mib); }
    }
    puts("done: 100 MiB");
    return 0;
}

First run inside the 20 MiB group, then without a group:

touched 10 MiB
Killed                           (exit status 137)
memory.max_usage_in_bytes: 20971520
memory.failcnt: 36

touched 100 MiB
done: 100 MiB

The program reached 10 MiB, and was killed before it printed 20, because the group could hold at most 20 MiB, and part of that was used by the shell and the program's own code. Exit status 137 is 128 + 9: the signal SIGKILL, sent by the group's out-of-memory killer (Level 7). The counter memory.max_usage_in_bytes shows that the group reached exactly the limit, and memory.failcnt shows that the limit was hit 36 times (the kernel first tried to reclaim cache and failed). Without the group, the program used 100 MiB and finished, and the other processes of the machine were never affected by the group's kill.

The key lines. A memory limit protects the neighbours. Its cost is the sudden death of the program that goes over. A good service sets the limit above its normal peak and logs the failure counter.

Group out-of-memory kill

A group out-of-memory kill is the action of the kernel when a memory cgroup reaches its limit and cannot free memory: it ends one process of the group (usually the one using the most) with SIGKILL, and leaves the processes outside the group alone.

Example: a process in a 20 MiB group that grows to 20 MiB is killed, and the exit status is 137.

Common mistake

Reading free inside a container and thinking it shows the container's limit. On many systems it shows the host's memory. The limit is in the cgroup files, and a program that sizes its cache from free may grow past its limit and be killed.

Check yourself: 1. Why does the program write to each page? 2. What does exit status 137 mean? 3. What did memory.failcnt count?