CSE Roadmap

Core tier · about 5 hours · 7 parts

Part 3 of 7 · 3 min read

Affinity

Think of it this way. A cook who has just arranged a station in a particular corner of the kitchen works fastest there. Sending the cook to another corner means setting up again. The analogy stops in one place: a cook can carry the knives along, while a task cannot carry the contents of a processor's cache (Level 2, page 6).

In plain words: When a task runs on a processor, its data sits in that processor's caches. Moving it to another processor starts again with empty caches, a cost that appears as a burst of slow memory reads. For this reason, a scheduler prefers to keep a task where it was: soft affinity. It is a preference, not a rule, and the balancer may break it when the imbalance is large. A program can demand a fixed set of processors, hard affinity, with sched_setaffinity or taskset (Level 2, page 6).

Both effects can be seen with the counter se.nr_migrations in /proc/<pid>/sched. Three spinning shell loops ran on the 2-processor machine for 4 seconds, and two spinning loops ran in another test:

2 spinners, 2 CPUs:      migrations 0 and 0       CPU time 3.99 s and 3.99 s
3 spinners, 2 CPUs:      migrations 4, 10, 14     CPU time 2.85, 2.64, 2.53 s
3 spinners, all on CPU 0: migrations 1, 2, 2      CPU time 1.32, 1.33, 1.33 s
  • Two spinners on two processors never moved. Each had a processor to itself, and soft affinity kept it there.
  • Three spinners on two processors moved several times each. Three tasks cannot divide evenly between two processors, so the balancer rotates them, and each gets about two thirds of a processor: the three CPU times add up to 8.0 seconds, all the capacity of 2 processors over 4 seconds.
  • Three spinners pinned to one processor made only one or two moves (the first one onto the allowed processor), and each got a third of it, 1.33 seconds.

On machines with several sockets or memory banks, there is a further effect, NUMA (non-uniform memory access): memory attached to one socket is faster to reach from that socket than from another. A scheduler for such a machine tries to keep a task, and its memory, on the same node. The test machine reports a single node (lscpu showed NUMA node(s): 1), so the effect is absent here, but servers often have two or more.

Soft affinity

Soft affinity is the tendency of a scheduler to keep a task on the processor it last ran on, so that its cached data stays useful, without a guarantee; hard affinity is a requirement set by the program that the task may run only on listed processors.

Example: two spinners on two processors each stayed on one processor (0 migrations), while three spinners moved 4 to 14 times in 4 seconds.

Check yourself: 1. Why do three tasks on two processors migrate, while two do not? 2. What did hard affinity (taskset -c 0) do to the migration counts? 3. Why is a migration costly?