CSE Roadmap

Core tier · about 5 hours · 7 parts

Part 6 of 7 · 5 min read

Diagnosis

Diagnosing a busy machine

In plain words: A mechanic checks the easy readings in a fixed order before opening the engine. The same works for a slow machine, and each step uses a reading from this level.

  1. Is the processor the bottleneck? In vmstat 1, the us and sy columns near 100 and id near 0 say yes. A high wa (waiting for I/O) says the disk is the cause instead.
  2. How deep is the backlog? The r column is the run-queue length: the number of tasks that are ready to run or running. A run queue much longer than the number of processors means that tasks wait for a processor, and each waits about r ÷ processors times its slice (page 3).
  3. Who is using it? top or ps -eo pid,ni,pcpu,comm --sort=-pcpu lists the largest users. Look at their nice values and classes.
  4. Is it fair, or is it priority? Compare each task's sum_exec_runtime and vruntime, and look for real-time tasks (CLS of FF or RR).
  5. Are tasks moving or pinned? se.nr_migrations and taskset -p <pid> show balancing and affinity (page 6).
  6. Choose the lightest remedy. The case below shows the options in order of increasing force.

The case: a 2-processor machine runs a latency-sensitive service next to four batch jobs that compute without stopping. vmstat showed r = 4, id = 0. The service's sleeps ended 1.8 to 2.0 ms late on average, and up to 8.5 ms late at worst. The remedies and their measured results:

Remedy Effect on the service
Raise the batch jobs' nice value to 10 Share of the batch jobs falls to 9.7 per cent each against a nice 0 task (page 5), but the wake-up latency stayed at 2.0 ms
Run the batch jobs under SCHED_IDLE Average wake-up latency fell to 0.07 to 0.21 ms
Pin the service to its own processor with taskset Not measured here, but removes the competition entirely (at the price of one processor for batch work)
Reduce the number of batch jobs to the number of processors Removes the backlog, r falls to 2

The lesson: first find which of the ideas of this level explains the symptom (backlog, share, priority, affinity, latency), and then use the mildest tool that changes it. Measuring before and after is part of the fix.

Run-queue length

The run-queue length is the number of tasks that are ready to run or running at a given moment, shown as the r column of vmstat, and it exceeds the number of processors when tasks have to wait for one.

Example: r = 4 on a machine with 2 processors means that two tasks are running and two are waiting.

Common mistake

Fixing latency with nice alone. A nice value changes shares, not necessarily how soon a sleeper runs (the test above). For background work, SCHED_IDLE or SCHED_BATCH is the better first tool, and for strict timing, a real-time class used with care.

Interview angle

Answer in your head first, then open it.

How do you find out why a Linux server is slow?

Check vmstat for us, sy, wa and r, then top for the large users, then the class and nice value of each.

What does the load average mean?

A damped average over 1, 5 and 15 minutes of runnable tasks and tasks in uninterruptible sleep.

What are the Linux scheduling policies?

SCHED_OTHER, SCHED_BATCH, SCHED_IDLE, SCHED_FIFO, SCHED_RR and SCHED_DEADLINE.

Can a real-time task freeze the machine?

Not completely: real-time throttling leaves normal tasks about 5 per cent.

What is the difference between SCHED_FIFO and SCHED_RR?

Both use fixed priorities, and SCHED_RR also rotates among equal priorities with a time slice.

Check yourself: 1. What does r = 6 on a 2-processor machine tell you? 2. Name the three first readings of the method. 3. Which remedy gave the best latency in the case?

Words you will meet

Word Meaning
Load average A damped average of runnable tasks over 1, 5 and 15 minutes
Scheduling policy The rule that decides how the kernel treats a task
SCHED_OTHER The default policy, with shares by weight
SCHED_BATCH Normal shares, meant for non-interactive jobs
SCHED_IDLE Runs only when nothing else wants the processor
SCHED_FIFO and SCHED_RR Fixed real-time priorities, without and with a time slice
SCHED_DEADLINE Earliest deadline first with stated run time, deadline and period
Real-time throttling A limit on the processor share of real-time tasks
Wake-up latency Time from becoming ready to first running
Run-queue length Number of tasks ready or running

Quick notes

  • The load average is an exponentially damped average, over 1, 5 and 15 minutes, of the number of tasks that are runnable or are waiting in uninterruptible sleep (usually for disk I/O), reported by uptime and in /proc/loadavg.
  • A scheduling policy on Linux is the rule that decides how the kernel treats a task: the real-time policies SCHED_FIFO and SCHED_RR use fixed priorities from 1 to 99, SCHED_DEADLINE.
  • Real-time throttling is a Linux limit on the share of processor time that real-time tasks may use in each period, by default 950,000 of every 1,000,000 microseconds, so that normal.
  • Wake-up latency is the time from the moment an event makes a sleeping task runnable to the moment that task starts to run on a processor.
  • The run-queue length is the number of tasks that are ready to run or running at a given moment, shown as the r column of vmstat, and it exceeds the number of processors when tasks have to wait for one.