Part 6 of 7 · 5 min read
Diagnosis
Diagnosing a busy machine
In plain words: A mechanic checks the easy readings in a fixed order before opening the engine. The same works for a slow machine, and each step uses a reading from this level.
- Is the processor the bottleneck? In
vmstat 1, theusandsycolumns near 100 andidnear 0 say yes. A highwa(waiting for I/O) says the disk is the cause instead. - How deep is the backlog? The
rcolumn is the run-queue length: the number of tasks that are ready to run or running. A run queue much longer than the number of processors means that tasks wait for a processor, and each waits aboutr ÷ processorstimes its slice (page 3). - Who is using it?
toporps -eo pid,ni,pcpu,comm --sort=-pcpulists the largest users. Look at their nice values and classes. - Is it fair, or is it priority? Compare each task's
sum_exec_runtimeandvruntime, and look for real-time tasks (CLSofFForRR). - Are tasks moving or pinned?
se.nr_migrationsandtaskset -p <pid>show balancing and affinity (page 6). - Choose the lightest remedy. The case below shows the options in order of increasing force.
The case: a 2-processor machine runs a latency-sensitive service next to four batch jobs that compute without stopping. vmstat showed r = 4, id = 0. The service's sleeps ended 1.8 to 2.0 ms late on average, and up to 8.5 ms late at worst. The remedies and their measured results:
| Remedy | Effect on the service |
|---|---|
| Raise the batch jobs' nice value to 10 | Share of the batch jobs falls to 9.7 per cent each against a nice 0 task (page 5), but the wake-up latency stayed at 2.0 ms |
Run the batch jobs under SCHED_IDLE |
Average wake-up latency fell to 0.07 to 0.21 ms |
Pin the service to its own processor with taskset |
Not measured here, but removes the competition entirely (at the price of one processor for batch work) |
| Reduce the number of batch jobs to the number of processors | Removes the backlog, r falls to 2 |
The lesson: first find which of the ideas of this level explains the symptom (backlog, share, priority, affinity, latency), and then use the mildest tool that changes it. Measuring before and after is part of the fix.
Run-queue length
The run-queue length is the number of tasks that are ready to run or running at a given moment, shown as the r column of vmstat, and it exceeds the number of processors when tasks have to wait for one.
Example: r = 4 on a machine with 2 processors means that two tasks are running and two are waiting.
Common mistake
Fixing latency with nice alone. A nice value changes shares, not necessarily how soon a sleeper runs (the test above). For background work, SCHED_IDLE or SCHED_BATCH is the better first tool, and for strict timing, a real-time class used with care.
Interview angle
Answer in your head first, then open it.
How do you find out why a Linux server is slow?
Check vmstat for us, sy, wa and r, then top for the large users, then the class and nice value of each.
What does the load average mean?
A damped average over 1, 5 and 15 minutes of runnable tasks and tasks in uninterruptible sleep.
What are the Linux scheduling policies?
SCHED_OTHER, SCHED_BATCH, SCHED_IDLE, SCHED_FIFO, SCHED_RR and SCHED_DEADLINE.
Can a real-time task freeze the machine?
Not completely: real-time throttling leaves normal tasks about 5 per cent.
What is the difference between SCHED_FIFO and SCHED_RR?
Both use fixed priorities, and SCHED_RR also rotates among equal priorities with a time slice.
Check yourself:
1. What does r = 6 on a 2-processor machine tell you?
2. Name the three first readings of the method.
3. Which remedy gave the best latency in the case?
Words you will meet
| Word | Meaning |
|---|---|
| Load average | A damped average of runnable tasks over 1, 5 and 15 minutes |
| Scheduling policy | The rule that decides how the kernel treats a task |
SCHED_OTHER |
The default policy, with shares by weight |
SCHED_BATCH |
Normal shares, meant for non-interactive jobs |
SCHED_IDLE |
Runs only when nothing else wants the processor |
SCHED_FIFO and SCHED_RR |
Fixed real-time priorities, without and with a time slice |
SCHED_DEADLINE |
Earliest deadline first with stated run time, deadline and period |
| Real-time throttling | A limit on the processor share of real-time tasks |
| Wake-up latency | Time from becoming ready to first running |
| Run-queue length | Number of tasks ready or running |
Quick notes
- The load average is an exponentially damped average, over 1, 5 and 15 minutes, of the number of tasks that are runnable or are waiting in uninterruptible sleep (usually for disk I/O), reported by
uptimeand in/proc/loadavg. - A scheduling policy on Linux is the rule that decides how the kernel treats a task: the real-time policies
SCHED_FIFOandSCHED_RRuse fixed priorities from 1 to 99,SCHED_DEADLINE. - Real-time throttling is a Linux limit on the share of processor time that real-time tasks may use in each period, by default 950,000 of every 1,000,000 microseconds, so that normal.
- Wake-up latency is the time from the moment an event makes a sleeping task runnable to the moment that task starts to run on a processor.
- The run-queue length is the number of tasks that are ready to run or running at a given moment, shown as the
rcolumn ofvmstat, and it exceeds the number of processors when tasks have to wait for one.