Part 2 of 6 · 2 min read
Questions first
Think of a doctor with a stethoscope, a thermometer and a blood-pressure cuff. The doctor does not use all of them on every patient. A cough leads to the stethoscope, a fever to the thermometer. The analogy stops in one place: a patient can tell the doctor how they feel, and a machine can only be asked through its numbers.
In plain words: A beginner learns commands as a list. A working engineer starts from a question and picks the command that answers it. The table maps common questions to tools and to the level where the idea was explained.
| Question | Command | Reads | Level |
|---|---|---|---|
| Which processes run, and in what state? | ps -eo pid,ppid,stat,comm |
/proc/PID/stat |
1 |
| How many threads has it? | ps -o nlwp -p PID |
/proc/PID/status |
2 |
| How much memory, virtual and real? | ps -o vsz,rss, pmap -x PID |
/proc/PID/smaps |
6, 7 |
| Which files has it open? | ls -l /proc/PID/fd, lsof -p PID |
/proc/PID/fd |
1, 9 |
| What is it waiting for? | ps -o wchan -p PID |
/proc/PID/wchan |
1, 3 |
| Is memory short? | free -m, vmstat |
/proc/meminfo |
6, 7 |
| Is the disk full or slow? | df -h, du -sh, vmstat |
statfs, /proc/diskstats |
8, 9 |
| Is the machine busy? | uptime, vmstat, top |
/proc/loadavg, /proc/stat |
3 |
| What system calls does it make? | strace -p PID |
the kernel's trace | 0 |
| Who may do what? | id, ls -l, namei -l, getcap |
the file system, /proc |
10 |
A worked example. A user reports "the program is slow". The questions come in an order: is the machine busy (uptime), is memory short (free -m), is the disk full or busy (df, vmstat), then which process is the cause (ps, top), then what that process is waiting for (wchan, strace). Each answer removes a branch, which is faster than trying commands at random.
The key lines. Most of these commands are only readers of /proc and /sys (page 2). When one command fails, its source file is still there to read directly.
Observability
Observability is the ability to find out what a running system is doing from outside, using the data that the system exposes: counters, state files, traces and logs, without stopping or changing it.
Example: reading /proc/PID/status shows the state and memory of a process while it runs.
Check yourself:
1. Which command shows the open files of a process?
2. Which file does uptime read its load numbers from?
3. Why ask the questions in a fixed order?