Part 6 of 7 · 3 min read
Checklist
A checklist and the fix
Think of a pilot's pre-flight list. The items are simple and are done in order, so that nothing is missed when the pilot is tired. The analogy stops in one place: a pilot's list is the same every flight, while a disk problem needs you to choose the next step from the last result.
In plain words: The steps that follow from the pages of this level are these. First, read df -h and df -i for the file system in question, and note which one is at 100%. Second, find the largest visible directories with du -x (the -x flag stays on one file system, page 6). Third, if du finds much less than df says is used, look for deleted open files in /proc/*/fd. Fourth, check for a mount hiding data: unmount a suspect mount point, or bind-mount the root file system somewhere else and look at the directory underneath. Fifth, compare apparent size and real use for large files. Sixth, read tune2fs -l for reserved blocks.
A worked example (made up to show the idea). A server's /var is 100% used. du -x /var finds 38 GiB, and df says 92 GiB used. The 54 GiB gap suggests a deleted open file. The command ls -l /proc/*/fd | grep deleted shows one logging process with a descriptor to access.log (deleted). The log rotation tool deleted the log without telling the service. Sending the service its reload signal closes and reopens the log, and df falls to 38 GiB. At 2 MB/s of writing into 20 GiB of free space, the disk would have been full in 20 x 1,024 / 2 = 10,240 s, which is 2.8 hours.
The key lines. The fix is to repair the cause, not only the symptom. A rotation setup that signals the program, a quota on the log directory, an alert at 80% use, and a smaller reserve on a data disk each prevent a repeat.
Full-disk checklist
The full-disk checklist is the ordered set of checks for a file system that reports no free space: free blocks and free inodes, the largest visible directories, deleted open files, hidden mounts, sparse files and reserved blocks. Each check separates one cause from the others.
Example: df -h, then df -i, then du -x, then ls -l /proc/*/fd | grep deleted.
Check yourself:
1. In which order would you check inodes and blocks?
2. What does the -x flag of du do?
3. What is the lasting fix for the deleted-log problem?
Words you will meet
| Word | Meaning |
|---|---|
| Apparent size | The length of a file as stat reports it. |
| Disk usage | The blocks that are allocated for a file. |
| Hole | An unwritten range of a sparse file. |
| Reserved blocks | Blocks that only the superuser may use. |
(deleted) |
Marker in /proc/PID/fd for a file whose name was removed. |
Quick notes
dfasks the file system for its counts.duadds up the blocks of the files that it finds.- A deleted file that is still open keeps its blocks, and
ducannot see it./proc/PID/fdshows(deleted). - A sparse file has holes with no blocks.
ls -lshows the length, anddushows the real use. - ext4 reserves 5% of the blocks for the superuser by default.
tune2fs -mchanges it. - Inodes are fixed in number.
df -ishows them. - Check in order:
df,df -i,du -x, deleted open files, hidden mounts, sparse files, reserved blocks.