Part 4 of 7 · 2 min read
Sparse files
Think of a long shelf with a label for each of a hundred slots, and only two slots hold anything. The shelf looks full length, and only two places take weight. The analogy stops in one place: a shelf has slots that you can see are empty, while a program reading a hole is given zeros.
In plain words: A sparse file has regions, called holes, that were never written. The file system records no blocks for a hole. A read of a hole returns zero bytes. The apparent size, from stat, is the highest offset. The real use is the number of blocks that were actually allocated. A virtual machine image, a database file, and a file made with truncate are often sparse. Tools that copy files without care may fill the holes and use the full size.
A worked example. On the image I made a 100 MB file with truncate -s 100M and then wrote one byte in the middle.
$ truncate -s 100M mnt/sparse.bin
$ ls -l mnt/sparse.bin size 104857600
$ du -k mnt/sparse.bin 0
$ printf 'x' | dd of=mnt/sparse.bin bs=1 seek=52428800 conv=notrunc
$ du -k mnt/sparse.bin 4
$ filefrag mnt/sparse.bin 1 extent found
At first the file had an apparent size of 100 MiB (104,857,600 bytes) and used 0 KiB. One byte at offset 50 MiB made the file system allocate one 4 KiB block, so the real use became 4 KiB. The file has one extent of one block, and the rest of its 25,600 blocks are holes. df did not change by 100 MiB. A du or ls that adds up the apparent sizes would claim 100 MiB.
The key lines. du reports real use (blocks) by default. du --apparent-size and ls -l report the length. The two answers to "how big is this file" are both right, and one of them is the answer to "how much space does it take".
Sparse file
A sparse file is a file in which some ranges have never been written. The file system stores no blocks for those ranges, called holes, and returns zeros when they are read. The file's apparent size can be much larger than the space it uses.
Example: a file made with truncate -s 100M has a size of 104,857,600 bytes and uses no blocks until data is written.
Check yourself:
1. How much space does a freshly truncated file of 100 MiB use?
2. What does a read of a hole return?
3. Which of ls -l and du shows the real use?