CSE Roadmap

Specialist tier · about 2 hours · 6 parts

Part 2 of 6 · 2 min read

Distributed filesystems

Think of a distributed filesystem like a library whose books are spread across many buildings instead of stored in one. A catalog tells a visitor which building holds which book, and each building keeps spare copies in case one burns down.

In plain words: A distributed filesystem spreads files across many ordinary machines in a datacenter rather than one large storage appliance. Each machine runs a process that serves the file blocks it holds over the network, and a central tracking service records which blocks live on which machine. File blocks are replicated across several machines, so a single machine or disk failure does not lose data. Because the data is already spread across many machines, a batch job can run its work on the same machine that holds the data it needs, instead of copying that data across the network first.

A worked example. The bookstore's year of web-server logs is too large for one disk to serve quickly. Stored on a distributed filesystem, the logs are broken into blocks spread across fifty machines, each block replicated on three machines. A batch job counting page views runs its work on whichever of those fifty machines already hold the relevant blocks, rather than pulling every block across the network to one machine first.

The key lines. No new SQL; a distributed filesystem is not queried with SQL. Below is the shape of a command listing blocks on such a system.

hdfs dfs -ls /logs/2026/
# Found 50 items, each block replicated 3x across the cluster

Distributed filesystem, computation near the data

A distributed filesystem stores files as blocks spread and replicated across many machines, with a tracking service recording block locations, so the combined space of many machines' disks behaves as one filesystem. Computation near the data means scheduling a processing task on a machine that already holds the data it needs, so the data does not have to move across the network first.

Example: the bookstore's log-counting job above runs on the fifty machines that already hold the log blocks, rather than copying all the logs to one machine.