CSE Roadmap

Core tier · about 5 hours · 8 parts

Part 4 of 8 · 2 min read

Block servers

Send only what changed.

For large files that change often, re-sending the whole file each time wastes bandwidth. Two ideas cut the traffic.

  • Delta sync. Only the blocks that changed are sent, not the whole file.
  • Compression. Each block is compressed, with a method suited to its file type. Text compresses with gzip or bzip2, and images and video need other methods.

A block is a piece of a file, with a unique hash (a fixed-length fingerprint of its content). The book uses 4 MB as the upper block size, following Dropbox. To rebuild a file, join its blocks in order.

Block servers do this heavy work. For a new file they: split it into blocks; compress each block; encrypt each block; and upload the blocks to cloud storage. For an edited file, they send only the changed blocks.

A worked example (made up). A 1 GB file is 1,024 MB / 4 MB = 256 blocks. A user edits the middle, and two blocks change. Delta sync sends 2 × 4 = 8 MB, not 1,024 MB. That is under 1 per cent of the file.

Consistency. The system needs strong consistency by default. Two devices must not show different versions of a file as the truth. Caches default to eventual consistency, so two steps are needed: keep cache replicas in step with the primary, and clear the cache entry when the database is written. For the database, the book chooses a relational database, because it has ACID transactions (Level 5) built in. A NoSQL store would need that logic written by hand.

Delta sync

Delta sync transfers only the parts of a file that changed since the last sync.

Example: editing one slide in a 200-block presentation uploads the one or two blocks that hold the slide.

Check yourself: 1. What four things does a block server do to a new file? 2. How many blocks does a 100 MB file have at 4 MB? 3. Why is a relational database chosen for metadata?