CSE Roadmap

Core tier · about 3 hours · 9 parts

Part 5 of 9 · 2 min read

Storage and bandwidth

How much data is kept and how much data moves, with the same round-number method.

Storage is how much data the system must keep. Bandwidth is how much data moves per second.

Think of a warehouse and a loading dock. The warehouse size is the storage. The number of trucks that can pass the dock each hour is the bandwidth. A big warehouse with a tiny dock still has a problem.

A worked example, using the learning site again. These numbers are made up.

Storage for progress records.

  1. Each saved record holds a user id, a module id and a time. Assume 200 bytes.
  2. 1,500,000 saves a day × 200 bytes = 300,000,000 bytes, which is 300 MB a day.
  3. Over a year: 300 MB × 365 is about 110,000 MB, which is 110 GB.
  4. 110 GB fits on one ordinary disk. Storage is not a problem for this part of the design.

Bandwidth for pages.

  1. Assume a page, with its images, is 500 KB.
  2. 2,400,000 page views × 500 KB = 1,200,000,000 KB, which is 1,200 GB, or 1.2 TB a day.
  3. Per second: 1.2 TB ÷ 86,400, about 14 MB a second. In network terms, 14 MB × 8 bits is about 110 megabits per second.
  4. That is a lot for a single small server to send. It is a good reason to put images on a content delivery network (Level 1 explains it).

So the same site has an easy storage problem and a harder bandwidth problem. Without numbers, a designer might have worried about the wrong one.

Bandwidth

Bandwidth is the amount of data that can pass through a connection each second, often measured in megabits or gigabits per second. A byte is 8 bits.

Example: a link that carries 100 megabits each second can send about 12.5 megabytes each second.

Over longer periods, add a retention rule. If data is kept for five years, multiply the yearly storage by five. Then add copies. If every record is stored three times for safety, multiply by three.

Check yourself:

  • A video site stores 2 million new videos a month, each 50 MB. How much is that per month in TB?
  • Why does the "kept for five years" rule matter?