CSE Roadmap

Specialist tier · about 4 hours · 8 parts

Part 2 of 8 · 1 min read

Scope and numbers

Storage is the star.

Think of a very large post office. Letters arrive from everywhere, are sorted into each person's box, and may carry parcels. People return again and again to read the newest letters, and almost never the old ones.

Questions and the answers assumed here.

  1. Scale? 1 billion users.
  2. Features? Send and receive mail, fetch all mail in a folder, mark read or unread, search by subject, sender or body, attachments, and anti-spam and anti-virus checks.
  3. Sign-in? Not covered.
  4. Protocols? Web access over HTTP, plus the old mail protocols for outside clients.
  5. Attachment limit? Up to 25 MB.

Non-functional requirements. High availability, with data copied over several nodes. Scalability that does not slow as the numbers grow. Flexibility, since the old protocols cannot support newer features.

Numbers (from the book).

  • Sending: each person sends 10 emails a day. 1,000,000,000 × 10 / 86,400 is about 115,000 per second.
  • Receiving: each person gets 40 emails a day. Metadata (everything except attachments) averages 50 KB per email.
  • Metadata for one year: 1,000,000,000 × 40 × 365 × 50 KB = 730 PB (petabytes).
  • Attachments: 20 per cent of emails have one, at 500 KB on average. For one year: 1,000,000,000 × 40 × 365 × 20% × 500 KB = 1,460 PB.

The finding: email is a storage-heavy system. Petabytes arrive each day, so storage, not request rate, shapes the design.

Metadata

Metadata is data about a message, such as sender, recipients, subject, body and time, apart from its attachments.

Example: the subject and the date of a message are metadata. The attached photo is not.

Check yourself: 1. How is the sending rate worked out? 2. Which is larger, one year of metadata or of attachments? 3. Why does storage shape this design?