Part 2 of 8 · 2 min read
Scope and numbers
What the feed does and how big it is.
Think of a newspaper stand that makes a custom paper for every customer. The editor can print every paper the moment a story arrives. Or the editor can wait, and gather stories when the customer walks up. Both work. They differ in when the effort is spent.
A news feed is the constantly updating list of posts from the people and pages a user follows. Similar questions ask for an Instagram feed or a Twitter timeline.
Questions and the answers assumed here (from the book's sample talk).
- Mobile, web or both? Both.
- Main features? A user publishes a post and sees friends' posts in a feed.
- Order? Newest first, to keep things simple.
- How many friends? Up to 5,000.
- How many users? 10 million daily active users (DAU).
- Content? Text, images and videos.
Requirements.
- R1. A user can publish a post.
- R2. A user can open a feed of friends' posts, newest first.
- R3. Opening the feed is fast, since it happens far more often than posting.
- R4. A new post appears in friends' feeds soon, not necessarily at once.
- R5. Posts are not lost.
Numbers (made up to show the idea). Each of the 10 million users opens the feed 5 times a day. That is 50,000,000 reads a day, or 50,000,000 / 86,400 = about 579 per second on average. Suppose 2 million posts are published a day, and a user has 200 friends on average. Delivering every post to every friend's feed is 2,000,000 × 200 = 400,000,000 writes a day, about 4,630 per second.
The feed cache stores post ids only. Keep the latest 500 ids per user, at 8 bytes each: 10,000,000 × 500 × 8 = 40,000,000,000 bytes, which is 40 GB. That fits in a small cache cluster.
The key point: reads are 579 per second, and fanout writes are 4,630 per second. The write side is the larger one, even though readers outnumber writers. This is why the design question is interesting.
Daily active users (DAU)
Daily active users are the people who use the product at least once in a day. It is the usual base for load estimates.
Example: 10 million DAU opening the feed 5 times each makes 50 million reads a day.
Check yourself: 1. What two flows does a feed have? 2. Why is the write side larger than the read side in the example? 3. Why store ids and not whole posts in the feed cache?