Part 1 of 7 · 1 min read
Why this page comes fifth in Level 10
In this module you will
- State the scope of a crawler and estimate pages per second and storage over five years.
- Name the parts of the crawler and trace one URL through all eleven steps.
- Explain why breadth-first order is used, and why a plain queue is not enough.
- Build a URL frontier with front queues for priority and back queues for politeness.
- Handle duplicates, spider traps, robots.txt, slow hosts and failures.
A crawler is a pipeline of many small stages with queues between them, so it is a good test of how well the earlier ideas combine: queues (Level 6), partitioning (Level 4), caching (Level 3) and failure handling (Level 8). It is marked Common and not Core, since it is asked less often than the first four. The book used is "System Design Interview" by Alex Xu, chapter 9. The numbers marked "made up" only show the idea.