Part 3 of 7 · 2 min read
The parts
One box per job.
The book splits the crawler into components. Each does one job.
| Component | Job |
|---|---|
| Seed URLs | The starting list. For a university, its domain name. For the whole web, pick by country or by topic so crawling spreads widely |
| URL Frontier | Holds the URLs still to download, in order. It is the heart of the design |
| HTML Downloader | Fetches pages over HTTP for URLs from the frontier |
| DNS Resolver | Turns a host name into an IP address for the downloader |
| Content Parser | Checks that a page is well-formed, since broken pages waste storage. It is a separate component so parsing does not slow downloading |
| Content Seen? | Tells whether this exact content is already stored |
| Content Storage | Keeps the pages. Most are on disk, and popular ones are in memory for speed |
| URL Extractor | Pulls links out of the page and turns relative links into full ones |
| URL Filter | Drops unwanted types, error links and blacklisted sites |
| URL Seen? | Tells whether a URL was already visited or is already in the frontier |
| URL Storage | Keeps the URLs already visited |
Content Seen? The book cites research that about 29 per cent of web pages are duplicates. Comparing two pages character by character is slow, so the system compares hash values of the pages instead. A page whose hash is already stored is dropped.
URL Seen? Without it, the crawler would add the same URL many times, load servers more, and might loop forever. A Bloom filter or a hash table is used.
The workflow.
- Seed URLs go into the frontier.
- The downloader takes URLs from the frontier.
- It asks the DNS resolver for addresses and downloads.
- The content parser checks the page.
- The page goes to Content Seen?.
- If the content is already stored, the page is dropped. If not, it goes to the link extractor.
- The extractor pulls out links.
- The links go through the URL filter.
- Filtered links go to URL Seen?.
- If seen before, nothing happens.
- If new, the URL is added to the frontier.
A worked example (made up). The downloader fetches a.com/index. It holds 40 links. The filter removes 5 (images and a blacklisted site). 28 of the remaining 35 are already in URL Seen?. The 7 new ones enter the frontier. Over time, most links on any page are already known, so each page adds only a handful of new work.
Check yourself: 1. Why is the content parser a separate component? 2. What does Content Seen? use instead of comparing whole pages? 3. In the example, how many URLs reach the frontier?