Part 5 of 7 · 2 min read
The downloader
Rules from sites, and where the time goes.
robots.txt. Many sites publish a file called the Robots Exclusion Protocol that says which pages crawlers may fetch. A crawler should read it first and follow it. To avoid fetching it again and again, its contents are cached and refreshed now and then.
Ways to make the downloader faster (from the book):
- Distributed crawl. Spread jobs over many servers, each running many threads. Split the URL space so that each downloader owns a subset (for example by hashing the host name, which also keeps one host on one server, which helps politeness).
- Cache the DNS answers. A DNS lookup takes 10 to 200 milliseconds, and many interfaces block the other threads while waiting. Keeping a local table of host name to address, refreshed by a periodic job, removes most of that time.
- Locality. Put crawl servers near the sites they fetch, since downloads are faster.
- Short timeout. Some hosts answer slowly or never. Set a maximum wait, and move on to other pages if the host does not reply in time (see timeouts, Level 8).
A worked example of the DNS gain (made up). At 800 pages per second, each needing a lookup of 100 ms on average, blocking lookups would tie up 80 thread-seconds every second. With a cache that answers 98 per cent of lookups from memory, only 2 per cent, or 16 lookups per second, are slow, which is 1.6 thread-seconds of waiting.
Robustness. To keep working through trouble:
- Consistent hashing to spread load among downloaders, so one can be added or removed without moving most work (Level 4).
- Save crawl state and data to storage, so a crashed crawl can restart from where it was.
- Handle exceptions without bringing the system down.
- Validate data to stop bad input from causing errors later.
Extensibility. Add new content types by plugging in new modules. The book's examples: a downloader for PNG images, and a monitor that watches for copyright or trademark misuse. The pipeline stays the same, with new stages added.
Check yourself: 1. What is robots.txt for? 2. Why is DNS caching a big gain? 3. Why save the crawl state?