!This gives away every stage. For a clean attempt, practise it first, then come back.

Before we start

Every search engine starts with a crawler. Google, Bing and every smaller one send programs around the web, downloading pages and following links, so that when you search, they already have a copy of the page.

The basic idea fits on a napkin: keep a list of URLs to visit, take one, download it, add the links you find to the list, repeat. A student can write that in an afternoon. The interview is about what goes wrong at scale. You must not overload anyone’s website. You must not download the same page a thousand times. And you must not get trapped by sites that generate links forever.

Same order as always: requirements, quick maths, data and API, a design that grows, then deep dives.

Minutes 0–5Requirements

Functional requirements

1. Start from seed URLs, download pages, extract their links, and keep crawling.
2. Save each page’s HTML for the indexing team.
3. Follow each site’s robots.txt and never overload a site.
4. Re-crawl pages so our copies stay fresh.
5. Don’t fetch the same URL twice, and flag duplicate content.

Out of scope: building the search index (another team’s job), and running JavaScript (a separate, much more expensive kind of crawler).

Non-functional requirements

1. Scale: about a billion pages a month.
2. Polite: at most about one request a second to any one site, unless its rules say otherwise.
3. Robust: slow servers, broken HTML and endless link traps mustn’t stall the crawl.
4. Efficient: don’t waste downloads on duplicates or pages that haven’t changed.
5. Easy to extend to new content types later.

Minutes 5–10Back-of-the-envelope estimates

That last line is the key insight. Raw speed is easy. What limits us is spreading work across enough different sites to stay polite.

Minutes 10–15Core entities and the API

Core entities

• URL to visit: the address, its priority, and which website it belongs to. The list of these is called the frontier.
• Page: a URL we’ve fetched, when, the result, a fingerprint of its content, and where the HTML is stored.
• Website (host): its robots.txt rules, how long to wait between requests, and when we last visited.

The API

A crawler has no public users, so its “API” is how its parts talk to each other:

Minutes 15–25The design, built up

Stage 1One machine, one queue

This works for a few thousand pages, and it’s exactly what most people write first. But one machine can’t download a billion pages a month. And there’s a worse problem: it fetches as fast as it can, so if a page links to 500 pages on one small site, it fetches all 500 back to back and slows that site down for its real visitors.

Stage 2Many fetchers, a shared frontier, pages in object storage

That’s fast. But we’ve made the politeness problem worse: now many machines can hit the same site at once.

Stage 3Politeness and no repeats: per-site queues, robots, a seen-URL check

Here’s the difference per-site queues make:

The hashing trick is the same one from lesson 5.2: the same key always lands on the same machine.

Stage 4Staying fresh: re-crawl by how often pages change

Checking the design against the requirements

Deep diveStaying polite

Bad solutionOne shared queue

URLs are taken in the order they were found. When one page links to 500 pages on the same site, all your fetchers hit that site at once. It slows down, and its owner blocks you.

Good solutionOne queue per site, with a delay

The frontier keeps a queue per website and a “not before” time for each. After handing out one URL from a site, it moves that site’s time forward. Fetchers pick from sites that are ready, so work is spread out naturally.

Great solutionPer-site queues, and each site owned by one fetcher

Hash each site’s name to pick a fetcher, so one machine handles all requests to a given site. Politeness is then a local decision on that machine, with no shared lock or central scheduler. Add a priority layer in front, so important sites are visited first, and you have the classic design from the Mercator crawler, which much of the field still builds on.

Also read each site’s robots.txt, cache it, and obey it. It’s an official standard now (RFC 9309). Some sites also ask for a specific delay between requests; respect that too.

Deep diveNot fetching the same URL twice

Bad solutionLook it up in a database for every link

A billion pages produce tens of billions of links. A database lookup for each one is far too slow, and the database becomes the bottleneck of the whole crawler.

Good solutionClean up URLs, then check a set in memory

First, normalise each URL so that different spellings match: lowercase the domain, remove #section parts, sort the query parameters, and drop tracking parameters like utm_source. Then check a set of seen URLs held in memory. Fast, but storing 10 billion full URLs takes hundreds of gigabytes.

Great solutionClean up URLs, then check a Bloom filter

A Bloom filter is a compact way to answer “have I seen this?” using a large array of bits. It never says “no” when the answer is “yes”, but now and then it says “yes” when the answer is really “no”. With 1% of those mistakes allowed, 10 billion URLs need only about 12 GB. For a crawler, occasionally skipping a page we’ve never seen is an acceptable price. (Lesson 3.2 shows databases using Bloom filters the same way, to skip files that can’t contain a key.)

Deep diveDuplicate content

Deep diveCrawler traps

Deep diveFreshness without re-downloading everything

Bad solutionRe-crawl everything on the same schedule

Visiting every page once a week means news front pages are days out of date, while pages that never change are downloaded over and over for nothing.

Good solutionRe-crawl based on how often a page changes

Compare the content fingerprint with the last visit. If it changed, come back sooner; if not, wait longer. News pages end up checked every few minutes, and old pages every few weeks.

Great solutionChange rate, plus “only send it if it changed”

When re-fetching, send the If-Modified-Since or If-None-Match header from the last visit. If nothing changed, the site replies 304 Not Modified with no page at all, saving bandwidth for both of you. And read sites’ sitemaps, which often list every page with the date it last changed.

Deep diveSlow and broken sites

The complete architecture

Here’s everything together: the full design from the last stage, with every part we added along the way. If you only remember one picture from this question, make it this one.

How one request travels through it:

1. Seed URLs go into the frontier, the list of URLs still to visit. It keeps one queue per website and remembers when each site may be visited next.
2. A fetcher asks for its next URL. It only gets one from a site that’s ready, so no site is hit too often.
3. The fetcher looks up the site’s address and robots.txt rules from a cache, then downloads the page with a short timeout.
4. It saves the HTML to object storage and records the fetch time and a fingerprint of the content.
5. A parser reads the page, pulls out the links, and checks each against the seen-URL filter. Only new ones go back to the frontier.
6. A scheduler puts pages back into the frontier when they’re due for a re-crawl, sooner for pages that change often.

How this answer gets scored

Interviewers don’t tick boxes, but their notes on this question tend to land on the same few moments. Here’s what each one sounds like from the other side of the table.

What the left column has in common: every choice points back to a requirement or a number from earlier. If you can say where each box came from, the interviewer will notice.