High-Level Design
Design a Web Crawler · Part 2
Problem context
Design a system that crawls the web: start from a set of seed URLs, fetch each page, extract its links, and keep going — billions of pages deep — so a downstream system (a search index, an archive, a training corpus) has the content.
In scope: fetching pages, extracting and enqueueing links, deduplicating URLs and content, and recrawling for freshness. Out of scope: the search index or archive itself, ranking, and rendering client-side JavaScript (a variant).
What a strong answer sounds like
State the decision, connect it to a requirement, and name the tradeoff. Keep the design focused on the workload in the prompt.
Ready for the high-level design interview?
The AI interviewer asks about this part of Design a Web Crawler. The interviewer guides you through topics one by one.