Web Crawler System Design Interview: Questions and Answers

Begin with scope and numbers before components. Ask which pages, how many per day, how fresh, whether scripts matter and what the downstream consumer needs. Then give the simplest working design, a breadth-first queue with a visited set, and say plainly that it will fail at crashes, politeness, URL duplicates and traps. Each later component is a response to one of those named failures, which is exactly how an interviewer wants to see the reasoning unfold.

Downloading one page is trivial. The difficult decision is which of billions of known URLs to fetch next, from which publisher, at what pace and when to come back, while respecting rules. Per-origin budgets, priorities, leases and recrawl intervals are all scheduling state, so the quality of the system is determined by that scheduler and its data model, not by the HTTP client.

10,000,000 ÷ 86,400 = 115.74 successes per second on average. Not every attempt succeeds, so at 80% success there are 144.68 attempts per second. If traffic is bursty at 8× the average, that is 925.93 successes and 1,157.41 attempts per second, plus extra requests for redirects and robots files. Using attempts rather than successes sizes workers and politeness budgets correctly.

At 120,000 compressed bytes per page, 10 M pages is 1.2 TB per day and 111.11 MB per second at burst. Extracted text at 24,000 bytes is 240 GB per day. With five days of raw bodies (about 6 TB) and thirty days of documents (about 7.2 TB) before replication, storage is dominated by retention policy, not by instantaneous rate. I would say these are assumptions and show how the figures scale if page size changes.

Little's Law: concurrency equals arrival rate times mean time per request. 144.68 attempts per second at 0.8 s gives about 116 in flight. That sizes connection pools and worker counts. It also shows why latency matters: if fetches slow to 2 s the same rate needs about 289 in flight.

Strings differ for the same page (HTTP:// versus http://, default ports, fragments, tracking parameters, dot segments), so the set misses duplicates, and in a process-local set two threads can both decide a URL is new and it disappears at restart. The fix is a normalizer plus a durable unique key where first-seen is decided by an atomic insert. Normalization must be conservative: lowercasing paths or dropping unknown query parameters can merge distinct pages.

• Which normalizations are unsafe? — Lowercasing paths, dropping all query parameters and stripping trailing slashes can merge different pages. Merge only provable equivalences and keep the raw URL.

No detector is perfect, so bound the damage. Combine allowed hosts, depth limits, a per-origin URL budget, caps on distinct values for a repeating query key, pagination limits and soft-404 detection. Record the reason code for every rejection so an operator can see that a calendar was cut at thirty days and decide whether that was too strict. The goal is a bounded cost, not perfect recognition.

Fetch it before any page of an origin, choose the group matching our user-agent, and apply the longest matching rule, with Allow winning ties. A 2xx is cached; a 4xx means the file is unavailable and crawling proceeds normally; 5xx or a timeout means unreachable, so pause that origin, using a recent good copy for a bounded time. Crawl-delay is not part of the standard, so I use it as one input to the rate rather than as authority. Robots is a publisher request, not an access control or authorization mechanism.

• What if robots changes while URLs are queued? — Each verdict records the policy version; the claim compares versions and rechecks before connecting.

With a shared per-origin bucket holding the next eligible time, in-flight cap and backoff state. A worker cannot fetch for an origin without claiming from the bucket, and claiming moves the clock in the same transaction. Per-worker sleeps cannot do this because their schedules coincide. A 429 with Retry-After sets backoff on the whole origin, not on one URL.

I group by origin (scheme, host, port) by default, because that is the unit a server is addressed by, and add stricter groups when a publisher asks, such as the registrable domain for a publisher with many subdomains. Grouping by IP is dangerous with CDNs and shared hosting. The right answer depends on what the publisher's infrastructure can bear, which is learned from feedback.

The frontier needs lookups a broker is not built for: best eligible URL per origin, uniqueness, attempt counts and next-due times. Those are indexed queries. A broker is still useful for outbound events where fan-out and at-least-once delivery are the need. I would keep the authoritative frontier in a table and use the broker only for index events.

• What would make you move the frontier off a relational database? — Sustained claim rates beyond what one shard per origin partition can serve, or storage growth that makes sweeps slow. The partition key stays the origin, so the move is to a partitioned store, not a redesign of the claim.

A worker claims a URL in a short transaction that sets state LEASED, an owner, an expiry and an incremented fence token. If it crashes, a sweeper returns the row to eligible after the lease expires, keeping the attempt count. The consequence is honest: the page may be fetched twice, so the design guarantees at-least-once attempts and makes the stored result idempotent.

• Can you reach exactly-once? — Not for network I/O. A response may have been produced and lost. Make the commit idempotent instead.
• How long should a lease be? — Slightly above the 95th-percentile fetch time, with one extension allowed while the fence is current. Too short produces duplicate fetches from healthy slow workers, too long delays recovery from crashes.

A slow worker may outlive its lease; by then another worker holds the URL with a higher token. If the slow one commits with its old token, the store rejects it, because the row now carries the higher fence. Time-based leases alone cannot prove who owns the work, since pauses and clocks lie; a monotonically increasing token can.

Choose the origin first, then its best URL. Keep origins in a heap keyed by next eligible time, add aging so waiting raises priority, and apply per-class quotas so no publisher takes more than a fixed share while others have eligible work. A flat priority list lets one large publisher with millions of high-priority URLs absorb the fleet.

Honor Retry-After first. Without it use exponential backoff with jitter up to a cap, applied to the origin on 429 and repeated 503, not just the URL. Cap attempts and move exhausted URLs to a terminal state with the last error. Treat 404 as normally final and 5xx and timeouts as retryable. Jitter prevents thousands of retries from landing in the same second.

Treat the target as a new candidate: normalize, check scope, evaluate the target origin's robots rules, take a bucket slot and validate the destination address. Bound hops (five) and stop on a repeated URL. Store both the original and final URL. Letting the HTTP library follow redirects silently skips every policy check.

• What if a redirect crosses to another origin that disallows the path? — The target is evaluated as a new candidate with its own robots rules and bucket, so the fetch is blocked there, and the original URL records the redirect and the block reason.

A public page links to a name that resolves to an internal address, and the crawler is tricked into reading internal data. Defend at connect time: only http and https; resolve the name; reject private, loopback, link-local, multicast and metadata ranges for IPv4 and IPv6; connect to the validated address and pin it so DNS rebinding cannot change it; repeat per redirect. Back it with network-layer deny rules, because a library bug should not be the only barrier.

• Why not check the hostname once? — A name can resolve to different addresses over time, so the check must apply to the address actually connected to.

Apply limits on streamed, decompressed bytes rather than on the declared length, abort the read at a cap such as 5 MB, check the content type against an allowlist and sniff the first bytes, and run parsers with time and memory limits. Suspicious responses go to a quarantined state with a reason and never reach the parser.

• What about a page that is huge but valid? — The size cap is a policy, not a bug detector: over-cap pages are stored as truncated or skipped with oversize, and an operator can raise the cap for a trusted origin.

A single 'success' hides where problems occur. If indexed pages drop, separate stage counts show whether the network, parser, storage or downstream pipeline is failing, and each stage can be retried independently. Quarantine and reason codes make the failure explainable.

URL dedupe asks whether the address is already known, before any network cost. Content dedupe asks whether fetched text is identical and stores it once, keeping every URL as provenance. Near-duplicate detection estimates high similarity to cluster documents and can merge distinct versions, so clusters are labels, thresholds are recorded and members are kept.

It removes lookups for new URLs at very large scale. Its 'definitely not seen' answer lets the URL be inserted without a database check; its 'maybe seen' answer must be confirmed in the exact store before discarding anything. For 1 billion URLs at 1 percent false positives the filter needs about 9.6 billion bits, roughly 1.2 GB. At 27 million discoveries per day I would not add it.

• What happens if the filter is lost in a restart? — Nothing incorrect: it only avoids lookups. Rebuild it from the exact store, and until it is rebuilt every URL goes to the exact check, which is slower but safe.

Use conditional requests with ETag/Last-Modified; a 304 saves bytes, parsing and index churn but still spends an origin slot. Treat validators as hints: a 200 with a new ETag is hashed, and the page counts as changed only if the content hash differs. Schedule by estimated change interval and importance, clamped and jittered, with a fixed share of budget reserved for new-URL discovery.

• What if a page changes but the ETag does not? — A server that keeps the same validator after a content change defeats conditional requests. A periodic unconditional fetch of a small sample per origin detects this, and the origin is marked validator-unreliable.

Measure first. Compare raw extraction with rendered extraction on a sample per page class and render only where raw HTML misses most of the content. The render lane is sandboxed, has separate queues and limits, falls back to HTML-only when saturated, and sends every sub-request through the same egress and politeness checks. Rendering all pages would cost about twelve times the compute in my example.

Write the body to object storage under a content-addressed key (safe to repeat); in one database transaction write the frontier update, document metadata and an outbox row; a relay publishes outbox rows at-least-once and the consumer dedupes by event id. Never publish before the commit: a crash would leave the index pointing at nothing.

Workers stop taking new leases, a bounded spool holds completed results and low-priority work is shed. If arrivals are 200 per second and completions fall to 50, backlog is (200−50)×7,200 = 1,080,000. Recovering at 250 per second against 200 inflow drains in 1,080,000 ÷ 50 = 21,600 s, six hours, but only if per-origin budgets permit; if the backlog sits in a few capped origins, those caps dominate. Retries after recovery must be jittered to avoid a reattempt storm.

• Why not just let workers keep fetching and queue results in memory? — A bounded spool is fine, but unbounded in-memory results lose work on a crash and fetching without lease state cannot honor origin budgets. Stopping new leases protects publishers; the spool protects finished work.

Each origin's bucket has one owning region at a time through an ownership lease with an increasing token. Very large origins can be split into preleased shares that sum to the cap. During a partition the old owner may keep working until its lease expires, so overshoot is bounded by lease length. The design offers fenced ownership with bounded overshoot, not an exact global counter.

• Why not one global bucket? — Every claim becomes a cross-region round trip and an unreachable region cannot crawl at all.

Request rate against each origin's limit, 429 and 5xx shares, oldest eligible URL age, blocked-by-robots and scope shares, parse yield, 304 rate, changed-page yield, dedupe ratio and freshness gap by priority class, plus stage counts for fetched, parsed, stored and index-ready. Reason codes on every state change make a complaint answerable from data.

• Which metric proves politeness rather than only throughput? — Per-origin request rate against its cap and the 429/503 share, plus the minimum gap between request starts. Throughput can rise while politeness falls, so they are tracked separately.

Verify it, record a suppression that immediately blocks future fetches, then delete or tombstone frontier rows, parsed documents, raw bodies and index entries per data class, with an audit trail. Keep the suppression record after the data is gone, so the content is not re-crawled later.

Crawl-delay is nonstandard; I honor it as an input but set the rate from my own caps and feedback. I would crawl faster only for an origin that has explicitly agreed, because blocks and complaints reduce long-run access. A kill switch per origin is part of the design.

Exact URL inserts become the bottleneck first, which is when a Bloom prefilter pays off. Frontier shards must split large origins. The render lane and storage retention need budgets. Regions and preleased budgets matter more, and observability must work per origin and per class. I would also revisit retention: 12 TB per day of raw bodies cannot be kept for five days cheaply.

A queue of URLs, a set of normalized URLs already seen, a loop that pops a URL, fetches it with a timeout, skips non-200 responses, parses the HTML, resolves each link against the page address, keeps only http and https, drops fragments, and appends unseen links. Add three cheap guards even at this size: a per-host delay, a robots check and a page limit. It is correct for a few hundred pages of one site. I then name where it breaks: a crash loses the queue, many workers flood a host, spellings multiply URLs and traps never end. Saying this out loud shows the later components are responses, not decoration.

The frontier is the durable set of discovered URLs with their state, priority and next-eligible time. A FIFO fails three ways. It ignores politeness: four million news-a URLs at the head make every worker wait on news-a's clock while other origins idle (head-of-line blocking). It ignores priority and freshness: an old archive page is fetched before today's front page. And it cannot hold state such as attempts, leases or backoff, so a crash or retry has nowhere to live. The frontier answers 'which origin is due, and which of its URLs matters most', which is an indexed lookup, not a queue pop.

• Could a FIFO per origin work? — Yes, a per-origin queue plus a min-heap of origins by next eligible time is the core of the design. It still needs durable state and priority inside each origin, so it is the bucket structure, not a plain FIFO.

No. It is a voluntary request to cooperating crawlers. A person or a badly behaved bot can ignore it, and the file itself advertises the paths it lists. Anything private must be protected by authentication and authorization, and a crawler must not treat a missing Disallow as permission to read what requires a login. For Atlas the practical rule is: obey robots, crawl only public pages, never log in, and treat a 401 or 403 as a stop.

It is a line some crawlers honor, asking for a number of seconds between requests, but it is not part of RFC 9309, and interpretations differ (seconds between requests versus a fetch window). Atlas reads it as one input to the origin's rate, takes the more conservative of it and its own cap, and never lets it raise the rate above the operator's cap. If a publisher needs a firm limit, an explicit agreement or an access-controlled interface is better than a nonstandard directive.

• What if Crawl-delay says 1 s but the origin returns 429 at that rate? — The observed behavior wins. The origin's backoff and the slower of the two rates apply, and the discrepancy is logged so an operator can contact the publisher.

429 and 503 are retryable: honor Retry-After first, else exponential backoff with jitter up to a cap, applied to the whole origin for 429 and repeated 503. 404 is normally final, but one or two later confirmations avoid dropping a page after a transient deploy error; 410 is final sooner. A DNS failure is retried a few times with backoff because resolvers fail transiently, then becomes a terminal reason after repeated failures across hours. Timeouts retry with backoff. In every case attempts are capped and the final state records the last error.

• Why jitter? — Without it, thousands of URLs that failed together retry together and recreate the overload that caused the failure.

Only what the server implements. An ETag is a validator the server chooses; 304 means the server says the representation matches the validator you sent. Neither guarantees the content is unchanged in any sense you care about: some servers change the ETag per request, some ignore conditional headers, and Last-Modified can be a deploy time instead of a content time. So Atlas uses them to save bandwidth and parsing, confirms a 200 by content hash, and stops trusting an origin whose validators are unreliable. A 304 still costs a request and a politeness slot.

State that decides what happens next or that cannot be rebuilt cheaply: URL identity and state, priority and next-fetch time, attempt count, lease owner and fence, per-origin bucket (next eligible time, in-flight, backoff, owner token), robots cache with policy version, validators and body hash, document metadata and provenance, the outbox, and suppression records. Raw bodies go to object storage with a retention period. In-memory-only state is limited to caches that can be rebuilt. The test for each item is: if the process restarts, does losing it cause a flood, a loss of work or a forbidden fetch?

The minimal design that is still honest: a conservative normalizer, a per-origin bucket in the same database as the frontier, robots rules with correct failure semantics, leased claims with a fence, safe egress, content hashing, conditional recrawl and an outbox. At ten million pages a day that fits one sharded relational database and a stateless worker pool. I would not build a Bloom prefilter (exact inserts are about 312 per second), a distributed graph of host leases across regions, near-duplicate removal, or a render lane for everything. I would add each only when a named bottleneck appears: lookup load for the filter, latency or residency for regions, storage cost for similarity, measured extraction loss for rendering.