Design a Distributed Web Crawler: From Script to Production

A web crawler discovers public pages, decides whether and when each one may be fetched, downloads it politely and safely, extracts text and links, removes duplicates, stores the result and schedules a later revisit. This page builds that system from a ten-line script to a multi-region crawler, and each new component appears only after the failure that demands it.

The running example is fictional. Atlas Discovery builds a topic-search feed and a change-monitoring service from public article and documentation pages. Its first crawler gathers pages from three invented publishers, news-a.example.test, docs-b.example.test and store-c.example.test, starting from user-provided seed URLs (a seed is an address you start from). Every rate, size and time on this page is a teaching assumption, not a benchmark, and none describes a real site or product.

One page is followed through the whole story: U-104, an article at news-a.example.test/articles/chip-supply. It enters through seed S-01, is normalized, approved by robots rules, scheduled for a host slot, leased to worker W-2, fetched, parsed, deduplicated, stored, mined for links and later re-checked with a conditional request that returns 304. An interactive 44-scene Visual Design Journey, embedded below at the visual-journey anchor, shows the same page moving through each stage, and chapters cite scenes by number.

What a crawler is, and what it is not

The concrete miss is simple. A major article appears on news-a on Tuesday. Atlas's feed does not show it until Friday, because a person has to remember to run the script. The obvious response, running the script from many machines at once, makes the second problem visible: the publisher's small server answers with 429 Too Many Requests, and a well-run site may block the offending address range. A crawler therefore has two jobs that pull against each other: find and refresh pages quickly, and never be a nuisance to the sites it reads.

Instead of a glossary, here is every term this page needs, defined at the moment U-104 meets it. If you arrived from a search result and skipped other lessons, nothing here depends on them.

Two more ideas are used later and are defined where they appear: fencing tokens (chapter 9), a number that lets storage reject a slow worker's stale write, and eventual consistency (chapter 16), the idea that regions agree after a delay rather than instantly.

Requirements come before boxes. For a crawler, the non-functional list carries more of the design than the functional one.

Atlas wants ten million successful page fetches per day. The arithmetic below separates quantities that people routinely confuse: discovered, eligible, attempted, successfully fetched, parsed, stored and index-ready URLs.

Discovery outruns fetching

Each successfully parsed page yields about 18 links, and 15 percent of them survive filtering and the exact already-known check: 10 M × 18 × 0.15 = 27 million accepted link discoveries per day. Separately, partner feeds and sitemaps submit up to 25 million candidate URLs per day. Whether those two inputs overlap is unknown, so show both cases. If the partner candidates are mostly the same pages the crawler discovers anyway, inflow is about 27 million per day and the backlog grows by 17 million per day. If they are independent, inflow is 52 million and the backlog grows by 42 million per day.

In either case ten million fetches cannot drain a sustained input of novel eligible URLs. A priority, admission and retention policy is not optional. It is part of the design, and it is the reason a crawler has a scheduler instead of a plain queue.

Start with the simplest thing that works, because it defines the vocabulary and shows exactly where it breaks. Version 0 is a breadth-first search over links: a queue of URLs to visit, a set of URLs already seen, an HTTP GET, an HTML parser and an output file.

Run against a few hundred pages of news-a, this works. The worker fetches S-01 and receives a 200 with about 120,000 bytes of compressed HTML (scene 5). It resolves each relative link against the page address, drops the fragment after #, keeps only http and https, and enqueues about eighteen new addresses, including U-104 (scene 6). Two small details matter later. First, the crawler stores the status, headers and a body hash separately from the raw bytes, because a few hundred bytes of metadata drive future decisions while 120,000 bytes need their own retention rule. Second, it records where each link was discovered, which becomes provenance in chapter 12.

What this version quietly assumes

• One process, so a crash loses the queue and the seen-set.
• One publisher at a time, so nothing limits the request rate.
• Perfectly clean URLs, so string equality means page equality.
• Friendly servers, so nothing validates where a connection goes or how large a reply is.
• Pages never change, so nothing is ever fetched twice on purpose.

The remaining chapters take these assumptions one at a time, in the order real operators meet them.

Within an hour of crawling a real site the queue contains the same article four times and a calendar that never ends. The visited set compares strings, so every spelling of an address counts as a new page.

The conservative rule is: merge only what is provably the same, keep the discovered spelling for audit, and treat the publisher's rel=canonical link as a hint rather than proof. A page can claim any canonical address; only content comparison (chapter 12) can show that two pages really match.

Traps need budgets, not perfect detectors

A calendar widget links every day to the next, so /calendar?day=1, ?day=2, ?day=3 are genuinely different addresses with no end. Faceted navigation (color × size × price filters) produces combinatorial URL counts. Soft 404 pages answer 200 for missing content, and pagination can run to page nine thousand. No rule catches every trap, so the design caps them: allowed hosts, a depth limit, a per-origin URL budget, a cap on distinct values for a repeating query key and a pagination limit. Rejections are recorded with a reason code so an operator can tell that the calendar was cut at thirty days and judge whether that was too strict.

Scenes 7 to 9 show the queue swelling to 40,000 entries for about 300 real pages, then shrinking to 312 admitted URLs once normalization and the scope guard run.

Adding workers makes the crawler faster and makes it dangerous. Five workers each sleeping five seconds between requests still send five requests at time zero when their sleeps happen to align, and the publisher answers 429 (scene 10).

The fix is to make the origin a first-class object. In this design an origin is scheme://host:port. That choice is deliberate: a host name, an origin, a registrable domain (like example.test shared by many subdomains) and an IP address are four different groupings, and publishers can be sensitive at any of them. The reference design groups by origin and lets an operator add stricter groups, for example one shared budget for all subdomains of one organization, when a publisher asks for it.

Each origin gets a bucket that stores next_eligible_at, max_in_flight, the current in_flight count and a backoff_until time. A URL may be dispatched only if its bucket is eligible. The check and the clock advance must be one atomic step: when W-2 and W-3 both ask at t=0 for news-a, one receives the slot for 0 to 0.8 seconds and next_eligible_at becomes 5 seconds. The other does not sleep and retry on its own. It receives nothing for that origin until the bucket says so. Note that W-2 finishing at 0.8 seconds does not make U-105 eligible earlier, because the limit is the gap between request starts, not the duration of a request (scene 12).

Feedback: 429 and Retry-After

Fixed spacing is a floor, not a promise of safety. When news-a answers 429 with Retry-After: 60, the bucket records backoff_until = now + 60 s so every URL for that origin waits, whichever worker would have taken it. With no header, the delay doubles per failure up to a cap and is randomized (jitter) so thousands of URLs do not return in the same second. Retries are capped per URL: 404 is usually final, 503 and timeouts retry a few times, and exhausted URLs go to a terminal state with the last error recorded (scene 23).

Before any page of an origin is fetched, the crawler reads that origin's robots.txt, announces its user-agent name and applies the rules that match it. Atlas follows the Robots Exclusion Protocol as standardized in RFC 9309.

In the story, store-c disallows /account/ and /cart/. U-107 under /account/orders is blocked before any connection opens (scene 14), while U-108 at /products/lamp proceeds to its bucket. The URL is marked BLOCKED with the policy version that blocked it. That version matters because rules change: a URL admitted under policy version 3 must be rechecked against version 4 just before it is fetched, and again after every redirect.

The version 0 process dies at three in the morning and takes eleven thousand queued addresses, the seen-set and every politeness clock with it (scene 16). Restarting from the seeds repeats a day of work and bursts every publisher at once. State that decides what happens next must live outside the process.

The frontier becomes a durable table. One row per URL holds normalized_url, origin_key, state, priority, next_fetch_at, attempt_count and lease fields. A unique key on the normalized URL makes first-seen detection a database decision instead of a thread race. Origin buckets are derived from the same table, so a restart means resume.

A plain message broker could deliver URLs, but a crawler needs time-ordered lookup by origin, priority, deduplicate-on-insert and retry counts. Those are table features. A broker still has a role, for example carrying index-ready events (chapter 15), but the authoritative record of what must be fetched is the table.

Claim with a lease

Workers claim work instead of reading it. In one short transaction the frontier picks an eligible origin that is under its caps, picks its best URL, sets the state to LEASED, writes a lease id, increments a fence token and sets an expiry. W-2 claims U-104 with fence 7 and a 60-second lease (scene 18). The fetch itself happens outside the transaction, because no database transaction can span an HTTP request.

Now the failure. W-2 fetches U-104 successfully and crashes before committing. After 60 seconds the sweeper returns the row to ELIGIBLE with the attempt count kept, and W-3 fetches the page again. The publisher saw two requests. That is the honest guarantee: at-least-once fetch attempts, never exactly-once network I/O. The design makes the stored result safe to write twice rather than promising that it will be fetched once (scene 19).

A harder failure: W-2 was not dead, only slow. Its lease expired, W-3 leased the URL with fence 8 and committed, and then W-2 wakes up and tries to commit with fence 7. The store compares the fence on the row with the writer's token and rejects the stale write (scene 20). Time alone cannot decide who owns the work because pauses and clocks lie; a token that only grows can.

With a durable frontier, fetch workers become stateless and horizontally scalable. The scheduler is the interesting part.

A single global queue ordered by priority sounds right and is wrong. If news-a holds four million waiting URLs it fills the head of the queue, and workers spend their time waiting for news-a's five-second clock while docs-b and store-c wait behind them (head-of-line blocking). The scheduler therefore chooses an origin first, then that origin's best URL: origins sit in a min-heap keyed by next_eligible_at, the scheduler pops the earliest eligible origin that is under its concurrency cap, and takes the highest-priority URL from that origin's bucket.

Fairness needs three mechanisms. Aging raises the priority of an origin as it waits. Class quotas stop big publishers from consuming the entire fetch budget (for example, no class takes more than a quarter of attempts while others have eligible work). A starvation cap guarantees every origin with eligible URLs is served within a bounded time (scene 21).

Scaling out is then simple but bounded. Throughput is roughly the smaller of global concurrency (about 116 in flight on average here) and the sum of per-origin rate caps over origins that have eligible work (scene 32). Replicas help only while many different origins are eligible at once. They never break a host cap, because the cap lives in the bucket rather than in the worker. Partition the frontier by origin, not by URL: the origin bucket is the coordination boundary, and sharding by URL would scatter one origin's budget across partitions.

The fetcher is where the crawler touches the hostile, messy internet. Treat every response and every address as untrusted.

Redirects

U-104 answers 301 to a different host. An HTTP client that follows redirects silently skips every policy check. Atlas treats the target as a fresh candidate: normalize, check scope, read the target origin's robots rules, take a slot in the target origin's bucket and validate its network address. Hops are bounded (five here) and a repeated normalized URL ends the chain. Both the original and final URL are stored (scene 22).

Where a connection may go (SSRF)

A public page can link to a hostname that resolves to an address inside Atlas's own network, where internal services and cloud metadata endpoints live. If the crawler connects, a stranger has made it fetch internal data. This is server-side request forgery. The defense is at connect time: allow only http and https; resolve the name; check every returned address against private, loopback, link-local, multicast and metadata ranges for both IPv4 and IPv6; connect to the validated address and keep it pinned for the connection, so a name server that answers differently the second time (DNS rebinding) cannot change the destination; verify TLS; and repeat all of it per redirect. Checking the hostname once before connecting is insufficient (scene 24).

Bounding responses

A few kilobytes of compressed data can expand to gigabytes, and a response labeled HTML can be a binary file. Limits apply to the streamed decompressed bytes, not only to Content-Length. The fetcher stops reading at 5 MB for a page, checks the declared type against an allowlist, sniffs the first bytes, and runs parsers with time and memory limits. Anything suspicious goes to a QUARANTINED state with the reason, and never reaches the parser (scene 25).

From bytes to a document

The parser detects the character set, reads the base address, extracts text, title, language and links, notes rel=nofollow according to policy, and records the canonical link as a hint. Fetch success, parse success, storage success and index readiness are recorded as separate stages, so a drop in indexed pages can be traced to the network, the parser or the pipeline after it (scene 26).

'Have I seen this?' has three different answers depending on what is being compared and when.

URL dedupe happens before any network cost. Discovery does an insert-if-absent on the unique normalized URL, so exactly one of two simultaneous discoveries wins regardless of thread or machine. An in-memory visited set cannot do this: two threads can both see a URL as new, and the set vanishes on restart (scene 27). Content dedupe happens after extraction: hash the extracted text. When U-104 and its print version /print return identical text, the document is stored once and both URLs map to it, with the places each link was discovered kept as provenance. Dropping the second URL would lose that history (scene 28). Near-duplicate detection (SimHash or MinHash style fingerprints) estimates that two pages are, say, 96 percent alike. It can cluster them for the indexer, but a threshold can merge a corrected article with its outdated version, so clusters are labels, every member keeps its document, the threshold is recorded and a bad merge can be undone (scene 29).

Bloom filters at the front door

At very large scale, a Bloom filter (a compact bit structure that can answer 'definitely not seen' or 'maybe seen') can avoid most database lookups. It never gives a false 'no', but it can give a false 'maybe'. The rule is therefore one-way: a 'no' lets the URL be inserted without a lookup; a 'maybe' must be confirmed in the exact store before any URL is discarded.

A crawl that fetches each page once is a snapshot. Six hours after Atlas fetched U-104 the publisher corrected the headline, and Atlas's index still shows the old one (scene 33). The gap between a page changing and Atlas noticing is its freshness gap, and it should be measured.

Recrawling is conditional. The row stores the ETag and Last-Modified validators from the last fetch. The recrawl sends If-None-Match with the ETag. If the server answers 304 Not Modified, a few hundred bytes cross the network instead of 120,000, nothing is reparsed or reindexed, and the row's last_checked_at is updated (scene 34). Two cautions. A 304 is still a request, so it spends an origin slot and politeness budget. And validators are hints: some servers send ETags that change with no content change, or ignore conditional headers. A new ETag with a 200 therefore triggers a parse and a content-hash comparison. If the hash differs, the page truly changed, a new version is stored and an outbox event is emitted (scene 35).

How often to come back is a scheduling problem. Atlas estimates each page's change interval from its history, scales by importance, clamps the result between a minimum and a maximum, and applies jitter so thousands of pages fetched together do not all fall due together (scene 36). A fixed share of the daily budget, 30 percent in this illustration, is reserved for discovering new URLs so that freshness work cannot consume the whole crawl.

For news-a and docs-b the HTML contains the article. For store-c product pages the fetched HTML is a two-kilobyte shell and the product text is drawn by JavaScript after load (scene 37). A plain fetcher extracts zero words.

A browser would see the content, but rendering costs many times the processor and memory of a plain fetch and executes scripts from many third parties. Atlas therefore decides by measurement: for each site or page class, sample raw extraction against rendered extraction. Where raw HTML already holds the content, rendering adds only cost. Where it does not, those pages go to a separate render lane.

The lane is sandboxed, has its own queue, CPU, memory and time limits and its own metrics. If it saturates, the crawler falls back to HTML-only for those pages and records the fallback. Nothing about the rules relaxes: every request the browser makes, including images and scripts, passes through the same safe egress gate and counts against the same origin bucket. A render lane is not a side door around robots or SSRF protection (scene 38).

A fetched, parsed page has to be stored and the indexer told. Four separate systems are involved: the HTTP origin, object storage, the metadata database and a message broker. No single transaction spans them, so the order of operations carries the correctness.

The write order is: (1) put the body in object storage under a content-addressed key, which is safe to repeat; (2) in one database transaction mark the frontier row FETCHED, write the document metadata and write an outbox row that says 'index this document'; (3) a relay process reads the outbox and publishes to the indexer, resending after a crash; (4) the indexer deduplicates by event id. An event therefore exists only if the document is durably stored, and a stored document eventually produces an event. A crash after step 1 leaves a harmless orphan body that a sweeper removes; a crash after step 2 resumes from the outbox (scene 30).

The remaining design is about operating the crawler at scale and across regions without losing its guarantees.

Backlog. Accepted discoveries exceed fetch capacity (scene 31), so admission decisions use priority, per-origin budgets and retention. Unfetched low-value URLs are dropped on purpose, and the drop is visible in metrics.

Regions. Atlas adds a European and a US region. Origin buckets are partitioned by origin and each bucket is owned by one region at a time through an ownership lease from a control plane. For a very busy origin, the rate budget can be split into pre-leased shares so regions need not coordinate on every request (scene 39). If a network partition leaves both regions believing they own news-a, the bucket accepts only the newest ownership token and rejects the older one (scene 40). Some duplicate fetches inside the old lease window are possible, so the guarantee is a bounded overshoot, not perfect global rate limiting across independent regional counters.

Outage. When the metadata database is down for two hours, workers stop taking new leases, a bounded spool holds finished results and expiring leases return. If 200 tasks per second keep arriving while only 50 complete, the backlog grows by (200 − 50) × 7,200 = 1,080,000 tasks. After recovery at 250 per second against 200 per second of inflow, draining takes 1,080,000 ÷ (250 − 200) = 21,600 seconds, six hours, assuming origin budgets allow that rate. If the backlog is concentrated in a few capped origins the true drain is slower (scene 41).

Observability. The crawler publishes per-origin request rate against its limit and the 429 share (politeness), share of fetches by origin class (fairness), queue age by priority and the oldest eligible URL (backlog), blocked-by-robots and blocked-by-scope shares, parse yield, 304 rate, changed-page yield and dedupe ratio, and freshness gap by priority class (scene 42). Every decision carries a reason code, so 'why did you crawl us at three in the morning' has an answer in the data.

Opt-out and deletion. A verified request goes to a suppression list that blocks future fetches, then frontier rows, parsed documents, raw bodies and index entries are deleted or tombstoned with each action audited.

Here is the mature crawler, with every component traced back to the failure that required it.

Four responsibilities summarize the design, and they are also the diagnostic questions when something breaks. Discover: turn links into a safe, deduplicated identity inside a scope with a budget. Decide: for each URL, whether it may be fetched, when, by whom and in what order. Fetch safely: connect only to validated addresses, bound every response, keep rendering optional and isolated. Refresh: come back when the page is likely to have changed, asking cheaply with validators, within a budget. A flood is a decision failure, a missing page is a discovery or scope failure, a private address fetched is a safety failure and an old article is a refresh failure (scene 44).