Web Crawler Design Tradeoffs: 24 Decisions Compared

Every design choice in the Atlas Discovery crawler had a plausible alternative. This page compares them using one frame: the problem that forces the choice, realistic options, what the reference design picks, what that costs and what circumstance would change the answer. Terms such as origin, frontier, lease and fence token are defined where they first matter, so the page stands alone. Decisions are numbered in the order they appear on this page and grouped by theme.

The first group decides what the crawler fetches next and who is allowed to be served.

Decision 1 · Breadth-first order or a prioritized frontier?

Plain breadth-first search covers a site evenly but treats a decade-old archive page like today's front page, and it makes a recrawl no different from a first visit. A prioritized frontier scores each URL (importance, expected change rate, age, depth) and picks the best first. Breadth-first is simple and gives good early coverage when nothing is known about the pages. The reference uses priorities with aging, which raises a waiting URL's score over time so low-priority pages are not postponed forever. The cost is a scoring function that must be tuned and monitored. Choose plain breadth-first for a one-off, small crawl with no freshness goal.

Decision 2 · One global FIFO queue or per-origin ready buckets?

With one FIFO, four million URLs from news-a sit at the head, and each worker that pops one must wait for news-a's five-second clock while docs-b and store-c wait behind them. That is head-of-line blocking. Per-origin buckets let the scheduler pick the origin that is due and skip the ones that are not. Buckets add shared scheduling state and a heap, and they require a clear answer to 'who advances the clock' (the claim transaction). A single FIFO is acceptable when you crawl a handful of origins you control.

Decision 3 · What counts as 'the host' for politeness?

Four groupings exist: the host name, the origin (scheme, host and port), the registrable domain (example.test and everything under it) and the IP address. Grouping by host alone lets http and https on the same machine each take a full budget; grouping by IP penalises small sites on shared hosting; grouping by domain is safest for one publisher with many subdomains but slows unrelated sites that share a registrable suffix, such as a blog platform. The reference groups by origin and lets operators add stricter groups where a publisher asks. Expect policy ambiguity: the right group is ultimately what the publisher's infrastructure can bear, which Atlas learns through 429 and latency feedback.

Decision 4 · Fixed spacing or load-feedback adaptation?

A fixed spacing of five seconds is predictable and easy to explain to a publisher, but it cannot tell when a site is struggling or when it could take more. Adaptive control watches status codes and latency, lowers the rate on 429, 503 or rising response time, and raises it slowly when everything is healthy. The risk is instability: raise quickly, drop quickly, and the rate oscillates. The reference sets a fixed floor and ceiling per origin, honors Retry-After exactly, and adapts inside that band with additive increase and multiplicative decrease. A fixed spacing alone suffices for a small crawler that fetches few pages per origin.

Decision 5 · Central scheduler or distributed host leases?

Central scheduler: one service owns every bucket and decides each claim, which gives exact eligibility and simple reasoning. At 1,157 attempts per second at burst, that is about one decision per millisecond, which one sharded database can serve, but it concentrates failure and adds a network round trip to each claim. Host leases: each worker group is assigned a set of origins for a period and schedules them locally, trading exactness at handoff for no per-claim round trip.

The reference uses a sharded central frontier inside a region, partitioned by origin, and host leases only across regions (Decision 23). Choose a pure lease model sooner if claim latency is a large share of total fetch time.

The frontier is the source of truth about what work remains, so its storage choices decide recovery behavior.

Decision 6 · Persistent database frontier or broker-only queue?

A message broker is excellent at delivering work and poor at the questions a crawler asks constantly: 'what is the best eligible URL for origin X right now?', 'is this URL already known?', 'how many attempts has it had?', 'when is it next due?'. Those are indexed lookups with time ordering and uniqueness, which a database provides. A broker-only design must bolt on a separate store for state anyway. The reference keeps the authoritative frontier in a table and uses a broker for the outbound events, where at-least-once fan-out is exactly the need. Choose a broker-only pipeline for stateless, short-lived crawls where losing the queue is acceptable.

Decision 7 · Shard by URL or shard by origin?

Sharding by URL hash spreads rows evenly, but a single origin's budget would then live on many shards; every shard would need to agree on one next_eligible_at. Sharding by origin keeps the budget local to one shard, so the claim is a local transaction. The price is skew: one giant origin concentrates load on a shard. The reference shards by origin hash and splits only the very largest origins into sub-partitions that share a fixed slice of the origin budget. If your publisher count is small and each is huge, URL sharding with a shared budget service may be necessary.

These decisions control how many pages the crawler believes exist.

Decision 8 · Exact database dedupe or a Bloom prefilter?

The exact path is one unique-key insert per discovered URL. At 27 million accepted discoveries per day that is about 312 per second on average, trivial for a database. At one hundred times that scale, the lookups dominate. A Bloom filter answers 'definitely new' without touching the database.

At 27 million per day the filter saves almost nothing and adds a failure mode, so the reference omits it and says when to add it: when exact lookups, not fetches, become the bottleneck.

Decision 9 · Strict normalization or conservative identity?

Aggressive normalization (lowercase paths, drop all query parameters, strip trailing slashes) shrinks the URL count and silently merges distinct pages, which loses content permanently and invisibly. Conservative normalization merges only what is provably equal and tolerates duplicates, which cost bandwidth that content dedupe later reclaims. Over-fetching is a visible cost, over-merging is invisible data loss; the reference accepts the visible cost. Per-origin rules (a maintained tracking-parameter list) provide the middle path.

Decision 10 · Trust the publisher's canonical link or verify content?

rel=canonical is a claim made by the page. Honoring it saves fetches, but it can be wrong, stale or hostile: a page can name an unrelated canonical, and a redirect may point at a retired address. The reference treats the canonical as a hint that raises the priority of the named URL and records a candidate relation, and merges only when content hashes confirm it. A trusted-publisher allowlist may skip verification when you control the site.

Decision 11 · Exact content hash or near-duplicate matching?

An exact hash of normalized text is cheap, has no false merges and finds true copies. It misses pages that differ by a timestamp or a related-links box. Near-duplicate fingerprints (SimHash, MinHash) find 96-percent-alike pages and let the indexer cluster them, but any threshold will occasionally join a corrected article with its older version. The reference uses exact hashes for storage dedupe and similarity only for labels on clusters, with the threshold recorded and each member kept so a bad merge is reversible. Use similarity-based removal only if storage cost, not precision, is the dominant concern.

These decisions determine what a single request is allowed to do and cost.

Decision 12 · Plain HTML fetch or selective JavaScript rendering?

Rendering every page costs many times the CPU and memory of a plain fetch and runs third-party scripts. Not rendering anything misses script-built content. The reference samples each page class and renders only where raw extraction recovers too little. Worked cost: suppose a plain fetch uses 20 ms of processor time and a rendered one 600 ms, and 5% of the 144.68 attempts per second need rendering.

Selective rendering needs a sample comparison process and a fallback when the lane saturates, but it is the only option that bounds third-party script risk and cost simultaneously.

Decision 13 · Robots fetch and caching strategy

Fetching robots.txt before every page doubles request counts; caching it too long keeps stale policy. The reference caches up to 24 hours (less if the origin says so), refreshes in the background, records a policy version and rechecks at claim time. On 4xx it treats the file as unavailable and crawls normally; on 5xx or timeouts it treats the origin as unreachable and pauses it, which is the conservative reading. A stricter operator may choose to pause even on 4xx for sensitive categories; a laxer one may keep using stale rules for longer, which should be a recorded decision.

Decision 14 · Fetcher-only SSRF checks or centrally enforced egress?

Embedding checks inside each fetcher means every language runtime and library version must implement the same ranges, DNS pinning and redirect logic. One missed code path is a vulnerability. A central egress gateway or network policy enforces destination rules for every outbound connection, including the render lane and any future library. It adds a hop and an operations dependency. The reference does connect-time validation in a shared library *and* a network-layer deny rule for internal ranges, so a bug in one layer is caught by the other.

These decisions spend the recrawl budget.

Decision 15 · Fixed recrawl cadence or adaptive scheduling?

A fixed daily cadence is predictable and trivial, but it wastes requests on pages that never change and revisits news pages too rarely. Adaptive scheduling estimates each page's change interval. Worked comparison: 1,000,000 pages, 100,000 volatile (change hourly) and 900,000 stable (change monthly).

The cost of adaptivity is an estimator that can be wrong after a behavior change (F28), so intervals are clamped and a minimum revisit budget keeps stable pages from being forgotten.

Decision 16 · Validators or unconditional fetches?

ETag and Last-Modified turn a recrawl into a request that may return 304 with no body, saving bandwidth, parsing and index churn. Validators vary in quality: weak ETags, ETags that change per request, clocks that lie. Unconditional fetches avoid trusting them at the price of the full body every time. The reference sends validators and verifies: a 200 with a new ETag is parsed and the content hash decides whether the page truly changed, and an origin whose validators are repeatedly misleading is marked untrusted. A 304 is still a request, so it consumes an origin slot; validators save bytes, not politeness budget.

Decision 17 · Flat priority or per-origin and per-class fair share?

A single global priority list lets a famous publisher with millions of high-priority URLs absorb the fleet, starving small origins whose pages are equally important to their audience. Fair share assigns each origin or class a share (no class above, for example, 25 percent while others have eligible work), with unused share redistributed. It reduces peak throughput on the biggest publisher and needs quota tuning. A flat priority fits a crawl where all pages belong to one trusted owner.

These decisions determine what happens when workers, networks or stores fail.

Decision 18 · At-least-once fetch attempts or 'exactly once'?

HTTP fetches cannot be made exactly-once: if a worker crashes after the server responded, the crawler cannot know whether the server saw the request. Pretending otherwise either loses work (never retrying) or lies about it. The reference guarantees at-least-once attempts and makes the committed result idempotent by fence token, content-addressed bodies and idempotent consumers. GET requests are safe to repeat; a crawler must never issue non-idempotent methods or follow links that perform actions.

Decision 19 · Short lease, long lease, or heartbeats with fencing?

A long lease (ten minutes) means a dead worker's URL waits ten minutes; a short lease (ten seconds) expires under healthy slow fetches and creates duplicates. Heartbeats extend a short lease while the worker is alive, at the cost of extra writes and the need for fencing, since a worker that stalls may resume after its lease lapsed. The reference uses a lease slightly above the 95th-percentile fetch time, one extension allowed with the fence token, and rejects any commit whose fence is stale. Duplicates then cost bandwidth, not correctness.

Decision 20 · Immediate retry or jittered exponential backoff?

Immediate retries recover a one-off glitch in milliseconds and hammer a server that is already failing. Exponential backoff reduces pressure but, without jitter, thousands of URLs retry in lockstep. The reference honors Retry-After first, otherwise doubles the delay with random jitter up to a cap, applies the delay to the whole origin on 429 or repeated 503, and stops after a limit with an explicit terminal state. A single immediate retry may be fine for local connection resets.

Decision 21 · Store full raw HTML or only extracted documents?

Raw HTML permits reprocessing with a better parser and supports audit, but costs 1.2 TB per day at 120,000 bytes per page and complicates deletion requests. Extracted-only storage is 240 GB per day and loses the ability to re-extract. The reference keeps raw bodies for five days (about 6 TB) and parsed documents for 30 days (about 7.2 TB), each with its own deletion path, and keeps hashes and metadata longer. If legal or audit rules demand an original, extend raw retention for that class only.

The last group concerns running across regions and deciding how hard to push.

Decision 22 · Single region or region-partitioned crawling?

A single region is simpler: one control plane and one set of origin buckets. Crawling from nearby regions cuts round-trip time and can respect data residency. Worked timing: a 160 ms round trip from a distant region versus 25 ms from a close one. Assuming TLS and request take about four round trips, a distant fetch spends 640 ms in the network and a nearby one 100 ms, a difference of 540 ms. With 116 requests in flight on average, cutting each fetch from about 0.8 s to about 0.26 s lets the same fleet do about three times the work (Little's Law: concurrency = rate × time).

The cost is ownership: each origin's bucket needs one owner at a time (Decision 23), policy changes must reach every region, and failover must not create double ownership. Stay single-region until latency or residency actually hurts.

Decision 23 · Global host state or preleased regional budgets?

One global bucket gives exact limits and makes every claim a cross-region round trip, which is slow and unavailable during a partition. Preleased budgets give each region a share of the origin's rate. Worked example: news-a is allowed one request per 5 seconds, which is 0.2 per second. Give the EU region 0.12 per second and the US region 0.08 per second; if a region loses contact, it keeps its own share. If a split-brain leaves both believing they own the whole bucket for a 30-second lease window, overshoot is at most 30 s × 0.2 = 6 extra requests, plus the in-flight ones, and the older ownership token is then rejected.

Decision 24 · Aggressive crawl throughput or conservative safeguards?

Higher rates finish crawls sooner; every complaint and block costs trust that is hard to regain, and a blocked crawler cannot crawl at all. The reference sets conservative per-origin caps, honors every signal, keeps per-origin kill switches and exposes controls so an operator can pause an origin within minutes. Faster crawling is justified only for origins that explicitly allow it, for example through a published agreement. Sustainability beats peak speed because the crawler's value depends on being allowed back tomorrow.