System Design · 8 chapters · ~14 min read
Web Crawler Failure and Scale Scenarios: 31 Incidents
What actually goes wrong in a production web crawler, and how is each failure detected, contained and recovered? This page walks 31 concrete incidents in the Atlas Discovery design, from URL traps and robots.txt changes through publisher overload, crashed workers and expired leases, SSRF and decompression bombs, false duplicates and stale validators to region split-brain, a two-hour storage outage with its backlog arithmetic and an opt-out deletion workflow. Each incident uses the same eight fields so failures can be compared.
Chapter 01
The Healthy Path, and How to Read the Incidents
A healthy crawl is dull: a URL is discovered, normalized, approved, scheduled, leased, fetched, parsed, deduplicated, stored and announced, with every state change carrying a reason code. This page takes each stage and breaks it with a concrete incident, using the same eight-point template every time so you can compare them and see how one failure type shares its fix with another.
Swipe to pan
| Template field | Question answered |
|---|---|
| Initiating event | What went wrong first? |
| Timeline | How does it unfold, with times? |
| Affected state | Which rows, buckets or stores are wrong? |
| Detection | Which signal or check notices? |
| Containment | How is damage limited right now? |
| Recovery | How does the system return to correct? |
| Idempotency boundary | Which step is safe to repeat and which is not? |
| Prevention | What change stops a repeat? |
Incidents are grouped by the stage that fails. The Atlas Discovery crawler, the three origins and the numbers match the other Atlas pages. Capacity facts: about 116 requests in flight on average, 144.68 attempts per second, 10 million successes per day. Incident numbers F01–F30 are stable references used by the interview page. Opt-out and deletion handling is covered separately as F31 because it is an operations and policy scenario.
Chapter 02
Identity and Scope Failures
The frontier is only as good as its idea of what a page is.
Swipe to pan
F01 · A calendar generates millions of URLs
| Aspect | What happens |
|---|---|
| Initiating event | /calendar?day=N links to the next day forever |
| Timeline | Over six hours the queue grows by 40,000 entries per hour for one origin |
| Affected state | Queue rows for one origin outnumber real pages 100 to 1 |
| Detection | Distinct-values-per-query-key and per-origin URL counts spike; admission rejects rising |
| Containment | Per-origin URL budget and cap on distinct values for the day key; reason code trap_budget |
| Recovery | Delete or park rows over budget; keep a sample for operators; raise the cap only per origin if justified |
| Idempotency boundary | Insert-if-absent keeps re-discovery harmless |
| Prevention | Scope guard with depth, per-origin budget and key-value caps; alert before the cap |
F02 · Two distinct case-sensitive paths merged
| Aspect | What happens |
|---|---|
| Initiating event | Over-eager rules lowercase every path |
| Timeline | Day 0 a rule ships; day 1 docs-b pages vanish from search; day 3 an editor notices; day 3 rule disabled and replay starts |
| Affected state | /Guide/Setup and /guide/setup become one identity; one page is never fetched |
| Detection | Page count for docs-b falls below the sitemap count; users report missing pages |
| Containment | Disable the rule for that origin; stop merging new URLs |
| Recovery | Replay the raw discovered URLs with corrected rules (kept in raw_discovered_url) |
| Idempotency boundary | Re-inserting by corrected identity is safe |
| Prevention | Conservative identity, per-origin rules, audit of rules fired, canary comparison to sitemaps |
F03 · Two threads discover one URL
| Aspect | What happens |
|---|---|
| Initiating event | Both parse pages linking to U-106 at the same moment |
| Timeline | t=0 both threads parse pages linking to U-106; t=2 ms both check the set; both enqueue |
| Affected state | An in-memory set lets both see 'new' and both enqueue it |
| Detection | Duplicate-attempt rate per URL |
| Containment | Unique key on normalized URL |
| Recovery | Dedupe duplicate rows; no data lost |
| Idempotency boundary | The insert is atomic: one wins, the other sees 'exists' |
| Prevention | First-seen decided only by the database, never by a process-local set |
F04 · Redirect loop and cross-origin redirect
| Aspect | What happens |
|---|---|
| Initiating event | docs-b redirects /a → /b → /a; another URL redirects to a third-party host |
| Timeline | t=0 /a → 301 /b; t=1 s /b → 301 /a; after five hops or a repeated URL the chain stops |
| Affected state | Hops accumulate until the cap of five; or the target host is unchecked |
| Detection | Hop-count histogram; redirects to unfamiliar origins |
| Containment | Stop at five hops or a repeated normalized URL; treat a new origin as a new candidate |
| Recovery | Mark FAILED with redirect_loop; the new origin passes its own robots and bucket |
| Idempotency boundary | Each hop is a separate state record |
| Prevention | Hop limit, repeated-URL memory, full gate re-evaluation per hop |
Chapter 03
Robots and Policy Failures
Policy is data that changes underneath queued work.
Swipe to pan
F05 · Robots disallow added while URLs are queued
| Aspect | What happens |
|---|---|
| Initiating event | store-c adds Disallow: /products/ at 10:00 |
| Timeline | 10:00 file refetched (version 4); 10:02 claim sees U-108 verdict at version 3 |
| Affected state | Queued product URLs with stale verdicts |
| Detection | policy_version_mismatch counter |
| Containment | Recheck at claim time; block U-108 before any connection |
| Recovery | Re-admit if a later rule allows it |
| Idempotency boundary | The block is a state change, repeatable |
| Prevention | Version on every verdict; short robots cache; recheck after redirects |
F06 · Robots returns 404, 503 or times out
| Aspect | What happens |
|---|---|
| Initiating event | One origin returns three different failures on three days |
| Timeline | Mon robots 404 (crawl); Tue 503 for 20 minutes (pause, last good copy); Wed timeout (pause, back off) |
| Affected state | 404: no restrictions published; 503 and timeouts: origin unreachable |
| Detection | Robots status counters by class |
| Containment | 404: crawl normally; 5xx/timeouts: pause, use last good copy for a bounded time, back off |
| Recovery | On recovery, refetch and clear the pause |
| Idempotency boundary | Refetching robots is safe |
| Prevention | Standards-based semantics; the conservative choice for unreachable; documented operator override |
F07 · Wildcard and longest-match edge
| Aspect | What happens |
|---|---|
| Initiating event | Allow: /account/help/ against Disallow: /account/; a buggy matcher picks the first rule |
| Timeline | Deploy at 09:00; 09:05 a path under /account/help/ is wrongly blocked or fetched; audit at 10:00 finds it |
| Affected state | A path under /account/ is fetched or wrongly blocked |
| Detection | Parser unit tests; blocked-versus-allowed spot audits |
| Containment | Disable the parser change; replay checks over recent fetches |
| Recovery | Re-evaluate and tombstone documents that should not have been fetched |
| Idempotency boundary | Re-evaluation is deterministic |
| Prevention | Table-driven tests from the standard's examples; longest match with Allow winning ties |
Chapter 04
Politeness and Scheduling Failures
Fairness and rate limits are enforced in shared state; they fail when it is bypassed or unavailable.
Swipe to pan
F08 · Two hundred workers sleep and fire together
| Aspect | What happens |
|---|---|
| Initiating event | Workers each wait five seconds without a shared clock |
| Timeline | t=0 200 workers wake together; t=0.2 s news-a answers 429 to most; t=1 s origin enters backoff |
| Affected state | At t=0 all 200 send to news-a; 429 follows |
| Detection | Requests per origin per second above its cap |
| Containment | Origin bucket claim advances the clock atomically; others get other origins |
| Recovery | Origin enters backoff |
| Idempotency boundary | The claim is atomic; retrying the claim is safe |
| Prevention | Shared per-origin slot reservation; no per-worker sleeps |
F09 · 429 with Retry-After: 60
| Aspect | What happens |
|---|---|
| Initiating event | W-2 receives 429 at t=10 s |
| Timeline | t=10 s 429 with Retry-After: 60; t=10 s bucket sets backoff_until=70 s; t=70 s plus jitter resumes |
| Affected state | All of news-a's queued URLs wait until t=70 s plus jitter |
| Detection | 429 share per origin |
| Containment | backoff_until set on the origin, not the URL |
| Recovery | Resume with jittered release |
| Idempotency boundary | State updates are idempotent |
| Prevention | Origin-wide backoff; honor the header exactly |
F10 · Repeated 503s with no Retry-After
| Aspect | What happens |
|---|---|
| Initiating event | docs-b degrades under load |
| Timeline | t=0 503; t=10 s retry 503; t=30 s retry 503; delays 10, 20, 40 s up to a 15-minute cap, then probe |
| Affected state | Delays grow 10 s, 20 s, 40 s … capped at 15 min |
| Detection | 5xx rate and latency per origin |
| Containment | Exponential backoff, jitter, circuit cooling |
| Recovery | Probe with one request; ramp slowly |
| Idempotency boundary | Backoff state is derived from counters |
| Prevention | Cap, jitter and a probing recovery |
F11 · A popular host absorbs the frontier
| Aspect | What happens |
|---|---|
| Initiating event | news-a has four million high-priority URLs |
| Timeline | Hour 0 news-a backlog arrives; by hour 2 small origins wait 90 minutes; alert fires; quotas apply |
| Affected state | Small origins' wait rises to hours |
| Detection | Oldest-eligible age by origin class |
| Containment | Per-class quotas and aging |
| Recovery | Starved origins served within the starvation cap |
| Idempotency boundary | Rescheduling is idempotent |
| Prevention | Origin-first scheduling; quotas; starvation alert |
Swipe to pan
F12 · Scheduler dies after reserving a slot
| Aspect | What happens |
|---|---|
| Initiating event | Slot reserved at t=0; process crashes before dispatch |
| Timeline | t=0 slot reserved; t=0.1 s scheduler crash; t=30 s reservation expires; sweeper releases it |
| Affected state | The origin looks busy: in_flight is 1, next eligible 5 s |
| Detection | in_flight stuck beyond lease time |
| Containment | Reservation carries an expiry; sweeper |
| Recovery | Sweeper releases the reservation; clock is not rewound (conservative) |
| Idempotency boundary | Reservation rows are idempotent |
| Prevention | Short reservation expiry; sweeper |
Chapter 05
Worker, Lease and Storage Failures
These incidents show the guarantee the crawler can honestly make: at-least-once attempts, idempotent results.
Swipe to pan
F13 · Worker crashes after fetch, before commit
| Aspect | What happens |
|---|---|
| Initiating event | W-2 receives the 200 for U-104 at t=1 s and dies |
| Timeline | Lease expires at 60 s; sweeper returns the row |
| Affected state | U-104 stays LEASED then ELIGIBLE with attempts=1 |
| Detection | Lease expiry counter |
| Containment | Sweeper requeues the URL; another worker fetches it, so the publisher sees a second request |
| Recovery | The second fetch commits normally; the duplicate request is honest, visible and counted |
| Idempotency boundary | Commit is fence-checked; body is content-addressed |
| Prevention | Short leases; do not promise exactly-once |
Swipe to pan
F14 · A slow fetch outlives its lease
| Aspect | What happens |
|---|---|
| Initiating event | W-2's fetch stalls for 70 s, longer than its 60 s lease |
| Timeline | t=0 W-2 leases U-104 (fence 7); t=60 s lease expires; t=61 s W-3 leases (fence 8) and commits; t=70 s W-2 tries fence 7 |
| Affected state | U-104 now belongs to fence 8; W-2 holds a stale result that must not overwrite it |
| Detection | Stale-writer rejections |
| Containment | Reject stale commit; discard result |
| Recovery | None needed |
| Idempotency boundary | Fence compare-and-set |
| Prevention | Fencing tokens; lease extensions only with the current fence |
F15 · Page stored but links lost
| Aspect | What happens |
|---|---|
| Initiating event | Document row written; the links batch fails |
| Timeline | t=0 fetch ok; t=1 s document row committed; t=1.1 s links batch fails; no retry path without a record |
| Affected state | The document exists, its outlinks never reach the frontier |
| Detection | Link count mismatch versus parsed output |
| Containment | Write links in the same transaction or via outbox |
| Recovery | Replay extraction from the stored body |
| Idempotency boundary | Insert-if-absent for links; document upsert by hash |
| Prevention | Same-transaction link writes or outbox |
Swipe to pan
F16 · Publish succeeds but commit fails, or the reverse
| Aspect | What happens |
|---|---|
| Initiating event | Event sent, then the database errors; or the commit succeeds and publishing fails |
| Timeline | Order A: t=0 publish ok; t=0.1 s commit fails. Order B: t=0 commit ok; t=0.1 s publish fails |
| Affected state | Index points at a missing document; or a document is never announced |
| Detection | Outbox backlog; index-versus-document reconciliation |
| Containment | Outbox pattern: event exists only if the row exists |
| Recovery | Relay resends; reconciler compares both sides |
| Idempotency boundary | Consumer dedupes by event_id |
| Prevention | Never publish before commit; outbox plus idempotent consumer |
Chapter 06
Network, Security and Parsing Failures
The crawler reads untrusted data and follows untrusted addresses.
Swipe to pan
F17 · DNS rebinding
| Aspect | What happens |
|---|---|
| Initiating event | A name resolves to a public address at validation and to 10.0.0.5 at connect |
| Timeline | t=0 validator resolves to a public IP; t=0.2 s connect-time lookup returns 10.0.0.5 |
| Affected state | If the client resolves twice, an internal service is reached |
| Detection | Egress denies to private ranges; logs |
| Containment | Pin the validated address to the connection; network deny rules |
| Recovery | Rotate any exposed credentials; review logs |
| Idempotency boundary | Connection retried only after re-validation |
| Prevention | Connect-time validation, pinning, network-layer denies |
Swipe to pan
F18 · Redirect to metadata or localhost
| Aspect | What happens |
|---|---|
| Initiating event | A page redirects to http://169.254.169.254/ or localhost |
| Timeline | t=0 U-104 returns 302 to http://169.254.169.254/; t=0 ms the hop is evaluated and refused |
| Affected state | The hop is refused |
| Detection | blocked_network counter |
| Containment | Refuse the hop; mark FAILED |
| Recovery | None |
| Idempotency boundary | Refusal changes no external state |
| Prevention | Per-hop scheme and network checks; metadata ranges blocked by network policy |
F19 · Decompression bomb
| Aspect | What happens |
|---|---|
| Initiating event | A 40 KB compressed response expands to gigabytes |
| Timeline | t=0 40 KB response arrives; as it expands the reader passes 5 MB and aborts at t=0.3 s |
| Affected state | The reader aborts at 5 MB decompressed |
| Detection | Memory and abort counters |
| Containment | Streaming limit on decompressed bytes; parser in a limited process |
| Recovery | Quarantine the URL |
| Idempotency boundary | Abort leaves no stored body |
| Prevention | Limit by streamed decompressed bytes, not Content-Length |
F20 · HTML label, binary payload
| Aspect | What happens |
|---|---|
| Initiating event | Content-Type: text/html on a PDF or garbage |
| Timeline | t=0 headers say HTML; first bytes show %PDF; sniff rejects at t=0 before parsing |
| Affected state | Sniffing sees a binary signature |
| Detection | Mismatch counter |
| Containment | Type allowlist; sniff; reject or quarantine |
| Recovery | Reprocess under the right handler if supported |
| Idempotency boundary | No document written |
| Prevention | Allowlist plus sniffing; no execution of unknown types |
Swipe to pan
F21 · The render lane exhausts CPU and memory
| Aspect | What happens |
|---|---|
| Initiating event | A script-heavy class floods the browser pool |
| Timeline | Hour 0 a script-heavy store-c class is marked render-required; by hour 1 render queue age passes 10 minutes |
| Affected state | Queue depth rises; browsers evicted |
| Detection | Lane CPU, memory and queue age |
| Containment | Separate queue and limits; fallback to HTML-only with render_skipped |
| Recovery | Scale or tighten limits; re-render later |
| Idempotency boundary | Render results are keyed by URL and content hash |
| Prevention | Isolation, budgets, fallback |
F22 · One broken charset poisons a batch
| Aspect | What happens |
|---|---|
| Initiating event | A parser crash on a malformed page kills a worker repeatedly |
| Timeline | t=0 a page with a broken charset crashes W-4; the batch is redelivered; W-5 crashes the same way |
| Affected state | The same batch fails over and over |
| Detection | Crash loops and quarantine counters |
| Containment | Per-page isolation and timeouts; quarantine the URL |
| Recovery | Fix the parser; replay quarantined pages |
| Idempotency boundary | Quarantine is a state, replay is safe |
| Prevention | Parser limits; skip poison pages; canary parsing |
Chapter 07
Duplicate and Freshness Failures
Dedupe and conditional requests fail quietly, so they need verification built in.
Swipe to pan
F23 · A Bloom false positive hides a new URL
| Aspect | What happens |
|---|---|
| Initiating event | The filter says 'maybe seen' for the only link to a valuable page |
| Timeline | t=0 the filter reports 'maybe' for a new URL; with the bug the URL is dropped; weeks later a gap is found |
| Affected state | If the pipeline discards on 'maybe', the page is never fetched |
| Detection | Exact-store confirmation counters; unexplained coverage gaps |
| Containment | Verify every 'maybe' in the exact store |
| Recovery | Backfill pages found missing |
| Idempotency boundary | Insert is idempotent |
| Prevention | The filter only avoids lookups; it never discards |
F24 · Different URLs, identical content
| Aspect | What happens |
|---|---|
| Initiating event | U-110 /print returns the same text as U-104 |
| Timeline | t=0 U-104 stored; t=3 min U-110 /print fetched; the hash equals U-104's; a second URL row links to the document |
| Affected state | Both rows reference one document; both provenance links kept |
| Detection | Dedupe ratio per origin |
| Containment | Hash on normalized text |
| Recovery | None |
| Idempotency boundary | Upsert by content hash |
| Prevention | Provenance table; never drop the second URL |
F25 · Near-duplicate merges two real versions
| Aspect | What happens |
|---|---|
| Initiating event | A corrected article joins the cluster of the outdated version |
| Timeline | Day 0 a corrected article arrives; fingerprint distance is below the threshold; it is clustered with the old version |
| Affected state | Users see the wrong version first |
| Detection | Cluster review sampling; complaints |
| Containment | Threshold recorded; members kept |
| Recovery | Split the cluster; reprocess |
| Idempotency boundary | Clusters are labels, not deletions |
| Prevention | Conservative threshold; audit; reversible clusters |
Swipe to pan
F26 · ETag changes with no content change
| Aspect | What happens |
|---|---|
| Initiating event | The origin generates a fresh ETag per request |
| Timeline | Day 0 origin begins issuing a new ETag per request; every recrawl returns 200; reparse yields the same hash |
| Affected state | Each recrawl is a 200 that reparses to the same hash |
| Detection | Changed-page yield near zero while 304 rate falls |
| Containment | Compare content hash; stop trusting that origin's validators |
| Recovery | Fall back to hash-based change detection |
| Idempotency boundary | Hash comparison decides change |
| Prevention | Per-origin validator reliability tracking |
F27 · A 304 mishandled as missing
| Aspect | What happens |
|---|---|
| Initiating event | Code treats the empty 304 body as a failure |
| Timeline | t=0 recrawl returns 304 with an empty body; a bug marks the document missing |
| Affected state | Documents appear deleted |
| Detection | Sudden drop in documents with 304 responses |
| Containment | Treat 304 as 'unchanged' and reuse the stored document |
| Recovery | Restore the affected rows from metadata |
| Idempotency boundary | 304 handling is read-only |
| Prevention | Link the validator to the stored document; tests for 304 |
F28 · A news site speeds up and the schedule goes stale
| Aspect | What happens |
|---|---|
| Initiating event | Publishing frequency doubles |
| Timeline | Week 0 the site publishes hourly; week 1 it publishes every 20 minutes; intervals lag until re-estimated |
| Affected state | Freshness gap rises for that origin |
| Detection | Freshness gap by class |
| Containment | Bounded fairness; temporary interval cap |
| Recovery | Adaptive scheduler shortens the interval |
| Idempotency boundary | Interval updates are derived from history |
| Prevention | Clamped intervals; regular re-estimation |
Chapter 08
Region and Platform Failures
Scale-out and outages stress the guarantees the design makes.
Swipe to pan
F29 · Region split-brain
| Aspect | What happens |
|---|---|
| Initiating event | A partition leaves both regions believing they own news-a |
| Timeline | t=0 partition; t=0 to 30 s both regions schedule news-a; t=30 s partition heals; token 42 wins over 41 |
| Affected state | Both schedule news-a for up to the lease window |
| Detection | Origin request rate above cap; token conflicts |
| Containment | Ownership token 42 beats 41; the older owner is fenced |
| Recovery | Reassign; reconcile duplicates |
| Idempotency boundary | Fence compare on the bucket |
| Prevention | Bounded overshoot by lease length; preleased shares |
Swipe to pan
F30 · Multi-hour storage outage
| Aspect | What happens |
|---|---|
| Initiating event | The central metadata store is unavailable for 2 hours |
| Timeline | t=0 store down; t=0 to 2 h arrivals 200/s, completions 50/s; t=2 h recovery; drain begins at 250/s |
| Affected state | Backlog = (200 − 50) × 7,200 = 1,080,000 tasks |
| Detection | Queue age; failed commits; spool occupancy |
| Containment | Workers stop leasing; a bounded spool holds finished results; shed low-priority work |
| Recovery | After recovery at 250 per second against 200 per second of inflow, drain = 1,080,000 ÷ 50 = 21,600 s = 6 h, provided origin budgets allow it; a backlog concentrated in capped origins drains slower |
| Idempotency boundary | Spooled commits replay by fence and hash; reattempt storms are jittered |
| Prevention | Spool limits, shedding rules, retry jitter, drills |
The fleet-wide estimate assumes the backlog is spread evenly over many origins. If most queued work belongs to a few origins that are rate-capped, the true drain time is set by those caps (for news-a at one request per 5 s, one million URLs would need about 58 days), not by fleet capacity. Always compute the drain per bottleneck origin before promising a recovery time.
Opt-out and deletion requests (F31)
Swipe to pan
F31 · A verified opt-out and deletion request
| Aspect | What happens |
|---|---|
| Initiating event | A publisher asks that its pages be removed |
| Timeline | Day 0 verified request received; day 0 suppression; day 1 to 3 deletions run; day 3 audit confirms |
| Affected state | Frontier rows, documents, raw bodies and index entries for that origin |
| Detection | Rows and index entries still present after the deadline |
| Containment | Suppression blocks new fetches immediately |
| Recovery | Delete or tombstone each data class; audit trail |
| Idempotency boundary | Deletion is repeatable by key |
| Prevention | Per-class retention, deletion map, suppression record kept |
Key takeaways
- 01Most failures are quiet: wrong merges, stale validators and lost links produce no errors.
- 02At-least-once attempts plus idempotent commits are the honest guarantee.
- 03Every shared limit lives in shared state; per-worker sleeps cannot enforce it.
- 04Security failures are stopped at connect time, per hop and by bounded streaming reads.
- 05Regional overshoot is bounded by the lease window.
- 06Recovery estimates must be computed per bottleneck origin.