Global CDN — Failure & Scale Scenarios

The origin region or the primary Origin Server path becomes unavailable.

What fails

The CDN can no longer refill cold or expired objects from the preferred origin path. Fresh edge hits are unaffected initially — they never needed origin in the first place. What's actually at risk is fresh misses, revalidations, and any content class where stale-if-error is not a safe fallback.

User impact

Warm cached assets may keep working for a while, but newly requested objects, short-TTL HTML, or uncached media paths begin to fail or serve stale content if the policy allows it.

Detection

Origin health checks, elevated 5xx rates on refill traffic, shield retry spikes, and rising stale-serve counters are the primary signals.

Recovery

Fail over to the secondary origin region or Origin Load Balancer + replicas, serve stale-if-error where policy allows it for that content class, and keep origin retries within a bounded budget — every edge independently retrying the same failing path at once just adds load to an already struggling origin. When origin does return, expect a refill surge as accumulated misses and revalidations all try to catch up at once; that recovery moment deserves its own attention, not just the outage itself.

Mitigations

Use multi-region origins, health-based origin selection, shield caches, bounded and shared origin retry budgets (not independent per-edge retries), and explicit stale-if-error policies scoped to content classes that can tolerate brief staleness.

One Edge PoP or metro location loses capacity or becomes unhealthy.

What fails

Clients that would normally land on that PoP can no longer be served there. Local cache warmth in that location is effectively lost until traffic shifts elsewhere.

User impact

Nearby users may see a small latency increase because they are rerouted to the next-best edge. If reroute is slow or incomplete, some connections may fail during the transition.

Detection

PoP-level health checks, route announcements dropping, sharp local error spikes, and traffic suddenly draining to neighboring metros reveal the issue.

Recovery

Withdraw the Anycast route or stop advertising the affected edge. Convergence is not instant or perfectly uniform — different networks observe and react to the withdrawal at different speeds — so a brief uneven transition is expected, not a bug. Restore the location gradually, only after ingress, cache, and WAF subsystems are healthy again.

Mitigations

Keep enough geographic overlap — and spare capacity at neighboring PoPs — that they can absorb shifted traffic without becoming overloaded themselves, and avoid making any one PoP uniquely responsible for customer config or control-plane state.

A Regional Shield Cache cluster is unavailable or badly degraded.

What fails

The normal fan-in layer between edges and origin disappears, so refill traffic risks becoming much more scattered and much more expensive.

User impact

Users may still receive cached content from local edges, but misses become slower and more likely to fail if the origin is not ready for the increased refill load.

Detection

Shield hit ratio collapses, origin refill concurrency jumps, and edge-miss latency increases sharply within the affected geography.

Recovery

Redirect misses to another shield region or, if necessary, temporarily bypass shield while tightening coalescing and admission controls to protect origin.

Mitigations

Deploy redundant shield regions, keep origin connection budgets, and make shield selection dynamic so the system can reroute refill traffic without a global emergency change.

Failover stampede risk

The recovery step itself can cause a second incident: redirecting every affected edge directly to origin, or all onto one backup shield at once, can overwhelm that fallback path immediately. Mitigate with controlled (gradual, not instant) failover, per-origin concurrency budgets, coalescing preserved at the fallback, and traffic moved in stages rather than all at once.

A hot object expires and thousands of requests arrive before it has been refilled.

What fails

Without protection, many edge or shield nodes attempt the same upstream fetch simultaneously. The cache layer stops being a buffer and starts amplifying load.

User impact

Latency rises suddenly and the origin can tip into overload, turning what should have been one slow refill into a widespread outage.

Detection

You see a synchronized jump in misses for the same key, elevated origin concurrency for one object, and a sudden drop in shield efficiency.

Recovery

Enable or tighten request coalescing (single-flight refill) at the shield, serve stale-while-revalidate where content policy allows it, and consider a bounded refill concurrency limit so the recovering origin isn't hit with unlimited parallel fetch attempts. Extending TTL during an incident can help, but it is not universally safe — for content where staleness itself causes harm, extending TTL just prolongs incorrect data instead of load.

Mitigations

Use shield-level collapse, TTL jitter for extremely hot keys so they don't all expire in the same instant, bounded refill concurrency, hot-key pre-warming ahead of predictable spikes, and versioned assets so popular objects don't need synchronized expiration at all.

An attacker causes incorrect or malicious content to be cached under a shared public key.

What fails

Cache-key design, header normalization, or trust boundaries break down, allowing a response meant for one context to be reused for many others.

User impact

Many users can receive stale, wrong, private, or malicious content from the edge even though the origin itself was never broadly compromised.

Detection

Unexpected cache-key cardinality, anomalous vary patterns, WAF signals, mismatched headers, and sudden spikes in incorrect-response complaints are strong indicators.

Recovery

Purge the affected objects, disable the vulnerable cache rule, tighten normalization, and if necessary force temporary origin bypass until the rule path is safe again.

Mitigations

Use explicit cache keys, limit which headers and query params vary cache behavior, never cache auth-dependent responses publicly, and validate origin-provided cache headers against policy.

Common root causes

This usually traces back to one of a few specific mistakes: cache-key normalization gaps (a header or parameter that affects the response isn't in the key), Vary misuse on a high-cardinality or attacker-influenceable header, auth/cookie handling mistakes that let an identity-specific response get cached under a public key, or trusting unsafe/unvalidated headers coming from origin.

The CDN becomes the first line of defense against a volumetric or application-layer attack.

What fails

The attack shape matters for what actually fails. A volumetric/network attack can saturate ingress link capacity. A connection-exhaustion attack can exhaust connection-handling capacity without necessarily saturating bandwidth. An application-layer attack of individually valid but expensive requests can overwhelm compute or origin capacity while looking like normal, low-volume traffic.

User impact

Legitimate users see increased latency, challenge pages, throttling, or outright failure if bad traffic is not filtered quickly enough.

Detection

Traffic shape anomalies, bot-score collapse, WAF match spikes, SYN or request floods, and unusual geographic concentration all point to active attack conditions.

Recovery

Recovery differs by attack type: volumetric floods are absorbed by scaling edge scrubbing and, if needed, rerouting or blackholing sources; connection exhaustion needs connection-level limits and faster resource reclamation; application-layer floods need request-level inspection, behavioral scoring, and targeted rate limits rather than raw capacity.

Mitigations

Keep DDoS scrubbing at the edge, isolate protected origin paths, rehearse emergency rule pushes, and make sure attack dashboards are regional because floods are rarely uniform everywhere.

One object or one small set of objects suddenly becomes globally hot due to a launch, sports event, or software release.

What fails

The challenge is not total traffic alone; it is traffic skew. A tiny subset of keys consumes a disproportionate share of bandwidth and refill attention.

User impact

If the CDN is prepared, users mostly see success because the content becomes hot at the edge quickly. If not, the first wave of misses can crush shield and origin before the cache warms.

Detection

Top-object concentration rises sharply, the same cache key appears across many PoPs at once, and origin refill graphs show a sudden spike on one asset family.

Recovery

Pre-warm the object when possible, pin it in RAM/SSD for a period, extend TTL carefully, and ensure request coalescing is active at the shield.

Mitigations

Use release-aware prefetch workflows for predictable hotspots, package large media into segments, and maintain enough shield capacity to absorb synchronized first-wave misses.

Already-hot object vs newly-cold-turned-hot object

These are meaningfully different. An object that is already cached and hot just needs its existing warm copies to keep serving — low risk. An object that is globally cold and suddenly becomes hot (a surprise viral moment, an unplanned release) is far more dangerous: the first wave of requests all miss simultaneously and create refill pressure on shield and origin before the object has had any chance to warm up anywhere.

Overall request volume rises sharply across many regions at once, not just on a single object.

What fails

Edge ingress, TLS termination, cache throughput, and WAF capacity are all stressed simultaneously. Even with good hit ratio, the fleet can run short on connection or CPU headroom.

User impact

Users may see queueing, slower TLS handshakes, increased tail latency, or aggressive rate limiting, especially in already busy metros.

Detection

Concurrent connections, per-PoP CPU and network saturation, rising p99 latency by region, and autoscaling lag across the fleet show the system is under broad pressure.

Recovery

Shift traffic across regions, add or scale edge capacity, and reserve origin capacity for truly uncached paths. Security controls should fail safely under load — degradation must not casually disable WAF or rate-limiting just to save cycles, since that trades a capacity problem for a security one.

Mitigations

Provision regional headroom, keep PoPs fungible, separate control-plane work from data-plane resources so purge storms or config pushes do not compete with live traffic during peaks, and enforce per-tenant fairness/admission controls so one workload cannot consume all of a PoP's capacity while others starve.

The CDN is serving content that is syntactically fine but semantically wrong: stale HTML, mismatched JS, or the wrong asset version.

What fails

Freshness policy, deployment sequencing, or purge propagation breaks down. The cache is technically working, but it is serving the wrong version of truth.

User impact

Pages can break in subtle ways: HTML references JS that some PoPs do not have yet, old logos linger, or users see different behavior depending on region.

Detection

Synthetic probes by region, checksum mismatches, elevated frontend error rates after deploys, and customer reports that differ by geography reveal the problem.

Recovery

Roll forward to a coherent version, purge affected paths, or revert to the prior fingerprinted asset set. During incident response, versioned assets are far safer than trying to reason about mixed mutable filenames.

Mitigations

Use Versioned / Fingerprinted Assets, deploy assets before HTML references, and keep purge dashboards so you know where old content still exists.

Why mutable HTML pointing at immutable assets is the classic trigger

This scenario is especially common when mutable entry documents (HTML, manifests) reference immutable fingerprinted assets: if the HTML update reaches some PoPs before the assets it references exist there, or before purge finishes propagating, a request can land on a coherent-looking but internally mismatched combination. Deploying assets before flipping references, and treating rollback to a prior coherent version as a first-class recovery option (not just forward-fixing), both reduce this risk. Synthetic regional verification after deploys catches the mismatch before real users do.

DNS or Anycast routing sends users to the wrong place: too far away, unhealthy, or overloaded.

What fails

Resolver geo-mapping can be wrong, a DNS steering rule can misclassify a region, a bad BGP advertisement or policy can steer traffic incorrectly, or traffic can get shifted onto a PoP that is technically healthy but already near its capacity limit — a subtly different failure than a PoP being down.

User impact

Users experience unexplained latency jumps, intermittent connection resets, or region-specific failures even though the CDN appears globally healthy in aggregate.

Detection

Per-region RTT changes, route-table diffs, synthetic probes from many ASNs, and unexpected traffic concentration in the wrong PoPs are the key indicators.

Recovery

Withdraw the bad Anycast route or correct the DNS steering rule, drain the overloaded region, and verify route convergence from multiple external vantage points before declaring the incident closed — routing health should be measured from outside the CDN's own view of itself, not only from internal telemetry.

Mitigations

Continuously test routing externally, keep DNS and Anycast health automation conservative, and make sure traffic engineering changes have safe rollout and rollback paths.

The central metadata/config service that holds durable configuration becomes unavailable.

What fails

New configuration, purge requests, and certificate changes can no longer be authored, validated, or distributed while the control plane is down.

User impact

Existing edge traffic is unaffected in the moment — every PoP keeps serving from its last-known-good local snapshot, since the request path never depended on a live central lookup. What breaks is anything that needs a new change: new purges, new config, new certs all queue up or fail until the control plane returns.

Detection

Control-plane health dashboards, a growing backlog of unapplied purge/config jobs, and failed config-author API calls are the primary signals.

Recovery

Restore the control plane, then reconcile the missed versions in order rather than pushing the entire accumulated backlog to every PoP at once — an uncontrolled backlog dump risks turning a recovery into its own incident.

Mitigations

Keep the data plane structurally independent of the control plane at request time, monitor backlog depth as a first-class metric, and design reconciliation to be paced rather than instantaneous.

A valid-looking but harmful cache, routing, WAF, or origin rule gets distributed to the fleet.

What fails

Unlike most failures on this list, nothing broke physically — the CDN itself becomes the source of a global outage purely through a configuration change.

User impact

Depending on the rule, this can look like sudden mass cache misses, mass purges, broken routing, or requests being wrongly blocked or wrongly allowed across every affected PoP at once.

Detection

Sudden fleet-wide behavior change correlated with a recent config deploy, error-rate spikes appearing simultaneously across unrelated regions, and automated health gates tripping during a staged rollout (if the rollout was staged).

Recovery

Roll back to the last known-good versioned config, which should be fast by design, and halt further rollout stages immediately rather than letting the bad version keep propagating to remaining PoPs.

Mitigations

Canary the change on a limited set of PoPs first, roll out region by region with automatic health gates between stages, keep every config version tracked so rollback is a version switch rather than a rebuild, and contain blast radius by design rather than relying on fast detection alone.

A new certificate, key, or TLS config fails to reach some PoPs, turns out to be invalid, or expires unexpectedly.

What fails

Affected PoPs cannot correctly terminate TLS for the impacted hostnames — this is a regional, hostname-scoped failure rather than a fleet-wide one in most cases.

User impact

Users hitting the affected PoPs see TLS handshake failures or certificate warnings; users routed elsewhere are unaffected, which can make the incident look smaller in aggregate dashboards than it actually is for the affected region.

Detection

Certificate-expiry monitoring with lead time before expiry, regional TLS error-rate spikes, and handshake failure metrics broken out by PoP rather than only in aggregate.

Recovery

Retain the prior valid certificate/config where safe rather than force-activating a bad one, stage activation of the new certificate the same way any other config change would be staged, and reconcile once the distribution path is confirmed healthy.

Mitigations

Monitor certificate expiry well ahead of time, use the same validate-version-stage-activate pipeline as any other config object, and keep secure, reliable distribution and reconciliation paths specifically for key material given its sensitivity.

The CDN caches a 404 or error response too aggressively, and continues hiding a resource that has since become available, or prolongs an upstream error longer than the outage itself lasted.

What fails

The negative-cache TTL for the affected status/path is longer than the actual duration of the underlying problem, so the cache — not the origin — becomes the reason the incident appears to continue.

User impact

Users (or bots) requesting a path that should now succeed keep receiving a cached error instead, even after origin has already recovered.

Detection

A gap between origin-side recovery signals (origin health checks look fine) and edge-side symptom signals (errors still being served) is the telltale sign — the origin thinks it's fine, but the edge disagrees, because the edge is answering from cache.

Recovery

Purge or revalidate the negatively cached entries for the affected paths, and shorten the negative-cache TTL for that content class going forward if it was set too aggressively.

Mitigations

Use short, status-aware negative-cache TTLs rather than one blanket TTL for all non-200 responses, and treat negative caching as a policy decision that trades origin protection against this specific correctness risk — not a free win.

One customer, one software release, or one event suddenly consumes a disproportionate share of a PoP's bandwidth, CPU, connections, or cache capacity.

What fails

Fleet-wide metrics can look entirely healthy while this happens, because the pressure is concentrated on one tenant and one shared resource pool rather than showing up in global averages.

User impact

Other tenants sharing that same PoP or resource pool see degraded latency or error rates for no reason connected to their own traffic — an effectively unrelated customer's spike becomes their problem too.

Detection

Per-tenant traffic and resource metrics are what actually catch this — a sudden spike in one customer's request volume, connection count, or cache footprint at a specific PoP, isolated from otherwise-normal fleet-wide numbers.

Recovery

Apply per-tenant quotas or admission control at the affected PoP, and shift the hot tenant's traffic toward capacity where it has more room, without needing to change anything about other tenants' service.

Mitigations

Per-tenant metrics as a standing requirement (not just fleet aggregates), enforced bandwidth/concurrency/cache-capacity quotas, and isolated resource pools for tenants whose traffic pattern makes shared best-effort capacity too risky.