Failure & Scale Scenarios

Upload volume suddenly increases well beyond baseline — a product launch, a regional event, a viral upload trend. Treat '10x' as illustrative, not a documented constant.

Detection: upload-session creation rate and credential-issuance rate both spike. Impact: because uploads go directly to object storage, raw byte throughput often absorbs the surge reasonably well, but several other components are not automatically safe: the upload-session API and credential-issuance path can themselves become a bottleneck under enough concurrent session creation; the underlying object-storage provider may enforce its own request-rate or quota limits; the finalize path's database writes see a proportional spike; and — usually the sharpest pressure point — the processing backlog grows because far more finalize events mean far more processing jobs enqueued in the same window. Mitigation: autoscale the upload-session/finalize path independently from playback capacity; watch provider-side quota/rate-limit headroom, not just your own server metrics; let the processing side absorb backlog gracefully (see Transcoding Backlog) rather than trying to eliminate the backlog entirely; track network egress/cost implications of the spike, not just request counts. Recovery: as the surge subsides, session/finalize load returns to baseline and the processing backlog drains at normal capacity.

Illustrative numeric trace (planning numbers, not measurements; they reuse the Complete Design estimate). Assumptions: baseline 58 uploads/s needs about 62,500 busy cores on average; the processing fleet is provisioned at 2× that average (125,000 cores); the surge is 10× for one hour; an assumed processing SLO of 30 minutes.

Upload bytes and session creation absorb the surge, but processing is where the time goes. That is why the page's advice to watch queued media minutes and oldest-job age matters: queue depth in jobs says nothing about whether the SLO is breached, and the honest response is to tell creators processing is delayed while the backlog drains.

A major creator publishes, and views for one specific video jump far above baseline within minutes. Treat '100x' as illustrative, not a documented constant.

Detection: a sharp, video-specific spike in manifest and segment request rate, concentrated on one video ID. Impact: once that video's segments are cached across many edges, the CDN absorbs most of the surge — but 'origin barely moves' is not automatically true. The moments right after publication (cold edges everywhere), any cache-key fragmentation for that content (see Cache-Key Fragmentation), a spike in manifest requests specifically (which may cache differently than segments), and regional skew in where the surge is concentrated can all still create real origin/shield pressure. Metadata reads for the video spike too, but a well-designed metadata cache absorbs the repeats; view-count writes never touch the primary database directly regardless of volume, since they flow through asynchronous aggregation. Mitigation: origin shield/tiered caching to collapse cross-edge miss fan-in during the cold-start window; request coalescing at every cache tier; origin-side rate limiting; correct cache-key design so authorization parameters or accidental headers don't fragment the cache for this exact moment. Recovery: as the video's segments finish propagating across the edge footprint, cache hit ratio climbs back toward normal and origin load returns to baseline.

Illustrative numeric trace (planning numbers, not measurements). Assumptions: one video normally has 50,000 concurrent viewers and a viral surge multiplies that by 100 to 5 million; each viewer fetches a 4-second, 3 Mbps segment (1.5 MB) every 4 seconds; during the first minute after publication edges miss 10% of segment requests and the origin shield with request coalescing absorbs 99% of those misses; once edges are warm they miss 0.1% and the shield again absorbs 99%.

The surge is a CDN story only when those ratios hold. Without the shield, the cold-start 125,000 requests/s would reach the origin directly, and cache-key fragmentation that drops the warm-phase edge hit ratio from 99.9% to 90% puts 125,000 requests/s on the shield instead of 1,250, a 100× difference; that sensitivity, not the 100× itself, is what the cold-start and cache-key mitigations protect.

Incoming processing jobs sustain a rate the worker fleet cannot match, even after autoscaling to its practical ceiling.

Detection: oldest-job age and queued-media-minutes both trend upward past their normal range — not just raw queue depth, which can be misleading (see Architecture Deep Dive -> Processing Backpressure). A two-hour 4K job and a 20-second clip are not equivalent backlog units. Impact: left unmanaged, every creator waits longer, and a priority scheme with no fairness protection can starve low-priority jobs indefinitely. Mitigation: priority-aware scheduling with an explicit fairness/aging mechanism, so lower-priority jobs still make bounded forward progress instead of starving completely; scale workers against a work-weighted signal, not raw job count; alert on SLO attainment, not just backlog size. Recovery: as worker capacity catches up or the upload surge subsides, the backlog drains — nothing already-finalized is lost, since jobs sit safely in durable storage the entire time.

A large batch of content becomes uncached simultaneously — a CDN configuration change, a mass cache purge, or a huge batch of brand-new uploads all going live at once.

Detection: a sudden, broad spike in edge miss rate across many objects at once, rather than concentrated on one popular video. Impact: without protection, this looks like a large number of edges all missing at once for content that was never cached in the first place — and edge-local request coalescing alone does not solve this, because it only collapses duplicate misses at the same edge, not the fan-in across every edge missing independently and each reaching origin. Mitigation: an origin shield/tiered cache is what actually collapses cross-edge fan-in (see Architecture Deep Dive -> CDN Cache Hierarchy); origin-side rate limiting and autoscaled origin capacity absorb the remaining burst; staggering cache-affecting changes (rolling purges rather than one global purge) avoids creating this storm in the first place. Recovery: as content re-populates across edges (through the shield layer rather than each edge hitting origin independently), miss rate returns to baseline.

A worker crashes mid-job, partway through producing a rendition.

Detection: the job's lease/visibility timeout expires without an acknowledgment. Impact: that job's output is incomplete; no other job or worker is affected. Mitigation: the queue/lease mechanism redelivers the unacknowledged job to a healthy worker; the retry writes to attempt-specific staging keys rather than overwriting any viewer-visible object, so a crash mid-write can never corrupt or half-write something a viewer or the CDN might be reading (see Architecture Deep Dive -> Safe Output Publication). Recovery: the job completes on retry; if it fails repeatedly, it moves to a dead-letter/quarantine state for investigation rather than blocking other jobs indefinitely.

Part of the object storage layer becomes degraded or unavailable.

Detection: elevated error rates or latency on origin reads/writes, ideally broken down by failure type. Impact: the blast radius depends heavily on which kind of failure this is — request/API-level degradation (throttling, elevated latency), a regional endpoint failure, a specific object becoming unavailable, a write failure during upload/publication, or replication lag between redundant copies are meaningfully different situations with different scopes. It is not accurate to assume provider replication automatically makes every one of these transparent to the platform. Mitigation: already-cached content at CDN edges is unaffected regardless, since it never needs to reach origin; for genuine origin-side failures, redundancy across independent failure domains means healthy copies can often serve traffic while the affected set recovers — but this depends on which failure category actually occurred. Recovery: replication catches the affected copies back up; durability came from redundancy, not any single storage node's health, but the recovery timeline and user-visible impact differ meaningfully by failure type.

The primary metadata database becomes unavailable.

Detection: write failures and read-path errors against the primary. Impact: new uploads can't finalize; fresh metadata writes (edits, privacy changes) are blocked. Whether playback and browsing keep working depends specifically on whether the playback-authorization and manifest-lookup path was designed to survive this — a cached-reads story only 'covers most playback' if that dependency was deliberately built to tolerate primary-database unavailability, not by default. Mitigation: a previously-provisioned read replica gets promoted to primary. Recovery: once failover completes, writes resume — but replica promotion is not instantaneous, and it carries real replication-lag/RPO implications: whatever wasn't yet replicated to the promoted replica at the moment of failure is genuinely at risk of being lost or requiring reconciliation.

An entire origin region becomes unreachable.

Detection: health checks across that region's services fail broadly. Impact: new uploads and processing local to that region fail; already-cached CDN content for that region's viewers may be unaffected for cached objects, but a cache miss on content whose only copy lives in the failed region cannot be filled no matter how the CDN is configured. Mitigation: global routing can redirect new uploads/processing to another healthy region only where the metadata write-ownership model, replication, and fencing actually support that redirection for the affected video/partition (see Architecture Deep Dive -> Multi-Region Ownership) — it is not automatic. An in-progress multipart upload tied to the failed region's storage session generally cannot seamlessly continue elsewhere unless the upload architecture explicitly supports that. Recovery: once the region returns, it resumes serving local traffic; any backlog that accumulated during the outage drains through normal processing, and any ownership/fencing state used during failover needs to be reconciled before the recovered region resumes writes.

A major popularity spike, left unmitigated, threatens to propagate pressure backward through the whole system.

Detection: rising latency and error rate that starts at one component and spreads to callers of that component. Impact: without protection, retries pile on top of retries as clients and internal services time out and retry against an increasingly overloaded dependency; connection pools and threads get held open waiting; latency propagates backward from wherever the pressure originated toward every service in the request path — a single video's popularity becoming a platform-wide problem. Mitigation: the CDN and origin shield absorb almost all read traffic before it reaches the metadata tier; the metadata cache absorbs repeated reads that do get through; asynchronous view aggregation means a view spike never touches the database directly; and timeouts, circuit breakers, retry budgets with jitter, and load shedding protect whatever is still under real pressure, converting a potential outage into, at worst, a brief, contained degradation. Recovery: as retry pressure is shed and backoff/jitter spreads retries out over time rather than in lockstep, the system drains back to normal load.

One part/chunk of a multipart upload fails to land — a dropped connection, a transient storage error.

Detection: the client's per-part acknowledgment never arrives, or the part upload returns an error. Impact: none beyond that one part, if the client retries correctly — the parts that already landed are untouched. Mitigation: retry only the failed part, using its stable part identity; do not restart the whole upload. Recovery: once the retried part lands and is acknowledged, the upload proceeds normally toward finalize. See Architecture Deep Dive -> Multipart & Resumable Upload.

A client abandons an upload for long enough that the session passes its expiration.

Detection: a client attempts to upload a part or finalize against a session that has already expired. Impact: without clear handling, the client gets a confusing error and doesn't know whether to retry or start over. Mitigation: the server returns an explicit, unambiguous 'session expired, start a new upload' response rather than a generic failure — restart/renew semantics should be a first-class, well-defined response, not an inferred edge case. Recovery: an automated cleanup process removes any orphaned parts, temporary objects, and the stale session record associated with the expired session, so abandoned attempts don't quietly accumulate storage cost.

Finalization commits successfully server-side, but the client never receives the response — a dropped connection, a timeout on the client's end.

Detection: the client, having received no response, retries the finalize call. Impact: without idempotent finalize, the retry could create a duplicate video record and schedule a second, independent processing workflow for the same source. Mitigation: finalize must be idempotent — a retried call for the same upload_id discovers the already-finalized result and returns it, rather than re-running finalize logic from scratch. See Architecture Deep Dive -> Upload Finalization. Recovery: the client's retry returns the same finalized video/upload it would have gotten from the original call; no duplicate is created.

The metadata write that marks an upload finalized commits successfully, but the corresponding processing-job publish to the queue fails or never happens.

Detection: a finalized video with no corresponding processing job appears, or an outbox/job-dispatch lag metric grows. Impact: without a durable handoff mechanism, this finalized upload silently never gets processed — the creator sees 'processing…' forever. Mitigation: a transactional outbox / durable job record repairs this handoff: the job row is committed in the same transaction as the finalize write, so its existence is guaranteed regardless of whether the initial publish attempt succeeded; a separate dispatcher retries publishing unpublished rows until the queue acknowledges them. See Architecture Deep Dive -> Durable Processing Handoff. Recovery: the dispatcher eventually publishes the job; processing proceeds normally, just later than if the first publish attempt had succeeded.

The same processing workflow is delivered to workers twice — a natural consequence of the durable, at-least-once handoff described above.

Detection: two workers (or the same worker twice) pick up the same processing_job_id. Impact: without safe handling, this risks two independent sets of output being produced and racing to publish, or wasted duplicate compute. Mitigation: duplicate execution is safe by construction — both attempts write to their own attempt-specific staging keys, so they can't corrupt each other's output; the safe-publication boundary means only one validated attempt's output actually gets published as the active version, and a second attempt finishing after publication simply becomes a no-op or gets discarded per policy. Recovery: no data corruption occurs regardless of which attempt 'wins'; the wasted compute from the duplicate is a cost, not a correctness issue. See Architecture Deep Dive -> Safe Output Publication.

A worker's lease/visibility timeout expires — triggering redelivery to a second worker — but the original worker is actually still alive and still processing, just slow.

Detection: two workers are simultaneously active on the same job after a lease expiry; both may eventually attempt to finish and publish. Impact: this is exactly the overlapping-attempt scenario safe publication is designed for — if handled naively (both writing to the same viewer-visible key), this could corrupt output or produce inconsistent renditions. Mitigation: each attempt writes to its own versioned staging path; validation and the publication boundary ensure only one complete, validated attempt's output becomes active, regardless of which one happens to finish first or last. Recovery: whichever attempt validates and publishes first wins; the other's staging artifacts are garbage-collected later as an abandoned attempt.

The uploaded original is corrupted or unreadable — a bad upload, a truncated transfer that still passed multipart completion.

Detection: the media-probe stage fails to parse the file, or produces implausible characteristics. Impact: without a clear terminal path, the job could retry indefinitely against a source that will never succeed. Mitigation: probe/validation rejects the source outright, or routes it to a quarantine state for investigation, with a bounded number of retries at most — not infinite retry. The creator should receive a clear, actionable failure rather than an indefinite 'processing…' state. Recovery: the creator is prompted to re-upload; the corrupted source and any partial processing artifacts are cleaned up.

The uploaded original uses a codec or container the processing pipeline doesn't support.

Detection: the probe stage identifies the source codec/container and finds no matching decode path. Impact: without a defined policy, the job could fail repeatedly with no clear resolution for the creator. Mitigation: route to a terminal-failure state with a clear message, or to an alternate processing path if the platform maintains one for specific known formats — but not an indefinite retry loop against a source the pipeline fundamentally cannot decode. Recovery: the creator is informed and can re-upload in a supported format if no alternate path exists.

In a processing DAG with multiple parallel rendition transcodes, one representation fails while the rest complete successfully.

Detection: per-representation state (see Architecture Deep Dive -> Processing DAG) shows one representation in a failed state while its siblings succeeded. Impact: the platform has to decide, as an explicit policy, whether to publish a reduced ladder (missing the failed rendition) or treat this as an overall failure because a mandatory minimum rendition set wasn't met. Mitigation: define this policy explicitly ahead of time rather than deciding ad hoc per incident — for example, some renditions might be considered mandatory (a baseline quality every device needs) while others are best-effort. Recovery: the failed representation can be retried independently without redoing the ones that already succeeded, thanks to per-representation state tracking.

An incomplete or still-in-progress set of segments/renditions is somehow reachable by a viewer before processing has actually finished.

Detection: a playback request succeeds against a video whose processing/validation state was not actually complete. Impact: the viewer could receive a broken manifest, missing segments, or a rendition that isn't actually ready — a direct violation of the requirement that publication only exposes complete, validated output. Mitigation: this should be structurally impossible if the publication boundary is enforced correctly — playback should only ever resolve against the active, published version pointer, and that pointer is only updated after validation succeeds (see Architecture Deep Dive -> Safe Output Publication). If this occurs, it indicates a bypass of the publication boundary somewhere in the path. Recovery: treat this as a bug in the publication boundary itself and fix the bypass; in the moment, roll back the active-version pointer to the last known-good version if one exists.

A manifest referencing a set of segments becomes reachable before all of those segments have finished being written and validated.

Detection: a player requests a segment listed in the manifest and gets a 404 or incomplete response (see Segment 404 below) shortly after a video was marked ready. Impact: playback fails or stalls for early viewers. Mitigation: manifest publication must happen strictly after its full segment set is written and validated, as part of the same publication boundary — never published speculatively or ahead of its dependencies. See Architecture Deep Dive -> Manifest & Packaging and Safe Output Publication. Recovery: once the underlying cause (a race in the publish sequencing) is fixed, this should not recur; existing affected videos need their publication step re-run correctly.

A transcode output passes validation but turns out to be genuinely broken in a way validation didn't catch — visual corruption, audio/video sync issues, or similar.

Detection: creator or viewer reports, or a downstream quality-monitoring signal, flags the published version as broken. Impact: viewers requesting that version get a degraded or broken playback experience. Mitigation: because outputs are versioned and the previously-active version was never deleted immediately (see Architecture Deep Dive -> Storage Lifecycle), the active-version pointer can be rolled back to the last known-good version quickly. Recovery: roll back the active manifest/version pointer; quarantine the bad version for investigation rather than deleting it outright, so the failure can be diagnosed.

A video is reprocessed (a codec migration, a re-encode at a new rendition ladder) and the reprocessing workflow fails partway through.

Detection: the reprocessing job's stage state shows a failure before the new version validated. Impact: none for current viewers, if the publication boundary is respected — the previous version remains the active, playable version throughout. Mitigation: never switch the active-version pointer until the new version has fully validated; a failed reprocess simply leaves the old version in place. Recovery: the reprocessing job is retried (using its own attempt-scoped staging, same as any other processing job) without any interruption to currently-playable content.

The processing queue/broker itself becomes unavailable.

Detection: the dispatcher's publish attempts start failing broadly. Impact: new processing jobs can't be dispatched to workers — but finalized uploads are not lost, because the durable job record was committed in the same transaction as the finalize write, independent of the queue's health. Mitigation: the dispatcher retries publishing until the queue recovers; finalized-but-undispatched uploads remain durably discoverable the entire time. Recovery: once the queue recovers, the dispatcher drains its backlog of unpublished job rows; processing resumes without any upload having been silently dropped.

The dead-letter/quarantine queue for repeatedly-failing processing jobs starts growing faster than normal.

Detection: DLQ growth-rate alert. Impact: if left uninvestigated, this can mask a systemic issue — a new codec variant the pipeline can't handle, a regression in a specific transcode profile, a bad deploy — affecting many creators' uploads at once, not isolated corrupt files. Mitigation: alert on DLQ growth rate, not just absolute size, and investigate for a systemic cause whenever growth accelerates rather than assuming every entry is an isolated bad source file. Recovery: once the systemic cause (if any) is fixed, affected jobs can be replayed from quarantine rather than requiring creators to re-upload.

A single CDN edge location becomes unhealthy.

Detection: elevated error rate/latency from that specific edge. Impact: viewers routed to that edge see degraded playback. Mitigation: reroute to an alternate healthy edge/PoP where the CDN's routing supports it — but this should not be assumed to unconditionally bypass all the way to origin as the fallback; that risks the exact unrestricted-fallback overload this module warns against elsewhere. Recovery: once the edge recovers or is removed from rotation, traffic returns to a normal routing pattern.

The CDN provider itself has a broad outage — an advanced, higher-severity scenario.

Detection: widespread failure across many/most edges simultaneously, correlated with a known provider incident. Impact: without a fallback, the platform's entire edge-caching layer is unavailable at once — a direct-to-origin fallback at this scale is exactly the unrestricted-fallback risk called out earlier and can overload a finite origin. Mitigation: an alternate CDN provider (multi-CDN, an advanced option — see Trade-offs & Decisions -> Single CDN vs. Multi-CDN) or a controlled, deliberately degraded mode (bounded origin fallback with rate limiting and load shedding) depending on the platform's actual availability requirements. Recovery: once the provider incident resolves, traffic returns to normal CDN routing.

The origin shield/tiered-cache layer itself becomes unavailable.

Detection: shield-layer health checks fail; edge misses that would normally resolve at the shield start reaching origin directly. Impact: without the shield's fan-in collapsing, origin can face the full cross-edge miss amplification the shield exists to prevent — especially dangerous if this coincides with a viral event. Mitigation: a defined fallback hierarchy (an alternate shield instance/region, or a bounded, rate-limited direct-to-origin path) rather than unlimited direct origin access. Recovery: once the shield recovers, cross-edge fan-in collapses back through it and origin load returns to normal.

A signed authorization query parameter, an accidentally-varying header, or a cookie ends up as part of the CDN cache key for otherwise-identical content.

Detection: cache hit ratio drops sharply for content that should be highly cacheable, alongside a rise in origin request rate for objects that are supposedly popular and should be well-cached. Impact: effectively every viewer gets their own unique cache entry for the same underlying segment, destroying the hit ratio and pushing far more traffic to the shield/origin than the actual content popularity would suggest. Mitigation: cache-key configuration must deliberately exclude authorization parameters (or normalize them) from the cache key while still enforcing them at the authorization layer — see Architecture Deep Dive -> CDN Cache Key. Recovery: correcting the cache-key configuration restores the hit ratio; no data was lost, but cost and load during the fragmentation window can be significant.

A player fetches a manifest that no longer matches the actual current segment set — a version mismatch between manifest and segments.

Detection: segment requests derived from the manifest return errors or mismatched content. Impact: playback fails or shows visibly wrong content. Mitigation: versioned publication is what prevents this by design — a manifest for version N only ever references version N's segments, which never change once published; the failure mode this scenario describes generally indicates a bypass of that versioning discipline (e.g., a manifest cached longer than its version's validity, or a manifest and segment set updated non-atomically). Recovery: ensure manifest and segment publication happen together as one atomic version switch, and audit for any caching layer holding a manifest past its version's validity.

A player requests a segment listed in the manifest and gets a not-found response.

Detection: elevated segment 4xx rate, ideally attributable to specific videos/versions. Impact: playback stalls or errors for the affected segment. Mitigation: the player retries the request and/or falls back to an alternate representation where possible, rather than failing the entire playback session on one segment error. On the platform side, this should trigger investigation into whether it's a publication/origin/CDN mismatch (see Manifest Published Before Segments and Stale Manifest above) rather than being dismissed as a one-off. Recovery: once the underlying mismatch is fixed, segment 4xx rate returns to baseline; in the moment, the player's retry/fallback logic keeps the specific viewer's session going where possible.

A viewer's network connection degrades or drops entirely mid-playback.

Detection: the player's throughput estimate falls sharply, or segment fetches start timing out. Impact: the playback buffer drains as new segments can't be fetched in time. Mitigation: the ABR logic reduces requested quality for subsequent segments as buffer health degrades, and retries the current segment fetch — a graceful quality reduction is strongly preferred over an immediate stall. Recovery: if the network recovers before the buffer fully drains, playback continues at a lower quality and climbs back up; if the buffer does drain, the player surfaces a playback/rebuffering event rather than failing silently.

A viewer's available bandwidth drops sharply and suddenly during playback — a much steeper case than gradual degradation.

Detection: a sudden, large drop in the player's measured throughput. Impact: without a fast response, the player continues requesting a rendition the network can no longer sustain, draining the buffer toward a stall. Mitigation: the player should lower the requested bitrate proactively, before a stall actually occurs where possible — reacting to the throughput drop rather than waiting for the buffer to empty. Recovery: as bandwidth recovers, the player climbs back up through the ladder, generally more conservatively than it dropped, to avoid immediately oscillating back into a rendition the network can't yet sustain.

The player switches representations back and forth rapidly, producing a visibly unstable quality experience.

Detection: an unusually high rate of representation switches per minute for a given playback session. Impact: a jittery, distracting viewing experience — arguably worse for perceived quality than staying at a slightly lower, stable rendition. Mitigation: smoothing/hysteresis logic — requiring a sustained trend before switching up or down, weighting buffer occupancy over instantaneous throughput noise, and avoiding switching on every small fluctuation. Recovery: once smoothing logic is tuned correctly, switch frequency drops back to only genuinely warranted quality changes.

A video's visibility changes from public to private, but a cached metadata/authorization result still reflects the old, public state.

Detection: a playback authorization succeeds for a video that should now be private, correlated with a recent visibility change. Impact: a real access-control failure, not just a minor staleness inconvenience — someone can watch content the owner just restricted. Mitigation: a privacy-restricting change must not remain accessible solely because an ordinary metadata cache TTL hasn't expired yet — this class of change needs explicit, immediate invalidation of any cached authorization result, not just the normal cache refresh cycle. See Architecture Deep Dive -> Playback Authorization. Recovery: invalidate the specific cached entries immediately upon the privacy change, and audit how long the exposure window was if it occurred.

A video is deleted, but its segments remain cached at CDN edges and are still technically retrievable by anyone with the URL.

Detection: a request for a deleted video's segment succeeds against a CDN edge after deletion was recorded in metadata. Impact: the platform's authoritative state says the video is gone, but the content is still reachable at the edge — inconsistent with the deletion the creator/platform requested. Mitigation: the deletion workflow should stop authorization/discovery promptly (an immediate, cheap step) and perform CDN purge/version revocation according to policy — recognizing this second step may not be instantaneous globally. Be explicit and honest about that gap rather than promising instant global removal. See Architecture Deep Dive -> Video Deletion. Recovery: CDN purge/invalidation eventually removes the cached copies; the deletion workflow tracks completion so this isn't left indefinitely inconsistent.

A video is deleted or made private, but it still briefly appears in search results.

Detection: a search query surfaces a video that authoritative metadata already marks deleted/private. Impact: a discoverability inconsistency — but importantly not necessarily an access-control failure, as long as playback authorization is the final, independently-enforced gate (see Playback Authorization). Mitigation: prioritize deletion/privacy index updates ahead of the ordinary ranking-freshness queue, since the cost of a stale result here is different in kind. Regardless of index freshness, playback authorization must independently refuse access — the search index showing something briefly is a UX gap, not the actual security boundary. Recovery: once the prioritized index update propagates, the entry disappears from search results.

The same view event is delivered to the aggregation pipeline more than once — an expected consequence of at-least-once delivery somewhere in the telemetry path.

Detection: two events with the same stable event identity arrive within the aggregation window. Impact: without deduplication, the view count would be inflated. Mitigation: bounded, identity-based deduplication within the aggregation window suppresses the duplicate. Recovery: none needed for events within the dedupe window — this is expected, handled behavior; see View Event Loss below for the harder case of a duplicate arriving after the window has already closed.

A view event is dropped somewhere in the telemetry path and never reaches the aggregation pipeline at all.

Detection: generally not directly detectable per-event; inferred from aggregate telemetry-pipeline health metrics (see Telemetry Pipeline Outage below) rather than any single missing view being individually noticed. Impact: the displayed view count undercounts by a small, generally imperceptible amount for a user-facing popularity signal. Mitigation: where the product requirement explicitly allows bounded approximation for this counter (the common case for a user-facing display count), this is an acceptable, designed-for trade-off — the count should not be presented as exact if this can occur. Recovery: none needed if approximation is the accepted policy; if exactness were required, a fundamentally stricter delivery guarantee would be needed for the whole telemetry path, which is out of scope for a display counter.

A viral video's view events all route to the same aggregation key/partition, overwhelming that single reducer even though writes are already batched.

Detection: one partition/reducer shows dramatically higher throughput and/or growing lag compared to its peers. Impact: that video's view count updates lag further behind than other videos', and in a severe case the hot partition could affect the aggregation pipeline's overall health. Mitigation: shard or hierarchically aggregate the hot key specifically — local/partitioned partial aggregates merged later, or dynamic re-partitioning for detected hot keys. See Architecture Deep Dive -> Hot View Counter. Recovery: once the hot-key mitigation is active, that video's aggregation lag returns toward normal even while it remains disproportionately popular.

The playback-telemetry/view-event pipeline becomes unavailable entirely.

Detection: view-event ingestion rate drops to zero or errors broadly. Impact: view counts and other playback analytics stop updating. Mitigation: playback itself must continue uninterrupted — telemetry is not on the critical path for actually serving video, and its importance level (buffer events locally and retry later vs. accept some data loss) should be an explicit design decision made ahead of time, not discovered during the outage. Recovery: once the pipeline recovers, buffered events (if the client/edge buffers them) flush and aggregation catches up; any events that were designed to be dropped rather than buffered are an accepted, bounded loss.

A remote region's replicated metadata copy falls behind the authoritative writer region.

Detection: replication-lag metric between the authoritative region and a read-replica region exceeds its normal range. Impact: a remote region can show a stale title, description, or — more seriously — a stale visibility/privacy state, unless privacy-sensitive changes are handled with a stricter propagation policy than ordinary field updates. Mitigation: treat privacy/deletion-relevant fields with a higher-urgency replication or verification path than ordinary metadata (consistent with how Search Index Stale Delete and Metadata Cache Stale Privacy are handled) — an ordinary title edit tolerating replication lag is a very different risk than a stale 'this is still public' read. Recovery: once replication catches up, remote reads reflect current state; any privacy-sensitive access that occurred during the lag window needs the same audit treatment as any other authorization gap.

A network partition separates regions in a way that could let more than one region believe it holds authoritative write ownership for the same video/partition.

Detection: conflicting metadata versions for the same video appear after the partition heals, or a fencing check fails during an attempted failover. Impact: without fencing, both regions could accept writes for the same video during the partition, producing genuinely conflicting updates that are difficult to reconcile automatically after the fact. Mitigation: fence the previous authoritative writer before any other region is allowed to accept writes for that video/partition — an explicit ownership handoff, never an automatic dual-accept. See Architecture Deep Dive -> Multi-Region Ownership. Recovery: once ownership is unambiguous again, any conflicting writes that did occur during the partition need manual or policy-driven reconciliation; this is exactly the failure mode a correct fencing design prevents from happening in the first place.

A playback cache miss needs to fetch from origin, but the region routing sends it toward a region that never received a replicated copy of this specific object.

Detection: an origin fetch in the routed-to region returns not-found for an object that does exist elsewhere. Impact: playback fails for that viewer, even though the content genuinely exists somewhere in the platform. Mitigation: this is a direct consequence of selective replication (see Trade-offs & Decisions -> Replicate All Content vs. Selective Replication) — a cache miss cannot be silently satisfied by failing over to a region that was never given a copy of that specific object. Either the routing needs to know which regions actually hold a given object and route misses accordingly, or the platform accepts this as a real availability trade-off for content that wasn't replicated broadly. Recovery: fetch from a region that does hold the object, at the cost of higher latency for that request; this is not something a regional failover can fix if no replicated copy exists anywhere reachable.

The worker executing a video's deletion workflow crashes partway through — after some steps (say, stopping new authorization) but before others (purging CDN cache, deleting derived artifacts).

Detection: a deletion workflow's tracked steps show an incomplete state past its expected completion window. Impact: the video is in an inconsistent state — authorization already stopped, but artifacts and cache entries not yet cleaned up. Mitigation: the deletion workflow must be retryable and observable, tracking which steps completed so a retry resumes rather than restarts, and an operator can see exactly how far any given deletion got. Recovery: the workflow is retried (automatically or via an operational sweep for stuck deletions) until all steps report complete; this is the same durable-workflow discipline used elsewhere in the design, applied to deletion.

Spend spikes well beyond expected levels — not a correctness failure, but an operational one worth treating with the same seriousness.

Detection: cost/spend monitoring on egress, compute, and cache-miss-driven origin traffic crosses an alert threshold. Impact: viral traffic driving CDN egress, a cache-key fragmentation bug driving unnecessary origin egress, an accidental mass-reprocessing trigger, or a codec-ladder expansion that wasn't cost-modeled can each independently create a real cost spike. Mitigation: monitor egress, compute, and cache hit ratio as first-class operational signals, not just an after-the-fact billing review — several of the correctness fixes elsewhere in this page (correct cache-key design, bounded retry, safe publication preventing wasted reprocessing) are also the cost mitigations for this scenario. Recovery: once the specific driver is identified (a bug vs. genuine organic virality), address it directly — a cache-key bug gets fixed; genuine viral organic growth is a capacity-planning conversation, not an incident.

Scenario classes at a glance