Recommendation System — Failure & Scale Scenarios

One show, product, or topic suddenly becomes globally hot and the whole fleet wants the same recommendations at once.

What fails

The Trending/Popularity Aggregator, hot caches, and candidate-source shards for the viral topic experience extreme skew. The challenge is less total traffic than concentration on a tiny set of keys — and the same hot item can also dominate the merged candidate pool from Candidate Merge & Deduplication if its source is not quota-bounded.

User impact

If the system is prepared, users still get fast relevant results because the hot topic is cached and pre-aggregated well. If not, latency rises as repeated candidate retrieval and ranking work pile up on the same assets, and the feed can start feeling like the same item everywhere.

Detection

Watch for a sharp rise in top-key concentration, hot-category cache churn, repeated requests for the same trending source across many regions, and a rising single-source contribution share in merged candidate pools.

Recovery

Pin the hottest popularity lists in cache, widen fan-out capacity for that trend shard, and temporarily bias responses toward precomputed trending assets rather than full recomputation — without removing the per-source quota that keeps one item from crowding out everything else.

Mitigations

Precompute regional trending lists continuously, use request coalescing on hot recommendation keys, keep the popularity pipeline horizontally scalable, and enforce a hard cap on how much of any response a single trending item can occupy.

A small set of accounts drive extremely high recommendation traffic compared with the rest of the population — celebrity users, but just as often an automated client, a refresh-loop bug, or a high-volume enterprise tenant.

What fails

Personalized caches, profile lookups, and session-state updates for those accounts become disproportionately hot. One account's activity — human or automated — can create repeated invalidations and ranking work.

User impact

The hot accounts themselves may see latency spikes or repetitive fallback results. Nearby infrastructure may also suffer if their keys are not isolated well.

Detection

Per-account request-rate outliers, repeated cache invalidations on the same key families, and unusual skew in profile-read traffic reveal the pattern — cross-checked against client type, since a real celebrity user and a broken polling client look similar in raw traffic shape.

Recovery

Introduce stronger per-account rate limiting and request coalescing first; isolating extremely hot accounts into separate shards helps with genuinely legitimate heavy usage but does nothing for a misbehaving client that a rate limit would stop outright.

Mitigations

Rate-limit and coalesce at the account level by default, shard intelligently for legitimate heavy accounts, use short-lived session overlays instead of rewriting full profiles constantly, and keep hot-account analytics so the serving fleet is not surprised by skew.

The Ranking Service gets slower under load and begins missing its latency budget.

What fails

Any of several dependencies can be the cause: feature-hydration latency, model-server saturation (CPU/accelerator/memory), GC or runtime pauses, or simply a growth in candidate count feeding the ranker faster than capacity grew.

User impact

Users may receive stale cached lists, less personalized popularity fallbacks, or fewer recommendation modules because the system skips expensive ranking work.

Detection

p95 and p99 ranking latency, rising timeout counts, higher fallback rate, model-server queueing time, and candidate-count trends are the clearest signals — correlating them narrows down which dependency is actually responsible.

Recovery

Reduce candidate count, shed optional feature lookups, switch to a cheaper fallback model or pre-ranker, route more traffic to cached outputs, and scale the ranking fleet horizontally until latency stabilizes.

Mitigations

Enforce tight per-stage budgets, keep ranking inputs compact via batched feature hydration, maintain a cheaper fallback model as a standing option, and rehearse degraded modes (load shedding) that preserve useful results without full scoring depth.

The Feature Store becomes unavailable, returns stale data, returns partially missing fields, or serves the wrong schema version — four different failure modes that need different responses.

What fails

Unavailable means no reads succeed at all; stale means reads succeed but reflect an old snapshot; partial-missing means some fields in the bundle are absent; wrong-version means the bundle does not match what the calling model expects. Candidate Generation and Ranking Service need to recognize which one they are facing.

User impact

Personalization quality drops, with severity depending on the failure mode — a stale snapshot for a preference feature is a minor quality dip, but a stale snapshot for an eligibility or privacy-relevant feature is a correctness problem, not just a relevance one.

Detection

Feature-read error rates, stale-feature age, missing-field rate per feature name, and version-mismatch counters together distinguish which failure mode is occurring.

Recovery

Switch to the last known-good feature snapshot for the general case, but do not treat a stale snapshot as automatically safe for eligibility or privacy-sensitive features — those need their own current-state recheck regardless of feature-store health (see Eligibility Filtering).

Mitigations

Replicate the online store, version features explicitly, keep fallback feature bundles cached close to ranking, decouple page availability from perfect feature freshness, and hard-code the rule that eligibility checks never rely solely on a feature-store snapshot.

The ingestion or streaming pipeline falls behind and fresh user actions stop reaching downstream systems quickly.

What fails

The Trending/Popularity Aggregator, Real-Time Signal Processor, and profile builders all begin operating on older-than-expected data — measured as event-time lag (how old the data is), which is different from simple processing throughput.

User impact

Recommendations feel stale: recently watched topics do not influence the next page, fresh viral content rises slowly, and hides or skips take too long to matter.

Detection

Consumer-specific lag SLOs (not one global number), oldest-unprocessed-event age per consumer, delayed popularity updates, and growing mismatch between session actions and recommended content reveal the problem.

Recovery

Prioritize the most latency-sensitive consumers first, apply backpressure or pause nonessential batch consumers, and use session-local caches so the newest actions still influence immediate requests — but recognize that pausing consumers does not help if the broker or storage layer itself is overloaded; that requires capacity relief, not consumer-side throttling.

Mitigations

Separate critical streaming consumers from heavy offline jobs, provision headroom in the event path itself (not just at the consumer level), and keep clear, per-consumer freshness SLOs instead of one pipeline-wide freshness target.

A large number of new users or new items enter the system with almost no behavior history.

What fails

Profile-driven retrieval has little to work with, and collaborative or embedding-based sources may under-supply candidates for both new users and new items, since neither has accumulated interaction history yet.

User impact

New users see bland or repetitive recommendations, and new items struggle to get exposure unless they enter through trending, metadata, or reserved exploration slots.

Detection

Watch for low candidate counts on new accounts, weak engagement from zero-history cohorts, and a new-item exposure rate near zero for recently added catalog items.

Recovery

Increase reliance on regional trending, category-popular, metadata-based similarity, and reserved exploration slots until richer behavior accumulates for the affected users and items.

Mitigations

Design explicit cold-start branches for both users and items from day one (see Cold Start: New Users and Cold Start: New Items), and track new-item exposure and coverage as an ongoing metric, not a one-time launch check.

The system keeps serving recommendations that were once correct enough but are now outdated for the user's current context — though "outdated" covers two very different severities.

What fails

Cache TTLs, profile refresh cadence, or real-time invalidation rules are too loose. It matters a great deal which kind of staleness this is: benign preference staleness (interests shifted slightly) versus correctness-critical staleness (a deleted item, an out-of-stock product, or a privacy/eligibility flag that changed).

User impact

Benign staleness shows up as repeated or slightly-off items — annoying but harmless. Correctness-critical staleness shows up as recommending something the user cannot actually get, or shouldn't be shown at all — a trust failure, not just a relevance one.

Detection

Track repeated-item rate and cache age for the benign case; track post-response availability/eligibility-mismatch rate separately for the correctness-critical case, since the two need different alert thresholds and different urgency.

Recovery

For benign staleness, tighten cache invalidation on strong signals and lower TTLs. For correctness-critical staleness, treat it as a priority incident — the fix is not a shorter TTL, it is confirming the eligibility recheck is actually querying current state (see Eligibility Filtering).

Mitigations

Keep explicit, surface-specific TTL policy for the benign case, and never let correctness-critical fields (availability, deletion, privacy/eligibility flags) depend on a cache TTL at all — they belong in the mandatory final recheck regardless of what any cache says.

The feedback loop starts over-amplifying what already performs well, creating echo chambers or runaway popularity bias.

What fails

The ranking and retrieval system rewards immediate engagement too strongly, so already popular topics monopolize candidate pools and the same interests get reinforced repeatedly — a specific case of exposure bias, where what the system chose to show shapes what it later learns from.

User impact

Users see less diversity and less discovery. Niche interests disappear, catalog coverage shrinks, and the product can feel manipulative or stale even when engagement metrics look superficially healthy.

Detection

Monitor category concentration, creator repetition, declining catalog coverage, reduced exploration-slot exposure, candidate-source domination (see Candidate Source Dominates), and sharp divergence between short-term engagement and long-term satisfaction metrics. Because this is a gradual drift, a well-designed experiment specifically comparing exploration levels is often what actually surfaces it.

Recovery

Increase diversity controls, widen exploration traffic, lower the weight of the hottest popularity features temporarily, and inspect whether one candidate source is dominating candidate pools.

Mitigations

Treat bias control as a serving concern: diversity rules, exploration budgets, and coverage monitoring belong in the online architecture (see Exploration vs. Exploitation), not only in offline analysis — and impression/exposure logging is what makes the bias measurable at all.

The cache hierarchy loses effectiveness or correctness at exactly the moment the serving fleet needs it most.

What fails

The Recommendation Cache, profile caches, or hot item-metadata caches experience poor hit rate, invalidation bugs, or stampedes on hot keys — and in more serious cases, a key-scoping bug or version mismatch (see Cross-User Cache Leak) can serve wrong or stale-eligibility data outright.

User impact

Latency rises, ranking load increases sharply, and users may see either stale lists or more generic fallbacks as deeper services become overloaded — or, in the key-scoping failure mode, another user's data entirely.

Detection

Watch cache-hit rate, cache age, refill concurrency, ranking QPS spikes, sudden growth in deep-store reads after a cache event, and cache-key collision audits for the more serious correctness failure modes.

Recovery

Enable request coalescing, restore the last good cache policy, pin the most important fallback lists, and slow cache invalidation churn until deeper services recover. If a key-scoping or version-mismatch bug is found, invalidate the affected entries immediately rather than waiting for natural TTL expiry.

Mitigations

Use layered caches, careful cache-key scoping treated as a correctness contract, soft TTL refresh, and per-surface invalidation rules so one hot key does not collapse the whole personalization path — and never let a cached list bypass the final eligibility recheck.

A producer changes an event's schema — a renamed field, a changed type, a new required field — without coordinating with consumers.

What fails

Downstream consumers (the profile builder, the popularity aggregator, feature pipelines) either crash, silently drop the new field, or misinterpret it under the old schema.

User impact

Depending on which consumer breaks, users see stale profiles, wrong feature values, or missing signals — often with no visible error on the page itself.

Detection

Schema-validation rejection counters, a spike in dead-lettered events, or a consumer's parsed-field-null-rate suddenly rising.

Recovery

Roll back the producer to the previous schema version or fast-follow with a compatible consumer update, then replay the affected window from the durable log once consumers are fixed.

Mitigations

Version the event schema explicitly, require backward-compatible changes (additive fields, no silent type changes), and validate schema compatibility in CI before a producer deploys (see Event Contract & Ingestion).

The same user action is delivered more than once, from client retries, at-least-once delivery, or a replay.

What fails

A consumer that is not idempotent double-counts the event — inflating a popularity counter, double-appending to bounded history, or duplicating an impression log entry.

User impact

Recommendations can over-weight one action, and quality metrics computed from the affected logs become unreliable.

Detection

Duplicate-event-ID rate, unexpected counter growth relative to unique-user activity, and reconciliation checks against a sampled ground truth.

Recovery

Deduplicate using the stable event ID against a bounded recent-ID window, and recompute any aggregate that was corrupted from the durable log.

Mitigations

Make every event consumer idempotent by construction, keyed on event ID, rather than relying on producers to guarantee exactly-once delivery (see Event Replay & Idempotency).

The client fails to report that an item was actually rendered, even though the server logged it as returned.

What fails

The gap between "returned" and "rendered/exposed" in the impression log goes unnoticed, so the system believes it showed something the user never saw.

User impact

No direct user-facing impact, but every downstream measurement — click-through rate, exploration coverage, experiment results — is quietly biased.

Detection

A persistent gap between returned-count and rendered-count per surface or client version, especially concentrated on one app version or network condition.

Recovery

Treat "returned" as a lower-confidence signal until the rendered event arrives or a timeout confirms it did not, and exclude affected sessions from quality metrics if the client-reporting gap is severe.

Mitigations

Make rendered-impression reporting a first-class, monitored client contract, not an optional analytics nice-to-have (see Impression & Exposure Logging).

Events arrive well after the window they belong to has already been aggregated.

What fails

A late event either gets dropped, or is folded into the wrong (current) window instead of the one it actually belongs to.

User impact

Aggregates like "last 24h interest" can be slightly wrong for affected users — usually not severe unless late arrivals are frequent.

Detection

Watermark-lag metrics, and a rising rate of events arriving after their window's grace period closes.

Recovery

Reprocess the affected window once late events are accounted for, if the aggregate is important enough to correct; otherwise accept the bounded inaccuracy.

Mitigations

Define an explicit grace period and late-arrival policy per aggregate, keyed on event-time rather than processing-time (see Event-Time vs. Processing-Time).

A client bug or malicious actor floods the event pipeline with malformed or excessive events.

What fails

Ingestion capacity is consumed by low-value or invalid events, starving legitimate traffic and downstream consumers.

User impact

Real users may see delayed profile updates or degraded freshness while the pipeline drains the flood.

Detection

A spike in schema-validation rejections, per-client event-rate outliers, and rising ingestion queue depth.

Recovery

Rate-limit or block the offending client/source, drain the backlog with prioritization for legitimate traffic, and quarantine the bad batch rather than let it reach aggregation jobs.

Mitigations

Per-client rate limits at the Event Collector, schema validation at ingestion rather than downstream, and anomaly detection on event-rate-per-source.

A bug in the profile-building job writes incorrect or malformed data into a user's durable profile.

What fails

The corrupted profile record produces nonsensical candidate retrieval or ranking features for that user.

User impact

Affected users see irrelevant or broken recommendations, potentially with degraded page behavior if the malformed record breaks a downstream parser.

Detection

Schema/shape validation on profile writes, anomaly checks on profile field distributions, and user-reported irrelevant-recommendation spikes.

Recovery

Roll back to the last known-good profile snapshot for affected users and rerun the profile-building job from the durable event log once the bug is fixed.

Mitigations

Validate profile-builder output before publishing, keep versioned snapshots for fast rollback, and treat profile writes with the same rigor as any other production data mutation (see Profile State Model).

Profile reads or writes concentrate on a small set of storage partitions, usually from a skewed user population or a poorly chosen shard key.

What fails

The hot partitions saturate, slowing profile reads for every user whose data happens to share that shard — not just the heavy users themselves.

User impact

A seemingly unrelated slice of users experiences slow or degraded recommendations because they share infrastructure with a hot segment.

Detection

Per-partition latency and load metrics revealing sharp skew relative to overall traffic.

Recovery

Rebalance or re-shard the hot partition, and add a request-level cache in front of the hottest keys to absorb read pressure immediately.

Mitigations

Choose a shard key that distributes load evenly rather than purely by raw user ID if usage is skewed, and monitor per-partition load continuously, not just fleet-wide averages.

One candidate-generation source — collaborative, ANN, popularity — misses its deadline on a request.

What fails

If the orchestrator waits for it anyway, the whole request's latency grows; if it enforces the deadline, that source's candidates are simply absent for this request.

User impact

A likely-minor relevance dip if the deadline is enforced correctly; a broad latency spike across the page if it is not.

Detection

Per-source timeout rate, and its correlation with overall request latency distribution.

Recovery

Confirm the per-source timeout is actually being enforced; if the source is systemically slow, temporarily lower its quota or route around it.

Mitigations

Every candidate source has a hard timeout and a defined fallback behavior, verified under load, not just documented in the contract (see Candidate Source Contract).

A candidate source returns zero candidates for a request — from an empty index shard, an unmatched query, or a genuine cold-start case.

What fails

Depending on how much of the candidate pool that source normally contributes, the final set can be thin or skewed toward the remaining sources.

User impact

Recommendations lean harder on whichever sources did return candidates — usually fine with multiple sources, worse if the system was over-reliant on the one that failed.

Detection

Per-source empty-response rate, and candidate-pool-size distribution dropping below expected floors.

Recovery

Confirm the merge stage tolerates an empty source gracefully; if the pool falls below a safe floor, widen fallback sources for that request.

Mitigations

Never let ranking depend on any single source meeting a minimum count; design merge logic to work with a partial set of sources by default (see Candidate Merge & Deduplication).

One candidate source — often the cheapest or fastest — ends up contributing most or all of the merged candidate pool.

What fails

The intended diversity of multi-source retrieval collapses into a de facto single-source system, even though the architecture looks multi-source on paper.

User impact

Recommendations narrow toward whatever bias that one source has (for example, pure popularity), reducing personalization and discovery.

Detection

Source-contribution-share metrics per response, watched for one source consistently exceeding its intended quota.

Recovery

Enforce or tighten the per-source quota in the merge stage so no single source can crowd out the others.

Mitigations

Set and monitor explicit per-source quotas as part of the candidate-source contract, not just a rough expectation (see Single-Source vs Multi-Source Retrieval).

A candidate index — a similarity table, a popularity table — has not refreshed within its expected freshness window.

What fails

Retrieval from that index reflects an outdated view of the catalog or behavior, without any error being raised.

User impact

Recommendations quietly lag behind actual current interest or catalog state — a soft quality regression, not an outage.

Detection

Index age/version metadata compared against its freshness expectation, alerting when the gap exceeds a threshold.

Recovery

Trigger an out-of-cycle rebuild, and if the staleness is severe, temporarily shift weight toward fresher sources.

Mitigations

Publish index age as an observable metric (see Recommendation Observability), with alerting tied to each index's documented freshness expectation.

A candidate index build produces corrupted or invalid data — truncated, malformed, or built from a broken input snapshot.

What fails

Queries against the corrupted index return wrong, missing, or nonsensical candidates.

User impact

Potentially severe: every request reading from the corrupted index sees degraded or broken recommendations until it is fixed.

Detection

Automated validation checks — size, checksum, sample-query sanity — run during the build-and-validate step before publication.

Recovery

Roll back the atomic pointer to the last known-good index version immediately; do not attempt to patch a live corrupted index.

Mitigations

Never publish an index without passing validation first, and always retain the previous version for instant rollback (see Candidate Index Publication).

The ANN/vector index serving embedding-based retrieval goes down or becomes unreachable.

What fails

Embedding-based retrieval contributes zero candidates until the index recovers.

User impact

Recommendations lean on remaining sources (popularity, collaborative, content-based); quality dips somewhat but the page still returns a useful response if multi-source retrieval is in place.

Detection

ANN-source error/timeout rate, and its candidate-contribution share dropping to zero.

Recovery

Fail over to a replica of the index if one exists, or run in a degraded mode without embedding-based candidates until it is restored.

Mitigations

This is exactly why embedding/ANN retrieval is one source among several rather than the sole retrieval mechanism (see Embedding / ANN Retrieval).

A feature bundle returns with some fields absent — a new item with no computed stats yet, or a partial store outage.

What fails

Ranking receives an incomplete feature vector for some candidates.

User impact

Minor if default-value semantics are well-defined; a quietly biased or broken score for those candidates if defaults are undocumented or wrong.

Detection

Missing-field rate per feature name, correlated with specific item cohorts (typically new items) or specific store partitions.

Recovery

Apply the documented default/fallback for the missing feature; if the missing-value rate is abnormally high, treat it as a feature-store degradation, not routine cold-start noise.

Mitigations

Require an explicit, documented default for every feature definition (see Feature Definition & Versioning) rather than letting missing values silently become zero or null.

Ranking scores candidates using a feature bundle computed under a different schema version than the model expects.

What fails

The model receives inputs shaped differently than it was trained on, without necessarily erroring out.

User impact

A quiet ranking-quality regression — the response still returns 200, but its relevance is degraded in a way that is hard to detect from system metrics alone.

Detection

Version-tag mismatch checks between model, feature schema, and index at request time, and, longer term, quality-metric drift.

Recovery

Reject or fall back to a known-compatible model/feature version combination rather than scoring with mismatched assumptions.

Mitigations

Enforce version compatibility checks in the serving path itself, not only as a deployment-process convention (see Ranking Model & Index Version Compatibility).

A feature's underlying computation has not refreshed within its expected window, so its value is technically present but outdated.

What fails

Ranking uses an accurate-looking but out-of-date signal, silently.

User impact

A subtle relevance dip that is easy to miss because nothing about the response looks wrong.

Detection

Per-feature freshness metadata compared against its documented freshness expectation.

Recovery

Trigger recomputation for the stale feature family, and consider excluding severely stale features from scoring temporarily.

Mitigations

Every feature carries a freshness expectation as part of its definition, monitored the same way index freshness is (see Feature Definition & Versioning).

The feature value a model was trained on and the value it sees in production quietly diverge.

What fails

The model's learned weights no longer match the actual distribution of inputs it scores at serving time.

User impact

Ranking quality degrades in ways that do not show up in any infrastructure metric — only in quality/engagement metrics, often gradually.

Detection

Comparing offline feature distributions used in training against online feature distributions in production for the same feature names.

Recovery

Identify and fix the divergent computation path (usually a definition or timing difference between offline and online), then retrain or recalibrate.

Mitigations

Centralize feature computation logic through the feature store's shared definitions rather than letting online and offline paths reimplement the same feature independently (see Feature Online/Offline Consistency).

A training pipeline joins current feature values against historical labels instead of the feature values that existed at each label's own timestamp.

What fails

The model trains on information it would never have had at real serving time.

User impact

Offline metrics look better than they should; live performance underperforms what offline evaluation predicted, discovered only after a rollout.

Detection

Point-in-time-correctness audits on the training-data-assembly pipeline, and a persistent gap between offline validation metrics and live experiment results.

Recovery

Rebuild the training pipeline with a proper point-in-time join, retrain, and re-validate before the next rollout.

Mitigations

Treat point-in-time correctness as a mandatory, explicitly verified property of the training pipeline, not something assumed because a feature store is in use (see Feature Point-in-Time Correctness).

The model-serving fleet behind the Ranking Service saturates under load — CPU, accelerator, or memory bound.

What fails

Scoring latency rises past the request budget, or the fleet starts shedding requests outright.

User impact

Rising fallback rate to cheaper ranking or cached lists, and potential latency spikes for requests that do get scored.

Detection

Model-server utilization, queueing time, and p99 scoring latency compared against the allocated budget.

Recovery

Shed load by reducing candidate count or switching to a cheaper fallback model or pre-ranker, and scale the fleet horizontally.

Mitigations

Keep the model-serving fleet's capacity headroom monitored against realistic peak candidate volume, and always have a cheaper fallback scoring path ready (see Multi-Stage Ranking).

A newly deployed ranking or retrieval model performs worse than the one it replaced, despite passing offline validation.

What fails

Nothing crashes — the model returns valid scores, just worse ones.

User impact

A silent relevance regression for whatever traffic share the new model is serving.

Detection

Canary/A-B quality metrics diverging negatively from control, and recommendation-quality observability catching the shift before it reaches full traffic.

Recovery

Roll back to the previous model version immediately using the retained rollback path; do not attempt to patch a bad model live.

Mitigations

Never skip the shadow/canary stages, and require a clear rollback trigger tied to live quality metrics, not just infrastructure health (see Model Deployment & Rollout).

A feature-schema change ships without updating, or without properly re-validating, the ranking model that depends on it.

What fails

The model scores using feature values it was not trained to interpret in their new shape.

User impact

A ranking-quality regression, likely broad if the feature is widely used, that returns 200s the whole time.

Detection

Automated contract checks between the model's expected feature-schema version and what is actually being served, run continuously, not only at deploy time.

Recovery

Roll back either the feature-schema change or the model version so the two are compatible again.

Mitigations

Coordinate feature-schema changes and model deployments through the same version-compatibility gate (see Ranking Model & Index Version Compatibility).

The distribution of ranking scores shifts unexpectedly — scores compress toward a narrow range, or a large batch of candidates score near-identically.

What fails

Downstream logic that assumes a healthy score spread — top-K selection, calibration-dependent business rules — behaves oddly.

User impact

Lists can look arbitrarily ordered or fail to differentiate clearly better items from mediocre ones.

Detection

Monitoring the statistical distribution of scores per request and per model version, alerting on sudden shifts.

Recovery

Compare against the previous model/feature version to isolate whether the shift traces to a specific rollout, and roll back if so.

Mitigations

Treat score-distribution monitoring as a standard part of recommendation observability, not just latency and error-rate tracking (see Recommendation Observability).

A post-ranking filter — a diversity cap, dedupe rule, business rule — removes so many scored candidates that too few remain for the response.

What fails

The final list is shorter than the UI expects, or thin enough that quality visibly suffers.

User impact

Users see an oddly short list, repeated filler items, or a fallback triggered unnecessarily.

Detection

Post-filter survival-rate metrics per surface, alerting when the rate drops below an expected floor.

Recovery

Loosen the offending filter's threshold temporarily, or fall back to a wider pre-filter set if post-filtering removed too aggressively.

Mitigations

Size post-ranking filters against the actual candidate volume feeding them, and monitor survival rate as a first-class metric (see Post-Ranking & Re-ranking).

The policy/eligibility filter itself fails open (allows everything through) or fails closed (blocks everything) due to a bug or dependency outage.

What fails

Fail-open risks showing policy-violating content; fail-closed risks an empty or near-empty response.

User impact

Fail-open is a correctness and trust incident; fail-closed is an availability incident — both serious, in different ways.

Detection

Policy-filter pass-rate metrics, alerting on both a sudden drop to near-zero (fail-closed) and a sudden rise to near-100% (fail-open, where abnormal for the surface).

Recovery

If fail-open is detected, stop serving from the affected path immediately even at the cost of availability; if fail-closed, restore the dependency or fall back to a conservative cached-safe list.

Mitigations

Default policy filters to fail closed, never fail open, and treat the eligibility/policy layer as a hard dependency worth its own redundancy (see Eligibility Filtering).

An item is deleted or removed from the catalog, but a stale candidate index or cache still contains it.

What fails

The post-filter eligibility recheck is supposed to catch this, but if it is skipped or itself reads stale data, the deleted item reaches the response.

User impact

Users click into a broken or missing item — a visible correctness failure, not just a relevance one.

Detection

Post-response click-through errors on deleted items, and eligibility-recheck coverage audits.

Recovery

Purge the item from caches and indexes explicitly on deletion, and confirm the post-filter recheck is actually querying current catalog state.

Mitigations

Treat item deletion as an event that must propagate to every cache and index with a bounded SLA, and never let the final eligibility recheck read from a source that might not reflect it (see Eligibility Filtering).

A commerce item goes out of stock, but candidate sources built on a slightly older catalog snapshot still surface it.

What fails

The same class of problem as a deleted item, but with availability state instead of existence.

User impact

Users see and may attempt to act on an item they cannot actually obtain — a trust and conversion problem.

Detection

Post-recommendation availability-mismatch rate: items shown that were already unavailable at response time.

Recovery

Tighten the freshness expectation on the availability signal feeding the eligibility recheck, or add a last-moment availability check for high-value surfaces.

Mitigations

Availability is a hard eligibility constraint, not a ranking feature — it belongs in the mandatory post-filter recheck, not just as a signal the model learns to downweight (see Eligibility Filtering).

A cache key collision or bug causes one user's cached recommendation list to be served to a different user.

What fails

The Recommendation Cache's key scoping (user ID, surface, locale) breaks down, usually from a key-construction bug or an overly aggressive cache-sharing optimization.

User impact

A serious privacy and relevance failure — a user sees recommendations, or worse, personalized context, belonging to someone else.

Detection

Cache-key collision audits, and user reports of clearly foreign recommendations are a strong, if late, signal.

Recovery

Invalidate the affected cache entries immediately and roll back whatever change altered key construction or sharing logic.

Mitigations

Treat cache-key scoping as a security-sensitive contract with its own tests, not just a performance optimization detail (see Recommendation Cache).

A hot cache key expires and many concurrent requests simultaneously try to recompute it.

What fails

The recompute storm hits candidate generation and ranking all at once for the same key, spiking load well beyond steady state.

User impact

Latency spikes for the affected surface right at the moment of expiry, potentially cascading into ranking-service overload.

Detection

A correlated spike in ranking QPS immediately following a hot key's TTL expiry.

Recovery

Apply request coalescing so only one request recomputes while others wait on that result, and extend the TTL briefly under load if needed.

Mitigations

Use soft TTLs with background refresh and request coalescing on hot keys by design, not as a reactive fix (see Recommendation Cache).

A user pages through a recommendation feed and sees the same item appear on two different pages.

What fails

Each page triggered an independent candidate-generation-and-ranking cycle, and something upstream — a cache refresh, an index update, a new event — shifted between the two requests.

User impact

A visibly repetitive, lower-quality experience, especially noticeable on long-scroll surfaces.

Detection

Cross-page duplicate-item rate within a single browsing session.

Recovery

Verify the pagination cursor or continuation token is actually pinning the candidate or seen-item state across pages for that session.

Mitigations

Implement session-scoped pagination consistency deliberately (see Recommendation Pagination & Session Consistency) rather than assuming independent per-page requests will naturally agree.

A user's experiment-variant assignment changes between requests instead of staying stable for the life of the experiment.

What fails

The assignment mechanism is not deterministic or not sticky — for example, it depends on something that changes between requests instead of a stable hash of user ID and experiment ID.

User impact

No direct user-facing harm, but the experiment's measured results become invalid, mixing treatment and control behavior for the same users.

Detection

Per-user assignment-consistency audits across requests within an experiment's duration.

Recovery

Fix the assignment mechanism to be deterministic and sticky, and discard or heavily caveat any experiment results collected during the drift window.

Mitigations

Assignment should be a pure, deterministic function of stable identifiers, verified with automated consistency checks, not just trusted by design (see Experiment Assignment & Logging).

The experiment/variant identifier fails to propagate into the impression/exposure log for some fraction of requests.

What fails

Those logged events cannot be attributed to a variant, silently shrinking the usable sample for analysis.

User impact

No direct user impact, but experiment conclusions drawn from incomplete logging can be misleading if the loss is not random — for example, concentrated on one client version.

Detection

Null/missing-variant rate in the impression log, watched per client version and surface.

Recovery

Backfill the missing variant tag where possible from correlated request logs, and exclude the affected window from analysis if it cannot be recovered.

Mitigations

Treat experiment-tag propagation as a required field in the impression log's schema, validated at write time (see Impression & Exposure Logging).

A bug in the Real-Time Signal Processor writes an incorrect or nonsensical session overlay for a user.

What fails

The next request merges that corrupted overlay with the durable profile, producing a distorted result.

User impact

Affected users see recommendations abruptly and inexplicably shift, often within a single session.

Detection

Anomaly checks on session-overlay content (unexpected size, invalid categories), and session-to-session recommendation-volatility metrics.

Recovery

Clear or reset the corrupted overlay for affected users and fall back to the durable profile alone until the bug is fixed.

Mitigations

Validate session-overlay writes the same way durable profile writes are validated, and bound how much influence the overlay can have on the final ranking (see Real-Time Session Overlay).

A single trending item's counter update rate overwhelms the popularity-aggregation storage for that key.

What fails

Increment operations on the hot counter queue up or get dropped, and the popularity signal for that item becomes inaccurate right when it matters most.

User impact

The trending item's rank may be inaccurate — overcounted or undercounted — during the exact window it is going viral.

Detection

Per-key write-latency and drop-rate metrics on the popularity aggregator, isolating the hot key.

Recovery

Shard or locally pre-aggregate counts for the hot key before they hit shared storage, and widen capacity for that specific shard temporarily.

Mitigations

Design the popularity aggregator for hot-key skew from the start — local pre-aggregation, sharded counters — rather than assuming uniform key access (see Popularity / Trending Retrieval).

The scheduled Batch Pipeline run that refreshes profile aggregates, features, or indexes runs significantly later than expected.

What fails

Every consumer of that batch's output serves an older-than-intended snapshot past its normal refresh point.

User impact

A gradual, usually mild quality drift — features and profiles feel slightly behind — unless the delay is extreme.

Detection

Batch-job completion-time monitoring against its scheduled SLA, and downstream freshness-metadata age.

Recovery

Prioritize and expedite the delayed job, and confirm the online path is serving the last good version rather than stalling.

Mitigations

Alert on batch-job lateness against an explicit SLA, and design every online consumer to serve its last good version by default when a fresher one is not yet available (see Offline Data Platform).

An index publication is interrupted partway through the atomic switch, leaving reads split between old and new versions.

What fails

The build-validate-switch sequence was not actually atomic in practice — a deploy race, a partial rollout, or a bug lets some servers read the new index while others still read the old one.

User impact

Inconsistent recommendations across requests to the same user, depending on which server instance handled the request.

Detection

Version-tag distribution across the serving fleet, alerting when more than one index version is live simultaneously outside an intentional gradual rollout window.

Recovery

Force all servers to a single consistent index version immediately, rolling back to the prior version if the new one is suspect.

Mitigations

Make the pointer switch a genuinely atomic, fleet-wide operation, or make gradual rollout an explicit, monitored process rather than an accidental side effect of deploy timing (see Candidate Index Publication).

A regional feature-store replica falls behind the primary region's writes.

What fails

Requests served in the lagging region read outdated feature values relative to what the same user would see in the primary region.

User impact

Regionally inconsistent recommendation quality for the same user depending on which region serves them, most noticeable for users who travel or connect through different edges.

Detection

Cross-region replication-lag metrics on the feature store, and cross-region quality-metric divergence for otherwise-comparable traffic.

Recovery

Prioritize catch-up replication for the lagging region, and consider routing affected users to a region with fresher data as a temporary measure.

Mitigations

Treat regional replication lag as a monitored SLA, not an assumed-negligible detail (see Multi-Region Serving).

An entire region's recommendation-serving stack becomes unavailable.

What fails

Requests that would have been served locally need to fail over — and can only do so safely if another region actually owns compatible, current artifacts and holds the appropriate fencing.

User impact

If regional ownership and fencing were built deliberately, users in the affected region see degraded but functional service from another region; if not, failover risks inconsistent or conflicting state, or an outright outage.

Detection

Regional health checks and cross-region failover-readiness audits, run proactively, not only during an actual incident.

Recovery

Fail over to a region with current artifacts and proper fencing; if no such region is ready, degrade to safe fallback behavior (popularity, cache) rather than attempting an unsafe failover.

Mitigations

Regional isolation must be built with real ownership and fencing ahead of time — assuming multi-region deployment implies safe failover is the mistake this guards against (see Multi-Region Serving).

Recommendation quality degrades measurably while every service continues returning 200 and every standard system-health metric looks normal.

What fails

The failure is not in any single component's health — it is in the emergent quality of what the pipeline produces, which no uptime dashboard measures directly.

User impact

Users experience worse recommendations for an extended period before anyone notices, since nothing paged an on-call engineer.

Detection

Only recommendation-quality observability — engagement relative to impressions, exposure concentration, new-item coverage, experiment health — can catch this, which is exactly why it has to exist as its own monitoring category (see Recommendation Observability).

Recovery

Once detected, work backward through recent changes — model rollout, feature-schema change, index publication, candidate-source mix shift — to find the actual cause, since the symptom alone does not point to one.

Mitigations

Invest in recommendation-quality observability as seriously as system-health observability; treat "everything is green" as insufficient evidence that the product is working well.

A change — a new candidate source, a richer feature set, a lower cache TTL — quietly multiplies infrastructure cost far beyond its relevance benefit.

What fails

Nothing breaks technically; the system just becomes far more expensive per request than intended, often noticed only when a cost review runs.

User impact

None directly, though the eventual response — emergency cost-cutting — can degrade quality if done hastily.

Detection

Cost-per-request or cost-per-recommendation-response tracked as a first-class metric, alerting on trend breaks, not just absolute spend.

Recovery

Identify which recent change drove the cost increase, and evaluate whether its relevance benefit actually justifies the cost.

Mitigations

Treat cost per request as something to budget and monitor deliberately, the same way latency and quality are — every added source, feature, or freshness improvement has a cost trade-off (see Freshness vs Cost and Feature Richness vs Serving Speed in Trade-offs & Decisions).