Recommendation System — Architecture Deep Dive

A recommendation request is a contract: what context comes in, and what guarantees the response must uphold before any retrieval work happens.

What the request carries

A user or session identifier, the requesting surface (home feed, related-items rail, search-adjacent slot), context signals such as locale and device class, an optional pagination or continuation token, and any explicit constraints the caller sets (a category filter, a maximum item count).

What the response must guarantee

A latency deadline the Recommendation Service will not exceed even if it must degrade quality to hit it, a bounded item count, full compliance with eligibility and policy rules regardless of which candidate source proposed an item, and ordering that stays stable enough that pagination does not visibly duplicate or silently drop items (see Recommendation Pagination & Session Consistency).

A click is not complete feedback on its own. To interpret it correctly, the system also needs to know what else it showed — and what it never gave the user a chance to see at all.

Four exposure states

Generated — a candidate source produced this item. Returned — the item survived ranking and eligibility checks into the response payload. Rendered / exposed — the client actually displayed it; a server-side "returned" event does not guarantee the user's screen ever showed it, since scrolling, backgrounding, or a dropped render can prevent that. Consumed — the user clicked, watched, purchased, or otherwise acted on it. Each state should be logged distinctly.

Why the distinction matters downstream

Without an exposure log, training data cannot separate "the user saw this and ignored it" from "the user never had the opportunity to see this at all." Collapsing that distinction biases every model trained on the resulting logs toward whatever the system already chose to expose — a selection-bias problem that reappears throughout this page, from Exploration vs. Exploitation to Recommendation Observability.

The event path is where the recommendation system earns the right to personalize anything at all — and where a sloppy contract quietly poisons everything built on top of it.

Ingestion entrypoints

Mobile apps, web clients, backend services, and player surfaces emit impression, click, watch, skip, like, save, and purchase events to the Event Collector. This layer validates schema, enriches with request IDs and surface metadata, and rejects or quarantines malformed payloads before they contaminate downstream consumers.

Durability before fan-out

The collector writes to a durable Event Store or replicated log before acknowledging success. That keeps the architecture honest: if downstream consumers lag or crash, the source history still exists and can be replayed (see Event Replay & Idempotency).

When an action happened (event-time) and when the pipeline observed it (processing-time) are different clocks. Late and out-of-order events are the normal case, not an exception to handle later.

Why the distinction matters here

A watch event delayed by a flaky mobile connection should still count toward the user's interest at the time it actually happened, not be discarded or folded into whatever the pipeline is processing right now. Aggregation jobs, profile builders, and the popularity aggregator all need to key on event-time, with explicit handling for events that arrive after their window has already been summarized.

Ordering scope is narrow

Sequence IDs assigned by an event collector typically establish order within a narrow scope — per connection, per partition, or per session — not a global order across the entire event stream. Anything that assumes cross-user or cross-session ordering needs its own explicit mechanism; the collector does not provide one for free.

Every consumer of the event log — profile builders, popularity aggregators, offline training jobs — will see some events more than once. The architecture has to make that harmless.

Stable event identity

Every event carries a stable ID so that client retries, at-least-once delivery, and manual replays do not double-count the same action in a profile aggregate or a trending counter. Consumers key their aggregation on that ID, not on arrival order.

What the durable log is — and is not — the source of truth for

The durable Event Store is the authoritative ingestion and replay boundary: anything a consumer needs to rebuild state can be recovered by replaying from it. That is a narrower claim than saying it holds all behavioral history forever. Retention windows are finite, and a separate offline data warehouse or lake commonly retains longer-term analytical history once events age out of the operational log (see Offline Data Platform).

The profile layer turns a large raw event history into a compact object the serving stack can query on every request.

Four layers, four refresh clocks

A useful way to decompose "the profile": a durable long-term profile built by batch jobs (interest scores, category affinity, long-window aggregates); a bounded recent-history window (the last N consumed or suppressed items, not the full lifetime list); a session overlay updated in near-real-time from the current visit (see Real-Time Session Overlay); and policy/eligibility context — locale, safety flags, consent state — that travels with the profile and is enforced downstream regardless of which candidate source or cache produced a result.

Read-optimized layout

The User Profile Service stores this state in a shape Candidate Generation and Ranking Service can fetch quickly — typically by user ID, with small payload sizes and predictable read latency. Large nested histories stay off the hot path.

Every candidate source the system fans out to — however it works internally — should honor the same contract, so Candidate Generation can treat them uniformly.

The simplest legitimate candidate source is a windowed popularity or trending count — the same aggregator that powered the very first non-personalized version of this system.

What it is good for

Windowed regional and global counts are cheap to compute, cheap to serve, and safe to fall back to when personalized sources are slow, cold, or unavailable. They also provide a floor of reasonable candidates for brand-new users and brand-new items that no personalized source has learned about yet.

Collaborative retrieval proposes items favored by users with similar behavior — "people with a similar interaction pattern also engaged with this."

Architectural role, not an algorithm claim

This page describes what collaborative retrieval contributes to the pipeline — a candidate source built from aggregate interaction patterns rather than item content — without committing to a specific technique. However it is computed internally, it still has to honor the same candidate-source contract as every other source: a version, a timeout, a quota, and a provenance label.

Embedding-based retrieval represents users and items as vectors and retrieves near neighbors from an approximate index. It is one candidate source among several, not a replacement for the retrieval layer.

Shape of the pipeline

Embeddings are produced offline or on a periodic refresh cadence → an ANN index is built and published from those embeddings → at request time, a query embedding is computed or looked up → the index returns a bounded set of nearest candidate IDs → those IDs are handed to merge and dedupe exactly like any other source's output.

Approximate means a trade-off, not a compromise

An ANN index trades a small amount of recall for much lower latency and memory versus exact nearest-neighbor search. Where that trade-off lands — how much recall is acceptable for how much speed — is a tuning decision for the specific deployment, not a fixed architectural constant.

Freshness and version compatibility

A newly added item has no embedding, and therefore no presence in the index, until the next build or incremental update — this is exactly why Cold Start for Items exists as a separate concern. Just as importantly, the query embedding must come from a model version compatible with whatever index version is currently published; publishing a new embedding model and a new index are coordinated events, not independent deploys (see Candidate Index Publication).

Candidate generation is where the system stops pretending it can reason about the whole catalog on every request — and merge is where several independently produced lists become one.

Parallel retrieval, bounded budgets

A candidate-generation request fans out to several sources at once — collaborative, content-based, embedding/ANN, popularity, and rule-based sources such as continue-watching. Each source runs under its own contract (timeout, quota, fallback), so the overall latency reflects the deadline the orchestrator enforces across sources and the time spent merging, not simply whichever source happens to be slowest; a source that blows past its deadline is dropped, not waited on indefinitely.

Merge, dedupe, and score normalization

Returned candidates are deduplicated across sources (the same item can legitimately surface from more than one), and their source-local scores are normalized or set aside entirely — raw scores from different sources are not directly comparable, so merge logic typically preserves provenance and lets ranking do the real comparison rather than trusting retrieval scores at face value.

Eligibility gets checked twice, for two different reasons: once early for efficiency, and once again right before the response goes out, for correctness.

Pre-filter

Early in the pipeline, obviously ineligible items are dropped — locale-blocked content, policy-blocked categories, items the profile's bounded history marks as already consumed. This exists mainly for efficiency: there is no reason to spend ranking budget scoring items that can never legally be shown.

Post-filter: the final recheck

Immediately before the response is assembled, the system rechecks hard constraints — availability, privacy or safety policy, licensing or region — against current state. This recheck exists because the indexes and caches candidates were pulled from earlier in the request may already be stale by the time ranking finishes; a cached or precomputed list must never bypass this step.

Suppressing items the user has already seen or consumed improves relevance — but only within a bounded, surface-specific window, not the user's entire lifetime history.

Bounded by design

Suppression draws on the bounded recent-history window described in Profile State Model, not a full event-log scan. Keeping it bounded is what makes the check cheap enough to run on every request.

Repeat policy is surface-specific

Not every surface wants the same repeat policy. A discovery feed usually suppresses recently seen items aggressively; a commerce reorder rail may deliberately re-show a previously purchased item. The suppression rule belongs to the surface's eligibility configuration, not to one universal system-wide default.

Feature hydration assembles the inputs each candidate needs before ranking scores it — and it does this in bulk, never one remote call per candidate.

Batched, not per-item

For a candidate set of a few hundred items, the ranking path issues a small number of batched multi-get lookups against the online Feature Store — not hundreds of individual round trips. This is what keeps feature lookup a fixed, bounded cost regardless of how many candidates ranking is scoring.

What gets assembled

User-profile features, item metadata, context features (device, surface, time of day), and session-overlay recency signals. Item metadata typically comes from a compact catalog cache alongside the feature-store reads rather than from the feature store itself.

A feature name is only useful if everyone consuming it agrees on what it means — which window it covers, how it is computed, and what a missing value implies.

Centralized definitions

Feature names, time windows, computation logic, and freshness expectations are versioned centrally as metadata, separate from where the values themselves live. That is what lets the online and offline stacks agree on what "7-day category affinity" actually means without re-deriving it independently.

Safe rollout of a definition change

When a feature's definition changes, the safe sequence is: recompute it offline under the new definition, validate the new output, publish it as a new version, and only then switch the online serving path over. Skipping straight to a live switch risks serving one definition online while offline jobs and the training pipeline still assume the old one.

Default-value semantics

A missing feature value needs an explicit, documented default or fallback behavior. An undocumented default silently changes what the model sees for cold items or users without anyone deciding that on purpose.

A training example for time T is only valid if every feature value it uses was actually available at or before T. Using a later value silently leaks the future into training.

How leakage happens

If a training pipeline joins today's current feature values against historical labels instead of joining each historical event against the feature values that existed at that event's own timestamp, the model trains on information it would never have had at real serving time.

Why it survives offline evaluation

Leaked features often make offline metrics look better, because the model is effectively cheating with information it will not have in production. The gap surfaces later — at serving time, or in a live experiment — when real performance falls short of what the offline numbers predicted.

The feature store is the contract between batch computation and online serving — three related but distinct concerns, not one universal "source of truth."

Three separate concerns

Feature definitions / metadata — the semantic contract described in the previous section. Offline feature data — large historical windows used for training and backfill, optimized for throughput. Online feature data — the low-latency serving copy that Candidate Generation and Ranking Service read on the hot path. Keeping these three explicit is what prevents "the feature store" from becoming a vague catch-all term.

Train/serve skew

Skew shows up when the offline value a model trained on and the online value it scores against in production quietly diverge — different computation logic, different freshness, or a definition change that only reached one side. A shared feature store reduces this risk by centralizing definitions and computation; it does not eliminate it outright, since a bug in either the offline or the online write path can still reintroduce skew.

No single ranking model searches or scores the entire catalog directly. Ranking only ever sees the bounded set that retrieval already produced, and even that set may be narrowed further in stages.

Stages

Retrieval hands ranking a bounded candidate set (illustrative order of magnitude: low hundreds). When that set is large enough that scoring all of it with the full model would blow the latency budget, an optional pre-ranker — a cheaper, lower-fidelity model — trims it down further. The main ranker then spends its full feature-rich scoring only on that smaller, already-promising set. Post-ranking applies final adjustments (see Post-Ranking & Re-ranking).

Why the split exists

A feature-rich model scoring every retrieved candidate is often the most expensive step in the request. Splitting scoring into a cheap pre-ranker plus an expensive main ranker lets the system afford a more sophisticated final model without paying its full cost on every candidate that will not survive anyway.

The ranking model, the feature schema it expects, and the candidate index or embeddings it scores against all version independently. The serving path has to confirm they are compatible before trusting a response.

Three things that can drift apart

A model trained against feature-schema version 4 should not silently score features served under schema version 5. A ranking model trained on embeddings from index version N should not blindly assume candidate IDs coming from index version N+1 mean the same thing. Version tags travel with the model, the feature bundle, and the index, and the serving path checks them.

Handling a mismatch

On a detected mismatch, the safer default is to fall back to a known-compatible prior version rather than score anyway with mismatched assumptions — an incompatible combination can still return a response and a 200 status while quietly producing degraded rankings.

After scoring, the serving layer still has work to do before the list is ready to return.

Top-N selection

The system selects the top-performing items and applies light post-processing: dedupe, diversity spacing, and business constraints (do not show two near-duplicate variants back to back, cap items from one creator or brand). Selecting the top-K does not always require a full sort of every scored candidate — a partial selection is often sufficient once the cut point is fixed.

Scores are not calibrated probabilities

Ranking scores are relative signals useful for ordering candidates against each other within one request. They should not be treated as calibrated probabilities of engagement, and comparing raw scores across different requests or different model versions is not meaningful without explicit calibration work.

Always showing the highest-scored items is itself an architectural decision, with a consequence: it maximizes short-term relevance while starving new items — and the model itself — of anything to learn beyond what it already favors.

Where this lives in the architecture

Reserved exploration slots in the response, controlled randomization or diversity rules applied at post-ranking, and coverage tracking across items and sources are concrete architectural choices — not just a warning to keep in mind. They typically sit in the post-ranking stage, informed by the candidate-source mix from merge/dedupe.

Why this depends on impression logging

Without the exposure log from Impression & Exposure Logging, the system cannot tell whether low engagement with a category means users dislike it, or means the system rarely exposed it in the first place. Exploration and impression logging are two halves of the same correctness problem: selection bias.

A new user has no interaction history, so personalized retrieval has nothing to work with. The system needs a deliberate fallback recipe, not a degraded version of the normal path.

What changes

Candidate generation shifts its source mix toward popularity retrieval, content/metadata-based retrieval, and any explicit onboarding signals (selected interests, referral context) instead of collaborative or embedding-based sources that need history. Ranking still runs — it simply has weaker personalized features to work with.

A newly published item has no interaction history and, if retrieval depends on embeddings, no presence in the ANN index until the next build. Left alone, the system will structurally never surface it.

Mechanisms that give new items a chance

Metadata or content-based retrieval as an interim source that does not require interaction history, explicit exploration slots or coverage quotas reserved for under-exposed items, and a bounded index-refresh cadence so new items enter the ANN index and other data-driven sources within a known window rather than an indefinite one.

Page 2 of a recommendation feed is a separate request. Unless the architecture does something deliberate, it can be generated independently of page 1 and disagree with it.

Why duplicates and reordering happen

If each page triggers a fresh candidate-generation-and-ranking cycle, and anything upstream changed even slightly between requests — a new event landed, a cache refreshed, an index republished — the same item can appear on two pages, or the relative order of items can shift between them.

Keeping a session consistent

A pagination cursor or continuation token that pins the candidate set (or at least the seen-item exclusion list) for the life of a browsing session lets later pages exclude what earlier pages already returned, even if underlying data shifted in between.

Caching reduces repeated retrieval and ranking work, but the cache hierarchy has to respect user context, freshness rules, and eligibility — it cannot bypass them.

What gets cached

Whole recommendation responses where reuse justifies it — an anonymous home feed, a short-lived personalized list, a hot category rail — plus smaller hot caches for item metadata, eligibility flags, and category-popular lists that reduce deeper-store lookups on the ranking path.

Freshness and stampede control

Short TTLs keep a heavy user from seeing the same list for too long, and a strong session or real-time signal can bypass a cached list for that user outright. Hot keys can still stampede when they expire; request coalescing, soft TTLs, and background refresh keep one expiring list from triggering a ranking storm across replicas simultaneously.

The session overlay is the fast path that lets the system respond to what the user is doing right now, without waiting for the next profile-rebuild cycle.

Capture and merge

The Real-Time Signal Processor consumes fresh events — the last clicked item, a skipped topic, a newly watched category — and stores a short-lived session summary optimized for rapid readback. At serving time, the Recommendation Service merges that overlay with the stable durable profile from Profile State Model, so the system stays grounded in long-term preferences while still reacting to sudden shifts in intent.

The offline platform is where large-history computation happens without blocking the request path — and where longer-term analytical history lives once it ages out of the operational event log.

Batch cadence and outputs

The Batch Pipeline typically runs on hourly or daily cadences depending on the feature family. It publishes refreshed profile aggregates, feature tables for the Feature Store, candidate-source indexes, and eligibility snapshots derived from the Item Catalog and event history — all as versioned inputs, not silent in-place updates.

Warehouse/lake vs. the operational log

A separate offline data warehouse or lake commonly holds long-term analytical history — for training, backtesting, and long-range analysis — distinct from the operational event log's replay window described in Event Replay & Idempotency. Conflating the two understates how much history the platform actually retains, and where.

Refreshing a candidate index — an ANN index, a similarity table, a popularity table — safely follows the same shape as any safe deployment: build the new version alongside the old one, validate it, then switch atomically.

The publication sequence

Build index version N+1 in the background while version N keeps serving → validate N+1 against size, coverage, and quality checks → switch reads to N+1 with a single atomic pointer update → retain N for a bounded window so rollback is immediate if N+1 turns out to be bad.

A new ranking or retrieval model earns its way into full production traffic through the same kind of staged rollout as any other risky change.

Comparing a new model, a new candidate source, or a ranking change fairly requires assigning users or requests to variants consistently — and logging which variant produced which response.

Stable assignment

The same user should consistently land in the same variant for the life of the experiment. Unstable assignment mixes treatment and control behavior for the same person across requests, which quietly invalidates whatever the experiment was trying to measure.

Logging the variant

The impression/exposure log from Impression & Exposure Logging carries the experiment and variant identifier alongside each logged event. Without that, post-hoc analysis has no way to attribute an outcome to the variant that actually produced it.

A recommendation system can be fully "up" — every service returning 200 — while quietly serving worse recommendations. Observability has to include quality signals, not only system health.

System health metrics

Per-stage latency, per-candidate-source timeout and fallback rates, cache hit rate, feature-store availability, and candidate-index freshness/staleness.

Quality / health metrics

Click-through and downstream conversion measured relative to impressions, not raw counts; the candidate-source contribution mix and how concentrated exposure is across items and sources; new-item exposure rate; and experiment health (assignment balance, logging completeness).

At scale, the hard part is not only serving more requests — it is preventing fan-out, ranking work, and feedback loops from growing faster than user traffic, across every region the system serves.

Sharding and fan-out containment

Candidate indexes and caches can be partitioned by geography, language, category family, or major product surface — one possible set of partition dimensions, not a universal prescription. Every new retrieval source or synchronous dependency expands the request graph, so production designs deliberately limit fan-out and enforce per-source budgets rather than adding sources casually.

Regional isolation requires ownership, not just deployment footprint

Deploying servers in multiple regions is not the same as regional isolation. True isolation requires locally owned artifacts — regional index replicas, regional feature-store copies, regional model versions — and fencing so two regions never believe they independently own the same write path at once. Without that ownership and fencing, a regional failure does not degrade gracefully; it risks inconsistent or conflicting state across regions.

Contracts to remember across the page