Designing a Recommendation System

We need to design a recommendation system that helps users discover relevant videos, products, posts, or songs from an enormous Item Catalog without scanning everything manually, while also improving engagement, retention, and marketplace efficiency.

Modern products face information overload. A catalog with millions of items is valuable only if the system can surface a useful next choice quickly. Relevance is therefore a system-design problem: every request needs a fast path from user context to a short ranked list, not a full scan of the catalog.

Why this system is interesting

Recommendation systems sit at the intersection of storage, streaming, caching, and online serving. The architecture has to combine a Recommendation Service at the front door, a durable Event Collector and Event Store, a read-optimized User Profile Service, a multi-source Candidate Generation stage, a low-latency Ranking Service, and a disciplined feedback loop that improves relevance without blindly amplifying popularity.

Functional requirements describe what the recommendation platform must do for end users, product teams, and downstream services.

Non-functional requirements are what separate a demo recommendation endpoint from a product-grade relevance platform.

Good scope discipline keeps this module focused on system architecture instead of turning it into a machine-learning course.

In scope

The online request path, Recommendation Service, Candidate Generation (including popularity, collaborative, and embedding/ANN retrieval as candidate sources), Ranking Service, User Profile Service, Feature Store, Event Collector, Event Store, Batch Pipeline, Real-Time Signal Processor, Recommendation Cache, Item Catalog, impression/exposure logging, eligibility and policy enforcement, model and index rollout, experimentation, cold start fallbacks, feedback loops, and scaling patterns for large catalogs and high request volumes.

Out of scope

Loss-function math, embedding derivations, model-training internals, specific collaborative-filtering algorithms, specific vector-database or ANN-library implementations, search-quality proofs, human-labeling workflows, and UI experimentation details. Those matter around the system, but they are not needed to explain the core serving architecture cleanly, and no specific production system's internals are described here.

Capacity planning for recommendation systems is driven by request rate, event rate, catalog size, and how many candidates each online request must process before a page can render. Every number below is an illustrative interview assumption, not a documented figure from any real system — the formulas are what matter.

Traffic assumptions

Formula: recommendation QPS = DAU × recommendation requests/user/day ÷ 86,400 × peak factor. Illustrative inputs: 60 million DAU, 25 recommendation requests/user/day across home, details pages, autoplay, and related-item modules → 1.5 billion recommendation requests/day → about 17,400 requests/sec average. Applying an illustrative 10× peak factor for bursty consumer traffic gives roughly 174,000 requests/sec peak across the global serving fleet.

Candidate and ranking pressure

Formula: ranking scores/sec = request QPS × candidates scored/request. If Candidate Generation returns an illustrative 400 candidates/request, peak ranking load is roughly 174k × 400 ≈ 69.6 million candidate scores/sec across the fleet. That does not mean one giant machine scoring 69.6 million items; it means the architecture must parallelize ranking aggressively, narrow candidates further with a pre-ranker where needed, and rely on caching so repeated requests do not always trigger full recomputation.

Event volume

Formula: event ingest/sec = DAU × events/user/day ÷ 86,400 × burst factor. If each DAU generates an illustrative 120 interaction events/day across impressions, clicks, watches, likes, skips, and purchases, that is 7.2 billion events/day, about 83,000 events/sec average, and roughly 665,000 events/sec peak with an illustrative 8× burst factor. At an illustrative 300 bytes/event, raw ingest is about 2.16 TB/day before replication and indexing overhead.

Feature reads and retrieval fan-out

Formula: feature reads/sec ≈ ranking QPS × feature bundles/request, served as batched multi-get lookups rather than one call per candidate (see Feature Hydration). Formula: retrieval fan-out = recommendation QPS × candidate sources/request — each additional parallel source multiplies fan-out load across the fleet, which is why source count and per-source budgets are a deliberate design choice, not a free addition.

Impression volume

Formula: impression volume = recommendation responses × items exposed/response. This is the write volume the impression/exposure log has to sustain — a separate, and often larger, stream than the click/action events counted above, since most exposures are never acted on at all.

Catalog and profile footprint

Illustrative assumptions: an Item Catalog of 100 million items at roughly 1 KB of online-serving metadata per item is about 100 GB before replicas and indexes. A compact 8 KB profile per active user across 60M users is roughly 480 GB of profile state distributed across low-latency stores and caches.

The core data model should be explained as entities and serving records, not as raw model tensors or training internals.

Keep the online data model compact. The serving layer usually needs stable IDs, eligibility flags, recency windows, categories, availability, and small profile summaries. Giant raw histories belong in the Event Store and offline systems, not in every request-time payload.

Every recommendation system begins with behavior capture. If the event pipeline is noisy, delayed, or lossy, every downstream layer becomes weaker.

Request-time capture

Clients and backend surfaces emit impression, click, watch, skip, save, and purchase events to the Event Collector. The collector validates schema, enriches events with request context, assigns timestamps or sequence IDs scoped to a connection or session (not a global order), and durably writes them before acknowledging success.

Durability before analytics

The Event Store is the authoritative ingestion and replay boundary for behavioral events — downstream systems such as the Trending/Popularity Aggregator, User Profile Service, and Batch Pipeline consume from that durable log rather than from fragile in-memory side effects. A separate offline warehouse or lake typically retains longer-term analytical history once events age out of the operational log's retention window (full detail in the Architecture Deep Dive).

Streaming and batch fan-out

The same event stream typically feeds multiple consumers: a Real-Time Signal Processor for session recency, a popularity counter service for trending lists, batch jobs that recompute profile aggregates, and monitoring jobs that verify event quality and lag. Every consumer needs to tolerate replayed or duplicate events without double-counting.

A User Profile Service turns raw history into a compact read-optimized summary that the serving stack can actually use inside a tight latency budget.

What belongs in a profile

Profiles store a bounded recent-item history (not the full lifetime log), category affinities, creator or brand affinities, negative signals such as hides or skips, language or region preferences, and eligibility/policy context. The goal is not to copy the whole event log; it is to expose the small subset of user state that Candidate Generation and Ranking Service need quickly. Not every ranking feature has to live here — some are looked up separately per request.

Refresh cadence

Some fields update in near real time through a short-lived session overlay, such as the last few session actions. Others update on slower cadences through the Batch Pipeline, such as 7-day and 30-day aggregates. A good design makes the freshness boundary explicit so teams know which parts are real time and which are periodic.

Read pattern during serving

The Recommendation Service should be able to fetch a compact profile by user ID in a few milliseconds. If a full profile is missing, the system should fall back cleanly to cold-start behavior rather than blocking the page.

Candidate Generation is the funnel that keeps the system tractable. It narrows an enormous catalog to a bounded, plausible set before expensive ranking begins — and it is a distinct stage from ranking, not a preview of it.

Parallel retrieval sources

A production system rarely trusts one source alone. Candidate Generation can query collaborative retrieval, content-based similarity, embedding/ANN retrieval, recent popularity from the Trending/Popularity Aggregator, and business-rule sources such as continue-watching or in-stock promotions. Each source honors a shared contract — a version, a timeout, a quota — and the service merges and deduplicates their output into one candidate pool. No single ranking model is ever asked to search the whole catalog directly; it only ever scores what this funnel already narrowed down.

Cold start and fallback strategy

For a new user with little history, the system cannot rely heavily on the User Profile Service. Instead it combines regional trending items, category-popular items, editorial seed lists, and lightweight contextual hints such as locale or device. New items face the mirror problem — no interaction history and no presence in an embedding index until the next refresh — so metadata-based retrieval and reserved exploration slots give them a path into candidate pools before they accumulate interaction history.

Dedupe, filtering, and eligibility

Before candidates reach ranking, the system removes out-of-stock or unavailable entries, policy-blocked items, region-ineligible content, and — depending on the surface's repeat policy — recently consumed items from a bounded suppression window. This pre-filter protects ranking latency, but the system still rechecks hard eligibility constraints immediately before the response goes out, since indexes and caches can go stale between retrieval and response.

The Ranking Service takes a bounded candidate set and decides which handful deserve screen space right now — spending its most expensive scoring only on items that already survived the candidate-generation funnel.

Ranking stages

For each candidate, the serving stack gathers user-profile features, item metadata, short-term session context, and business constraints through a small number of batched lookups against the Feature Store — never one lookup per candidate. When the candidate set is large enough, a cheaper pre-ranker trims it further before the full Ranking Service scores what remains; this multi-stage split is what lets the system afford a richer main model without paying its full cost on every candidate. After scoring, the service applies post-ranking adjustments such as diversity, dedupe across rails, or sponsored-slot reservation before returning the top list.

Latency budget

Ranking is expensive because it touches many items per request. That is why the platform constrains candidate count aggressively, keeps features in a low-latency Feature Store, and often caches top results for hot surfaces. A practical illustrative budget might allow a few milliseconds for profile lookup, a few tens of milliseconds for retrieval and scoring, and a small remainder for response assembly — the exact split is a tuning decision, not a fixed rule.

Output contract

The ranking output should be stable and explicit: item IDs, scores or debug ranks for observability (treated as relative ordering signals, not calibrated probabilities), reasons or rails for UI labeling when needed, and enough metadata for downstream caching and impression logging.

The Feature Store gives the online and offline sides of the platform a shared contract for feature definitions and freshness — it is three related concerns, not one monolithic "source of truth."

Without a centralized feature contract, the online and offline stacks drift. One service names a feature one way, another computes it on a different time window, and the ranking path quietly stops matching what the offline system expected. The Feature Store exists to make those definitions explicit and to reduce — not fully eliminate — train/serve skew as the underlying algorithms evolve.

Recommendation systems need caching just as much as CDNs do, but the cache keys are shaped by users, surfaces, freshness windows, and fallback rules rather than by static asset URLs.

TTL and invalidation

Like CDN caching, the key questions are what is safe to reuse and for how long. Anonymous or broad trending lists can use longer TTLs. Personalized lists need shorter TTLs and stronger invalidation when a user generates new session activity, changes locale, or consumes one of the recommended items.

Cache scope discipline

Do not over-share personalized caches. A cached list for one user, surface, and freshness window should not accidentally bleed into another user context. The Recommendation Cache usually keys on user ID or audience bucket, surface, locale, and sometimes a coarse freshness epoch.

Some of the most visible recommendation wins come from Real-Time Signal Processor paths that react to the current session faster than the slow profile pipeline can.

Session-level fast path

If a user clicks three cooking videos in a row, the next request should reflect that session intent immediately. The Real-Time Signal Processor consumes fresh events, updates a short-lived session context, and lets the Recommendation Service merge that context with the durable User Profile Service snapshot.

Feedback loop benefits and risks

The feedback loop is what makes the system improve over time, but it can also spiral into echo chambers or popularity bias if every new recommendation only reinforces the strongest existing trend. This is only measurable at all because of impression/exposure logging: without knowing what the system chose to expose, it cannot tell whether low engagement with a category means genuine disinterest or simply a lack of exposure. Production systems counter the bias risk with diversity caps, reserved exploration slots, category balancing, and health metrics that watch for over-concentration — exploration vs. exploitation is treated as a deliberate architecture decision, not just a warning (see Architecture Deep Dive and Trade-offs & Decisions).

Freshness boundaries

Not every signal must update every store instantly. Session context may refresh within seconds, while broader weekly interests refresh in batch. The architecture should make that split visible so teams can reason about latency, cost, and consistency clearly.

A strong system-design answer grows the architecture step by step. Each stage exists because the previous one stopped being good enough.

The 11-stage journey

1. Everyone Sees the Same Trending Content → 2. Store User Interactions → 3. User Profile System → 4. Candidate Generation → 5. Ranking Service → 6. Cold Start Problem → 7. Real-Time Signals → 8. Feedback Loop → 9. Feature Store → 10. Online + Offline Architecture → 11. Production Scale Architecture.

This evolution pattern is useful because every new box solves a named pressure. Trending solves empty-state simplicity. Event storage enables learning. Profiles compress history. Candidate generation avoids full scans. Ranking improves precision. Cold start handles missing data. Real-time signals improve session responsiveness. Feature stores align online and offline features. The final production design adds impression logging, safe model and index rollout, experimentation, and the caches, pipelines, and observability needed to run at scale with regional isolation.

By the final stage, the recommendation system is a coordinated serving platform: a front door, a candidate funnel drawing on multiple independently versioned sources, a multi-stage ranking pipeline, a profile layer, real-time session updates, a feature store, caches, impression logging, safe rollout machinery, and online/offline data pipelines working together.