Recommendation System — Trade-offs & Decisions

Should the system emphasize what is popular globally, or what is personally relevant for one user?

Problem

We need a default strategy for what the Recommendation Service returns. Pure popularity is easy to compute and cache, while deeper personalization demands more state and more online work.

Option A — Popularity-first

Use the Trending/Popularity Aggregator heavily. This is cheap, resilient, and good for anonymous or new users, but it produces repetitive results and under-serves niche interests.

Option B — Personalization-first

Lean on the User Profile Service, Candidate Generation, and Ranking Service for every request. This improves user fit, but raises latency, complexity, and sensitivity to missing data.

Decision

Why

Popularity gives the system a stable fallback and a broad discovery channel. Personalization adds the precision users expect after the platform has observed enough behavior.

Trade-offs

More popularity means more sameness. More personalization means more moving parts, more cache fragmentation, and more risk when profile or ranking components lag.

Should the final ranking decision happen entirely at request time, or should the system precompute as much as possible ahead of time?

Problem

We need to decide where ranking work should live. "Offline ranking" and "offline preparation" are easy to conflate, but they are different claims: offline systems can precompute features, embeddings, indexes, and even candidate lists — they do not typically make the final contextual ranking decision that has to reflect the current request.

Option A — Precompute-heavy

Precompute features, embeddings, and candidate indexes in the Batch Pipeline, and cache reusable outputs in the Recommendation Cache. This is cheap and predictable, but on its own cannot reflect session intent, current eligibility state, or rapid catalog changes.

Option B — Fully online ranking

Run the Ranking Service on every request using fresh profile and session context, with nothing precomputed. This is maximally flexible, but expensive and latency-sensitive, and it wastes cycles recomputing things (like embeddings or popularity tables) that rarely change per request.

Decision

Why

Offline systems are good at scanning history and publishing reusable assets. Online systems are good at reacting to the current user and current session — the final ranking decision is inherently a request-time decision, even though almost everything it reads was prepared ahead of time.

Trade-offs

The hybrid model is operationally richer: it requires managing feature versions, offline refresh cadence, index publication (see Candidate Index Publication), and online fallback behavior together, rather than reasoning about one simple pipeline.

Should the system rely mainly on precomputed assets, or should it update recommendations in real time after every user action?

Problem

We need a freshness strategy for changing user intent. Waiting for the next batch is cheaper; reacting instantly is more engaging.

Option A — Precompute-heavy

Let the Batch Pipeline and slower profile refresh jobs do most of the work. This simplifies the online path, but can make recommendations feel stale within the same session.

Option B — Real-time-heavy

Use the Real-Time Signal Processor aggressively so every major action influences the next request. This improves session responsiveness, but increases streaming cost and cache invalidation frequency.

Decision

Why

The stable long-term profile should come from durable history, while session intent should arrive through a smaller fast path with explicit scope and TTL.

Trade-offs

Too much precompute makes the experience stale. Too much real time makes the system expensive, noisy, and operationally fragile.

Should the system spend more time gathering signals for a slightly better answer, or answer faster with fewer features and fewer candidates?

Problem

The Recommendation Service sits on a user-facing request path, so more computation can hurt responsiveness even if it helps relevance.

Option A — Accuracy-first

Use more candidates, richer features, and more post-processing. This can improve ranking quality, but it expands p95 and p99 latency quickly.

Option B — Latency-first

Limit candidate count, reduce per-item features, and favor cached or precomputed assets. This keeps the UI fast, but may reduce personalization depth.

Decision

Why

Users experience slow recommendations as broken pages. A slightly less perfect list delivered on time often beats a theoretically better list that arrives too late.

Trade-offs

Tighter budgets demand more engineering effort in caching, precomputation, and feature-store design. Looser budgets increase abandonment and fallback risk.

Worked numbers (illustrative, using the Complete Design baseline of about 174,000 requests/sec peak and 400 candidates/request): widening to 800 candidates doubles peak ranking load from about 69.6 million to about 139.2 million candidate scores/sec, before any change in per-item feature cost. The relevance gain from the extra candidates has to be worth that doubled fleet cost and the longer p95/p99 tail.

How fresh should recommendations be, and how much infrastructure cost is acceptable to achieve that freshness?

Problem

Users benefit when recommendations react quickly to new activity, but every step toward freshness tends to increase compute, storage, and invalidation load.

Option A — Higher freshness

Push more event processing into real time, shorten cache TTLs, and refresh profile state frequently. This improves responsiveness, but raises streaming, storage, and ranking costs.

Option B — Lower cost

Rely more on batch updates and longer-lived caches. This is cheaper and simpler, but recommendations can feel stale during fast-moving sessions or news cycles.

Decision

Why

Not all recommendation surfaces need the same freshness. Home feed, autoplay, and breaking trends justify faster updates than long-tail archive browsing.

Trade-offs

Selective freshness adds configuration complexity, but it avoids paying real-time costs for every field and every surface.

Should the system pack in more user and item features, or keep feature lookup lean so the serving path stays fast and predictable?

Problem

Ranking quality often improves when the system considers more user, item, and context attributes. But every extra feature can add computation, storage, and potential inconsistency.

Option A — Feature-rich serving

Expose lots of user-history windows, item stats, session context, and cross-entity joins. This can help ranking, but it enlarges payloads and lookup cost.

Option B — Lean serving

Keep only a compact set of features on the hot path and precompute everything else offline. This protects latency, but may flatten personalization depth.

Decision

Why

The serving tier should read the minimum information needed to make a good decision. Wider feature sets are acceptable only when their value is clear and their latency cost is controlled.

Trade-offs

A leaner hot path improves speed and reliability, but it demands better offline preparation and more explicit prioritization of which signals truly belong online.

How much should the system show what the world is watching or buying right now versus what one user personally tends to prefer?

Problem

We need a blend between shared cultural moments and individual preference. Too much global trend can erase personalization; too much history can trap users in narrow loops.

Option A — Heavier global trends

Lean on the Trending/Popularity Aggregator so users see what is fresh, viral, or commercially important. This supports discovery, but can dominate niche interests.

Option B — Heavier user interests

Lean on User Profile Service history and ranking signals so every surface feels tailored. This improves fit, but can create sameness and feedback-loop concentration.

Decision

Why

Users often want a mix: familiar relevance, fresh discovery, and timely global moments. The architecture should let ranking combine those sources instead of forcing one winner.

Trade-offs

More trend weight improves freshness and social relevance. More profile weight improves fit. Both need guardrails so the feedback loop does not collapse into either generic popularity or an overly narrow bubble.

Should Candidate Generation rely on one retrieval technique, or fan out to several independent sources and merge their output?

Problem

A single retrieval technique is simpler to build and operate, but it carries a systematic blind spot: no one technique covers cold-start users, cold-start items, and steady-state relevance equally well.

Option A — Single source

Retrieve from one technique only, for example collaborative retrieval. One contract, one failure mode, simple to reason about — but it fails hard exactly where that technique is weak, with no other source to fall back on.

Option B — Multi-source

Fan out to several sources — popularity, collaborative, embedding/ANN, rule-based — and merge their output (see Candidate Merge & Deduplication). Improves coverage and resilience, but adds a real merge and score-normalization problem, and each source needs its own contract.

Decision

Why

No single retrieval technique covers cold start, discovery, and steady-state relevance equally well; independent sources also give the system a natural fallback when one source degrades or times out.

Trade-offs

More sources mean more operational surface area, more contracts to maintain, and a harder merge/normalization problem — in exchange for coverage and resilience a single source structurally cannot provide.

Once the system uses embedding-based retrieval, should nearest-neighbor search be exact or approximate?

Problem

Exact nearest-neighbor search guarantees the truly closest items, but its scanning cost grows with catalog size and eventually dominates the retrieval latency budget.

Option A — Exact nearest neighbor

Scan and compare against every embedding directly. Correct by construction, but computationally inappropriate once the catalog is large enough that a full scan cannot fit the retrieval budget — though it remains reasonable for genuinely small catalogs or offline evaluation.

Option B — Approximate (ANN) index

Use an approximate nearest-neighbor index. Trades a small, tunable amount of recall for large latency and memory savings (see Embedding / ANN Retrieval).

Decision

Why

The request-time budget cannot absorb exact-search cost at meaningful scale; the recall loss from a well-tuned ANN index is usually far cheaper than the latency it saves.

Trade-offs

Approximate retrieval occasionally misses a true nearest neighbor and needs its own freshness and version-compatibility discipline; exact search avoids that risk but does not scale to a large catalog on the request path.

Should candidate lists be generated ahead of time in batch, or generated fresh for every request?

Problem

Precomputed candidate lists are cheap to serve but reflect the world as of the last batch run. Fully online generation reacts to the current session, but repeats retrieval work on every single request.

Option A — Precomputed candidate lists

Generate candidate lists in batch and serve them as-is. Cheap, but cannot reflect same-session signals or current eligibility state.

Option B — Fully online candidate generation

Run source fan-out and merge fresh for every request. Reacts to session context immediately, but repeats work that changes rarely between requests.

Decision

Why

This mirrors the split in Offline Preparation vs Online Ranking: heavy asset-building work belongs offline, request-specific composition belongs online.

Trade-offs

This still leaves per-request orchestration cost that a fully precomputed list would avoid, in exchange for staying responsive to session context and current eligibility state.

Should ranking score every retrieved candidate with one model, or narrow the set further with a cheaper pre-ranker first?

Problem

A single expensive model scoring every retrieved candidate is simple, but its cost scales directly with however many candidates retrieval hands it.

Option A — Single-stage ranking

One model scores the whole candidate set. Simpler to build, version, and monitor — but caps how rich that model can afford to be once candidate volume grows.

Option B — Multi-stage ranking

A cheap pre-ranker trims the set first, then a richer main ranker scores what remains (see Multi-Stage Ranking). Affords a more sophisticated main model, but adds another model to train, version, and monitor — and the pre-ranker can discard a candidate the main ranker would have scored well.

Decision

Why

This follows the same earned-complexity principle used throughout this system: add the pre-ranker stage only once it solves a real budget problem, not preemptively.

Trade-offs

Multi-stage ranking risks losing a good candidate at the pre-ranker step; single-stage risks running out of latency budget before the main model can be as rich as quality goals want.

Worked numbers (illustrative, same baseline): if a cheap pre-ranker cuts 400 candidates to 100 for the main model, the main model scores about 17.4 million items/sec at peak instead of about 69.6 million, a 4× reduction. The pre-ranker still scores all 400 per request, so its own cost must be small enough for the saving to be real, and a good candidate dropped at that step is never recovered.

Should eligibility rules be enforced once, early in the pipeline, or rechecked again right before the response is sent?

Problem

Checking eligibility only once is cheaper. Checking it only at the end is always correct against current state, but wastes ranking budget on items that turn out to be ineligible.

Option A — Pre-filter only

Drop ineligible items early, before ranking. Cheap, but vulnerable to any staleness between the pre-filter check and response time — an index or cache that has since gone stale can leak an ineligible item through.

Option B — Post-filter only

Check eligibility only right before the response is assembled. Always correct against current state, but ranking spends full effort scoring items that end up filtered out anyway.

Decision

Why

Pre-filtering alone cannot guarantee correctness once indexes or caches are involved, since eligibility state can change between the pre-filter check and response assembly.

Trade-offs

Running both costs a small amount of duplicated work, in exchange for closing the correctness gap that either filter alone leaves open.

Should the durable user profile update on a batch cadence, or stream continuously as events arrive?

Problem

Batch updates are predictable and cheap, but the durable profile can lag a full batch cycle behind actual behavior. Fully streaming updates keep it continuously current, at meaningfully higher cost.

Option A — Batch-only

Recompute the durable profile on a scheduled cadence. Simple and cost-predictable, but not responsive within a single session.

Option B — Fully streaming

Update every field continuously as events arrive. Always current, but expensive infrastructure for fields that do not need sub-hour freshness.

Decision

Why

Most durable-profile fields — long-window interests, category affinity — do not need to update on every event. Reserving streaming infrastructure for the narrow session-overlay case avoids paying continuous cost for a freshness need that only exists for a small slice of the profile.

Trade-offs

This two-speed model adds the operational complexity of maintaining two update paths and merging them consistently at read time, in exchange for spending streaming cost only where it earns its keep.

Should the ranking path fetch the user's entire profile object, or fetch only the specific features a given request needs?

Problem

Fetching the whole profile is simple, but larger payloads travel across the network even when most fields go unused. Fetching only needed features is leaner, but requires the caller to know in advance which features a given ranking pass actually needs.

Option A — Fetch the full profile

One lookup returns everything. Simple, but the profile object tends to keep growing as more fields get added "just in case."

Option B — Fetch only needed features

Request specific feature bundles through the Feature Store. Smaller payloads and a cleaner contract, but adds a step of knowing what to ask for.

Decision

Why

A profile record that keeps absorbing every feature anyone might someday want stops being a fast, predictable lookup and starts becoming a second, poorly-versioned feature store.

Trade-offs

This requires clearer ownership boundaries between what belongs in the profile and what belongs in the feature store — a design-discipline cost in exchange for keeping both lookups fast and legible.

Should a personalized recommendation list be cached and reused, or recomputed on every request?

Problem

Caching is cheaper for repeat requests within a short window, but risks a visibly stale or repetitive list if TTLs run too long. Recomputing every time always reflects the latest state, but at full cost.

Option A — Cache personalized lists

Serve a cached list within a TTL window. Cheap and fast, but risks feeling stale or repetitive if not invalidated on real signals.

Option B — Recompute every request

Always reflects the latest profile, session, and eligibility state, but spends full retrieval-and-ranking cost on every request, including near-duplicate ones.

Decision

Why

Most of the value of recomputation is captured by reacting to real signals, not by recomputing on a fixed clock; short TTLs bound the staleness risk without paying full cost on every request.

Trade-offs

This requires the invalidation logic to actually work — a caching layer with no real invalidation path just accumulates staleness risk instead of avoiding it.

Worked numbers (illustrative, same baseline): at about 174,000 requests/sec peak, every 10 percentage points of cache hit rate removes about 17,400 full recomputations/sec. A 50% hit rate leaves about 87,000/sec to recompute and an 80% rate leaves about 35,000/sec. A longer TTL raises the hit rate but delays recommendations that should react to a fresh session signal, which is why invalidation triggers matter as much as the TTL.

Once a new ranking or retrieval model is ready, should it replace the current model quickly to capture its benefit, or roll out slowly?

Problem

A fast replacement captures quality improvements sooner, but skips the shadow and canary stages that catch infrastructure problems and quality regressions before they reach all traffic.

Option A — Fast full replacement

Deploy the new model to all traffic as soon as it passes offline validation. Fastest path to any real benefit, but offline metrics do not always predict live behavior.

Option B — Fully staged rollout, no exceptions

Always pass through shadow, canary, and gradual expansion (see Model Deployment & Rollout). Safest, but slows down even low-risk, well-validated changes.

Decision

Why

The risk a bad rollout poses — silently degraded recommendations returning 200s — usually outweighs the cost of a few extra days at a smaller traffic share.

Trade-offs

Staged rollout delays the benefit of a genuinely good model change, and requires maintaining shadow/canary infrastructure even for changes that turn out to be low-risk.

Should the system always show its highest-confidence recommendations, or deliberately reserve room to show less-certain or under-exposed items?

Problem

Pure exploitation maximizes short-term measured relevance, but starves new items and the model itself of feedback on anything outside what it already favors.

Option A — Pure exploitation

Always show the highest-scored items. Maximizes near-term measured relevance, but the system can never learn whether an under-exposed item would actually perform well.

Option B — Heavy exploration

Reserve a large share of exposure for less-certain items. Improves coverage and long-term learning signal, but visibly lowers near-term relevance if overdone.

Decision

Why

This is the serving-side counterpart to the correctness problem in impression/exposure logging: without some exploration, the system cannot tell whether an item's low engagement reflects genuine disinterest or simply a lack of exposure.

Trade-offs

Exploration has a measurable near-term relevance cost; the right amount depends on how much long-term coverage and learning value the product needs against how much near-term relevance it can afford to spend.

Should reads across profile, feature, and candidate-index data be strictly consistent with the latest write, or is eventual freshness acceptable?

Problem

Strict consistency guarantees every read reflects the latest write, but usually requires synchronous coordination that conflicts with the serving path's low-latency budget.

Option A — Strict consistency

Every read reflects the latest write everywhere. Guarantees correctness at read time, but the coordination cost is difficult to reconcile with a millisecond-scale serving budget across many independent data sources.

Option B — Eventual freshness

Reads may lag the latest write by a bounded window. This is what allows the online path to stay fast and horizontally scalable, at the cost of a known, bounded staleness.

Decision

Why

Most personalization signals tolerate a bounded delay without materially hurting relevance, while the serving path's latency budget cannot tolerate synchronous cross-store coordination on every request.

Trade-offs

Eventual freshness requires being explicit about which few things truly need a current-state recheck versus which can tolerate a staleness window — blurring that line is how stale data quietly becomes a correctness bug instead of a relevance trade-off.

Should the recommendation platform run as one global deployment, or split into independently operating regional deployments?

Problem

A single global deployment is operationally simpler, but every request may cross regional network boundaries, and a single region's failure has global blast radius.

Option A — Single global deployment

One set of artifacts and services serving all regions. Simple to reason about and operate, but concentrates both latency and failure risk.

Option B — Independent regional deployments

Each region owns its own indexes, feature replicas, and model versions. Bounds blast radius and keeps latency local, but requires fencing so regions never contend for the same write path, and multiplies the number of artifacts to keep compatible.

Decision

Why

A single global deployment concentrates risk and adds cross-region latency to every request; but regional isolation only delivers its resilience benefit if regional ownership and fencing are actually built, not left implicit.

Trade-offs

Regional deployments multiply operational surface area — more index copies, more feature replicas, more model versions to keep compatible — in exchange for genuine fault isolation and lower latency.