'%20stroke-width='24'%20stroke-linecap='round'%20stroke-linejoin='round'%3e%3cpath%20d='M196%2058%20C196%2042%20184%2032%20168%2032%20H112%20C84%2032%2062%2054%2062%2082%20C62%20108%2082%20126%20110%20126%20H148%20C176%20126%20194%20144%20194%20170%20C194%20198%20172%20224%20144%20224%20H88'/%3e%3c/g%3e%3ccircle%20cx='196'%20cy='58'%20r='14'%20fill='%236366F1'%20stroke='%23FFFFFF'%20stroke-width='6'/%3e%3ccircle%20cx='88'%20cy='224'%20r='14'%20fill='%23EC4899'%20stroke='%23FFFFFF'%20stroke-width='6'/%3e%3ccircle%20cx='62'%20cy='82'%20r='14'%20fill='%238B5CF6'%20stroke='%23FFFFFF'%20stroke-width='6'/%3e%3cpath%20d='M121%20102%20L121%20150%20L160%20126%20Z'%20fill='%23FFFFFF'%20opacity='0.98'/%3e%3c/svg%3e){kind=small}
System Design · Trade-offs · ~30 min read
Gaming Leaderboard System Design — Trade-offs and Decisions
Complete Design walked through one coherent architecture for FlowRank's leaderboard behind Arena Clash. Nothing about that architecture was the only option — every piece of it was a choice made against a specific workload (roughly 5,000 active boards, millions of players, high-frequency score updates, a real-time staleness budget measured in seconds, not milliseconds-everywhere). This page works through 28 of those choices individually: the genuine alternatives, what each one actually costs in correctness, latency, or operational complexity, the baseline recommended for this workload, and — just as important — the specific condition that would flip the recommendation. A few of these decisions are close to universal for any serious real-time leaderboard; most are not, and treating them as if they were is itself a design mistake.
Decision 01
Source of Truth: Database Aggregate vs Redis-Only State vs Event Log + Projections
Every other decision on this page depends on what the system treats as authoritative when things disagree — a crashed cache, a replayed event, a disputed score. The source-of-truth model is the one decision that is genuinely upstream of all the others.
| Option | Favors | Costs |
|---|---|---|
| Database aggregate as truth (every score lives in a relational/document store; rank is derived from it on demand or via periodic refresh) | Familiar transactional guarantees; easy to audit 'what is Ava's best score right now' with a single row read; backups and point-in-time recovery are off-the-shelf | Deriving rank from a raw table means either an expensive query per read (Decision 11) or a refresh job that is itself a second source of truth in practice |
| Redis-only state (the sorted set itself is the only record of scores and ranks; nothing durable behind it) | Simplest possible system — one data structure, one place to look, lowest latency for both writes and rank reads | A cache/store that is also the only copy of data is not a cache, it's an under-durabilized database; losing that process (or a bad FLUSHALL, a botched migration) loses the entire leaderboard with no record to rebuild from — directly violates the never-trust-a-single-ephemeral-copy posture this workload needs at millions-of-players scale |
| Event log as truth, with Redis (and/or SQL) as derived projections | The event log is append-only and replayable, so Redis state, SQL state, friends boards, and any future projection are all rebuildable from the same durable history; matches the idempotent-event-application invariant directly, since idempotency is a property of how the log is applied, not of the projection | Two moving pieces instead of one — the log itself and at least one materialized projection — and the projection can lag the log by a small, bounded amount, which has to be an explicit, communicated staleness budget rather than an assumed zero |
This is not a three-way tie. A pure database-of-record with no fast ranking structure cannot meet the latency targets this workload implies once a board has more than a trivial number of entries; a Redis-only system cannot meet the durability and auditability bar a real product needs the moment a score is disputed, a season needs recovery, or a player asks 'why did my rank change.' The event log plus projections model is the only one of the three that satisfies both constraints at once, and it is the model Complete Design and Architecture Deep Dive both assume.
Decision 02
Ranking Data Structure: SQL Ordered Index vs Redis Sorted Set vs Specialized Ranking Service
Given that the projection layer needs to answer rank and top-N queries fast, what structure actually answers them?
| Option | Favors | Costs |
|---|---|---|
SQL ordered index (ORDER BY score DESC, rank via COUNT(*) WHERE score > x) | No new infrastructure; transactional writes and reads in the same system; works fine for small boards or low query volume | Computing a live rank this way is effectively a full or index range scan on every read; at high-frequency updates and millions of players, the index itself becomes a write hotspot and rank queries scale poorly with board size |
| Redis sorted set (ZADD/ZSCORE/ZRANK family) | Purpose-built for exactly this access pattern — O(log N) insert, O(log N) rank, O(log N + M) range fetch for top-N or windows; in-memory speed matches the real-time latency budget directly | It is a bounded-correctness tool, not an unconditionally correct one at every scale — see Decisions 05 and 17 for where its native tie handling and single-key model break down |
| Specialized ranking service (purpose-built rank-index service or custom skip-list/Fenwick-tree service, possibly sharded with its own rank-merging logic) | Can be built to solve exactly the problems Redis sorted sets don't — true tie-break ordering as a first-class key, exact global rank across shards (Decision 17) — without bolting workarounds onto a general-purpose structure | Significant build and operational cost: it is new infrastructure the team owns end-to-end, with its own failure modes, instead of a widely-understood, pre-built primitive |
Decision 03
Score Policy: Highest Score vs Cumulative Points vs Fastest Time
'A new score comes in — what do we do with it' has no single universal answer. The policy is a product decision baked into the leaderboard's identity (game + mode + season + scope), and it changes what 'never overwrite a valid best' even means.
| Policy | Semantics | Trade-off |
|---|---|---|
| Highest score wins | Each event carries a candidate score; the stored best is max(stored, new) and anything lower is rejected without mutating state (Test D: Dev's late 9,200 after an accepted 9,700 is rejected, 9,700 stands) | Simple and intuitive for score-attack modes, but needs a precise definition of 'personal best' per player per board, and interacts directly with correction policy (Decision 21) when a stored best turns out to be invalid |
| Cumulative points | Each verified event adds to a running total (Test J: +100, +50, duplicate +50 → 150, not 200) — every event matters, not just the best one | Idempotent dedup is load-bearing in a completely different way here: a duplicate delivery of an additive event, if not deduped, silently inflates the total forever rather than just failing to improve a max — this policy has zero tolerance for dedup bugs compared to highest-score, where a duplicate of a lower score is merely a no-op |
| Fastest time wins | Lower is better; sort ascending instead of descending, and 'best' is min(stored, new) (Test K) | Everything above — storage, comparator, tie-break — has to be aware of sort direction; a codebase that silently assumes 'higher is better' everywhere is a real source of bugs when a fastest-time board is added later |
These are not three variations of one algorithm; they are three different reducers applied to the same event stream, and the leaderboard identity model (game+mode+season+scope) is exactly what lets different boards run different policies without the ranking service needing a single hardcoded rule.
Decision 04
Tie Handling: Shared Ranks vs Deterministic Unique Order vs Timestamp Tie-Break
Ben and Chloe both have 9,500 points. Do they share rank #2, or does one of them get #2 and the other #3? This single choice determines whether 'the rank' is even a well-defined, stable concept.
| Model | Favors | Costs |
|---|---|---|
| Shared ranks (competition ranking: tied players share a rank, next rank skips — e.g. two players tied at #2, next player is #4) | Matches how many real-world leaderboards and sports standings are communicated; intuitively fair — equal performance, equal rank | Rank is not a stable, unique position any more — 'who is #2' can have more than one correct answer, which complicates pagination (Decision 28), rank-window queries (Test F), and any UI that assumes one player occupies one row |
| Deterministic unique order with an explicit, published tie-break rule | Every player has exactly one rank, always — makes pagination, rank windows, and 'your current position' queries unambiguous and stable across repeated reads of the same state | Needs a tie-break rule that is itself fair and explainable, not an arbitrary artifact of insertion order or storage internals |
| Timestamp tie-break specifically (earlier achievement time wins the tie) | Rewards being first to reach a score, which is an intuitive and explainable fairness rule; combined with stable player ID ascending as a final tiebreaker, produces total, deterministic ordering with zero ambiguity, as shown in Test B (Eli ties Ben/Chloe's old 9,500 but arrives later, landing at #4 behind both) | Requires achievement timestamps to be trustworthy and precise enough to order close ties correctly, and requires every layer of the stack (Redis encoding, SQL query, API response) to apply the exact same rule, not three subtly different ones |
Decision 05
Tie Storage: Redis Equal-Score Lex Order vs Composite Score Encoding vs Secondary Ranking Index
Deciding the tie-break rule (Decision 04) is only half the problem. The sorted set has to actually store and enforce that rule, and Redis's own default behavior for equal scores is not the rule this leaderboard needs.
| Approach | What it does | Why it does or doesn't work here |
|---|---|---|
| Rely on Redis's default equal-score ordering (lexicographic by member ID) | When two members have the identical score, ZRANGE orders them by member-string comparison | This is the wrong tie-break entirely — it orders by player-ID string, not by who achieved the score earlier. Ben and Eli tied at 9,500 would be ordered by comparing p-ben and p-eli as strings, which has nothing to do with the 10:01:00-vs-10:06:00 achievement gap Test B actually requires. Relying on this default silently produces a leaderboard that looks correct (it IS sorted and deterministic) but enforces the wrong fairness rule |
| Composite score encoding (combine the real score with an inverted timestamp into a single sortable value) | Store a single numeric score such that comparing it alone reproduces 'higher score, then earlier timestamp' ordering — e.g. score * TIME_SCALE + (TIME_SCALE - timestamp_bucket), sized so the timestamp component never crosses a score boundary | Keeps the single sorted-set lookup fast and simple once encoded correctly; the cost is a more complex encode/decode step, a fixed precision ceiling on the timestamp component, and the player-ID final tiebreaker still needs a second-level comparison when two players have genuinely identical encoded values (extremely rare once sub-second timestamp precision is included, but not impossible) |
| Secondary ranking index (keep raw score in the sorted set for fast top-N, resolve exact tie order against a secondary structure — e.g. a SQL query or a small in-memory comparator — only when two members' scores are equal) | The sorted set stays simple (plain scores); tie resolution logic lives in one explicit, testable comparator rather than being smuggled into a numeric encoding | Adds a second lookup specifically on the tie path, and that comparator has to be invoked consistently everywhere rank is computed or displayed, not just in one code path |
textComposite score encoding sketch (conceptual, not a specific language)
# Goal: one sortable number that reproduces "higher score wins, earlier timestamp breaks ties"
# TIME_SCALE must exceed the largest timestamp bucket value used, so the
# timestamp term never bleeds into the score term.
encoded = (score * TIME_SCALE) + (TIME_SCALE - timestamp_bucket)
# Two equal scores: the one with the SMALLER timestamp_bucket (earlier time)
# yields the LARGER encoded value, so ZREVRANGE still surfaces it first.
# Player-ID ascending remains the final, explicit tiebreaker for the
# (very rare) case of identical encoded values.Decision 06
Write Processing: Synchronous Ranking Update vs Asynchronous Projection
When a verified score event is accepted, does the caller wait for the Redis projection to be updated before getting a response, or does acceptance happen first and the rank update follow asynchronously?
| Approach | Favors | Costs |
|---|---|---|
| Synchronous update (projection write happens inline before the client gets an acknowledgment) | The client's own next read is guaranteed to see its own just-submitted score reflected — a strong, easy-to-reason-about consistency story for the submitting player specifically | Couples submission latency to projection latency; a slow or momentarily degraded Redis directly slows down score acceptance, and a Redis outage can block the entire ingestion path rather than just the read path |
| Asynchronous projection (durable append is the acknowledgment; the Redis projection update follows as a decoupled consumer of the event log) | Ingestion latency and availability depend only on the durable log, not on the projection; projection consumers can be scaled, restarted, or rebuilt independently without touching the write path at all | Introduces a small, bounded lag between 'event accepted' and 'rank reflects it' — has to be an explicit staleness budget (seconds, not an assumed zero), and the client's own just-submitted score may briefly not show up in its own next read unless the client UI accounts for that |
Decision 07
Delivery Guarantee: At-Most-Once vs At-Least-Once with Idempotency
Network and client retries make 'exactly once' an unavailable option in practice. The real choice is between risking silently dropped score events and risking duplicate delivery handled correctly at the application layer.
| Guarantee | Favors | Costs |
|---|---|---|
| At-most-once (fire and forget; no retry on uncertain delivery) | Simplest client and simplest server — nothing to dedup | Any network blip, timeout, or ambiguous response silently drops a legitimate score event, with no mechanism to detect or recover the loss — unacceptable for a system whose entire value proposition is a trustworthy record of achievement |
| At-least-once with idempotent application | Retrying on any uncertainty (timeout, 5xx, connection drop) is always safe to do, because the server-side event ID dedup makes a duplicate delivery a no-op rather than a double-count (Test C: replaying evt-101 has no effect) | Requires a real idempotency mechanism — a durable, indexed dedup key per event ID, checked before the event is applied — not just 'the client usually doesn't retry twice' |
This decision is not really optional. Networks drop packets and clients legitimately retry on ambiguous timeouts regardless of what the server intends, so 'exactly once' is a claim about the network, not something the server can promise by itself. The only question the system actually controls is whether duplicates are detected and absorbed (at-least-once + idempotency) or left to silently corrupt state (at-most-once, or at-least-once without dedup).
Decision 08
Event Transport: Direct Service Call vs Durable Broker/Log
Once a score event is verified and accepted, how does it travel from ingestion to the durable log and on to the projection consumers (Decision 06)?
| Option | Favors | Costs |
|---|---|---|
| Direct service call (ingestion service calls the projection updater synchronously over RPC/HTTP) | Fewest moving parts; easy to trace one request end to end | Couples ingestion and projection availability directly — this is the same coupling Decision 06 already argues against — and gives up replayability: if a consumer was down, there is no durable record of what it missed unless something else is logging it anyway |
| Durable broker/log (events are appended to a durable, ordered, replayable log; projection consumers subscribe and process independently) | Decouples producer and consumer lifecycles entirely; gives every consumer (Redis projector, SQL writer, anti-cheat pipeline, analytics) its own independent position in the stream and its own replay capability after an outage or a bug fix; directly enables the recovery strategies in Decision 24 | Operational investment in running and monitoring the broker/log itself; introduces the at-least-once-delivery reality that makes Decision 07's idempotency non-negotiable |
Decision 09
Score Updates: Per-Event Updates vs Bounded Batching/Coalescing
Does each verified event trigger its own immediate projection write, or are multiple events for the same board (or even the same player) coalesced into a batched update?
| Approach | Favors | Costs |
|---|---|---|
| Per-event updates | Lowest possible lag between an event being accepted and the projection reflecting it (bounded only by the async hop itself); simplest mental model — one event in, one projection write out | At genuinely high-frequency update rates on a single hot board, this is one write operation per event with no amortization, which is exactly the write-volume pattern that stresses a single sorted-set key (Decision 18) |
| Bounded batching/coalescing (accumulate a short, fixed window of events — e.g. tens to low-hundreds of milliseconds — and apply them as one batched update per affected board) | Amortizes write overhead across many events, directly easing pressure on a hot board's single key; when the same player improves their score twice within one window, only the final best needs to be written, not both | Adds a small, bounded, deliberate lag on top of the async hop already accepted in Decision 06 — has to stay inside the stated staleness budget, and the batch window itself is a tuning dial that trades latency against write amortization |
Decision 10
Read Consistency: Strong Current Rank vs Bounded-Stale Materialized Rank
'Real time' is not the same claim as 'zero latency.' This decision makes the staleness budget explicit instead of leaving it implied.
| Model | Favors | Costs |
|---|---|---|
| Strong current rank (every read recomputes or re-fetches rank from the most recently written state, with no tolerated lag) | No staleness to reason about or explain to users | Forces every read to be tightly coupled to the latest write, which conflicts directly with the decoupled async projection model (Decision 06) that this workload needs for ingestion availability and throughput |
| Bounded-stale materialized rank (reads are served from a projection that is at most N seconds behind the durable log, with N explicitly stated and monitored) | Reads stay fast and decoupled from ingestion load entirely; the staleness bound is a concrete, testable number rather than a vague 'eventually' — and can be surfaced to clients if useful (e.g. a 'last updated Ns ago' indicator) | Requires actually measuring and alerting on projection lag, not just assuming the async hop stays small; a degraded projection consumer can silently widen the staleness window past what was promised unless it's monitored |
Decision 11
Query Patterns: On-Demand SQL Rank vs Precomputed Board Index
A player opens the leaderboard screen. Something has to answer 'what is my rank' and 'who's near me' right now — computed fresh each time, or read from something already maintained in rank order?
| Strategy | Favors | Costs |
|---|---|---|
On-demand SQL rank (SELECT COUNT(*) FROM scores WHERE board_id = ? AND score > ? per request, as introduced in Decision 02) | No separate index to maintain — the query reflects the table's current state exactly, with standard transactional guarantees | Cost scales with the number of rows ahead of the player on that board; at millions of players and frequent requests, this becomes a hot, expensive query pattern competing with every other read |
| Precomputed board index (the Redis sorted-set projection from Decision 02, kept current by the write path from Decision 06) | Rank and range queries are O(log N) regardless of request volume, decoupled from the underlying event volume on the write side | The index is a second thing that must be kept consistent with the source of truth within the stated staleness budget (Decision 10), and it needs its own rebuild path if it's ever found to have drifted (Decision 24) |
Decision 12
Friends Leaderboard: On-Demand Friend-Set Sorting vs Precomputed Personalized Boards
A friends leaderboard is not a filtered view of the global top-N — Test E makes this concrete: Ava's friends (Ben, Dev, Eli) rank among themselves as #1/#2/#3, which has no relationship to their global ranks (#2/#4/#5). Filtering the global top-N for friends would simply return an empty or wrong result whenever none of a player's friends are near the top.
| Strategy | Favors | Costs |
|---|---|---|
| On-demand friend-set sorting (fetch scores for exactly the player's friend set, sort that subset directly, at request time) | Always correct by construction — it computes rank over exactly the right subset, matching Test E's requirement directly — and needs no precomputed storage per friend-group | Cost scales with friend-list size per request; for a player with a very large friend/social graph, this is a non-trivial multi-key fetch-and-sort on every view |
| Precomputed personalized boards (maintain a small materialized sorted set per player, containing just their friend set, updated as friends' scores change) | Fast, O(log N) reads over a small N, same shape as the global board's access pattern | One small sorted set per player instead of one per global board is a very different storage footprint at millions of players; every friend's score change has to fan out to every board that includes them as a friend, which multiplies write amplification by average friend-list size |
Decision 13
Top-N Cache: TTL vs Versioned Invalidation vs No Cache
The global top-N for a popular board is read far more often than it changes meaningfully. Whether and how that read is cached in front of the projection is its own decision.
| Strategy | Favors | Costs |
|---|---|---|
| No cache (every top-N read hits the Redis projection directly) | Simplest; the projection is already fast, so this is a reasonable default for most boards | On a genuinely hot board, top-N reads can still add meaningful load on top of the write traffic already hitting the same structure |
| TTL-based cache (cache the rendered top-N for a short fixed duration, e.g. low single-digit seconds) | Simple to implement and reason about; bounds cache staleness to a known, fixed window | The TTL window is a blunt instrument — it's either too short to meaningfully reduce load or too long relative to the staleness budget (Decision 10), and there's no way to invalidate early even when a cheap signal (a version bump) says the cached value is now stale |
| Versioned invalidation (cache the top-N alongside a version/sequence number from the write path; invalidate precisely when that version changes, not on a timer) | Cache correctness is tied to actual state changes rather than a guessed time window — never serves data staler than the true last update, and can safely cache longer when nothing has actually changed | Requires the write path to expose a cheap, reliable version signal, and requires the cache layer to check it on every read, which is a small amount of extra coordination compared to a bare TTL |
Decision 14
Real-Time Transport: Polling vs SSE vs WebSocket
Rank changes need to reach a watching client inside the staleness budget established in Decision 10. The transport that carries those updates has its own trade-offs, independent of how fast the server-side projection itself is.
| Transport | Favors | Costs |
|---|---|---|
| Polling (client periodically requests current state) | Simplest client implementation; works through virtually any network/proxy setup with no special support needed | Either wastes requests when nothing changed, or misses intermediate updates between polls (which directly interacts with Decision 15's delta/snapshot recovery problem); poll interval vs server load is a constant tuning fight |
| Server-Sent Events (one-way server-to-client push over a long-lived HTTP connection) | Simple to implement relative to full-duplex WebSockets; built-in browser reconnect semantics; well-suited to the mostly one-directional nature of rank-change notifications | One-way only — fine for push notifications, but a client that also needs to send frequent messages back needs a second channel |
| WebSocket (full-duplex persistent connection) | Lowest-latency two-way channel; natural fit if the same connection is also used for other bidirectional gameplay/social features | More connection-management complexity (heartbeats, reconnect/backoff, server-side connection state at scale) than SSE for a feature that is fundamentally one-directional |
Decision 15
Feed Recovery: Delta Replay vs Full Snapshot vs Hybrid
A client reconnects after a gap and reports it last saw version 42, while the server is at version 46 (Test I). How does the client get back to a correct state?
| Strategy | Favors | Costs |
|---|---|---|
| Delta replay only (server resends the missed deltas, versions 43 through 46) | Minimal bandwidth — only the actual changes are sent | Only works if the server still has those deltas available (a bounded replay buffer); a gap larger than the buffer, or a client so far behind the buffer no longer covers it, has no valid delta path |
| Full snapshot only (server always sends complete current state on reconnect, regardless of gap size) | Always correct, regardless of how large the gap is or how long the client was disconnected | Wasteful for a genuinely small gap (e.g. one missed update) — sends the entire board state to fix a one-row difference |
| Hybrid (attempt delta replay when the gap fits the retained buffer; fall back to a full snapshot plus a fresh version marker when it doesn't) | Cheap in the common case (small gaps, which are the overwhelming majority of reconnects), correct in the uncommon case (large gaps, long disconnects, client restarts) | Requires both code paths to exist and be tested, plus a clear, explicit threshold for when to give up on delta replay and fall back to a snapshot |
Decision 16
Partition Strategy: By Board/Scope vs By Player vs Splitting a Single Massive Board
5,000 active boards across millions of players is a scale where 'just put it all on one Redis instance' stops being viable for at least some boards. The question is what the unit of partitioning should be.
| Strategy | Favors | Costs |
|---|---|---|
| Partition by board/scope (each game+mode+season+scope board is its own sorted-set key, and keys are distributed across nodes by consistent hashing of the board identity) | Matches the natural unit of ranking — a rank query never needs to span more than one key — and naturally spreads load across thousands of independent boards without touching the ranking semantics of any single one | A single extremely popular board (e.g. a flagship global season board) still lives on one key, so this strategy alone does not solve that board's own hot-key problem (Decision 18) |
| Partition by player (shard players across nodes, each node holding a slice of every board's membership) | Spreads write load for a given player's many board memberships onto one node, which can simplify some per-player operations | Breaks ranking locality badly — answering 'what is this board's rank for score X' now requires querying every player-shard and merging results, turning a native O(log N) sorted-set operation into a scatter-gather across every shard for every single rank or top-N query |
| Split a single massive board across multiple keys/shards | The only way to spread one board's write volume (not just the whole system's aggregate volume) across more than one node | Breaks the sorted set's native exact-rank guarantee directly — see Decision 17 — because a shard-local ZREVRANK only knows that shard's members, not the global position; this is not a free horizontal scale-out, it is a deliberate trade against exact global rank |
Decision 17
Exact Global Rank: Single Authoritative Sorted Index vs Distributed Rank Index vs Approximation
This is the single biggest limit of the Redis-sorted-set baseline at extreme scale, and it deserves to be named directly rather than glossed over: splitting one massive board across shards for write throughput breaks simple exact ZREVRANK, because a shard only knows its own members — shard-local rank is not global rank.
| Strategy | Favors | Costs |
|---|---|---|
| Single authoritative sorted index (keep the whole board on one key/node, accept that node's throughput ceiling) | Exact rank by construction — there is only one key, so ZREVRANK is correct by definition, no merging required | Caps that board's write and memory scaling to what one node can hold and process; this is exactly the ceiling that forces the question in the first place for a genuinely massive board |
| Distributed rank index (shard the board, but maintain a merge layer that can compute true global rank from per-shard positions and per-shard counts — e.g. by combining each shard's rank with the count of entries ahead of it on every other shard) | Preserves exact rank correctness while spreading storage and write load across multiple nodes | The merge computation is real engineering: it needs an up-to-date count of 'how many entries are ahead of this one' on every other shard, which is either a scatter-gather read on every rank query (latency cost) or a maintained cross-shard count structure (consistency and update-fan-out cost) — this is materially more complex than a single sorted-set key, which is exactly why Decision 02 flags a specialized ranking service as the escalation path here |
| Approximation (accept a bounded-error estimate of rank for very large boards — e.g. percentile bucket or sampled rank rather than an exact integer position) | Avoids the merge-layer engineering entirely; scales horizontally with no cross-shard coordination on the read path | The leaderboard can no longer answer 'you are exactly rank 1,482,003' truthfully — only something like 'you are in roughly the top 0.1%' — which is a real, user-visible product change, not just an implementation detail, and has to be an explicit, communicated choice |
Decision 18
Hot-Board Mitigation: Larger Node vs Cache/Read Replicas vs Repartition/Specialized Service
A sorted set is effectively single-threaded per key for writes — operations against one key execute one at a time, regardless of how many CPU cores the node has. A genuinely hot board (a flagship season, a viral event) will eventually hit that ceiling. There are escalating responses, not one fix.
| Mitigation | Solves | Does not solve |
|---|---|---|
| Vertical scaling (larger/faster node for that key) | Buys headroom cheaply and quickly for a board that's hot but not yet at the practical ceiling of single-key throughput | Does nothing for the structural limit — a single key is still processed by effectively one execution path no matter how much faster the hardware underneath it is; this delays the problem, it doesn't remove it |
| Cache and read replicas in front of the write-owning node (Decision 13's versioned top-N cache, read replicas for range/rank queries) | Offloads read traffic from the primary key entirely, which is often the majority of a hot board's total load (far more people check rank than submit a new score at any given moment) | Does nothing for write throughput — if the write volume itself (not reads) is the bottleneck, replicas and caches don't help, since all writes still funnel through the single primary key |
| Repartition the board (Decision 16's shard-a-single-board option) or move it to a specialized ranking service (Decision 02's escalation) | Actually increases write throughput ceiling, by spreading the board's own write volume across more than one node/key | Reopens the exact-global-rank question (Decision 17) — this is the most structurally invasive option and should be the last resort, not the first reflex |
Decision 20
Season Reset: Delete/Reuse Keys vs Versioned Immutable Season IDs
When S7 ends and S8 begins, what actually happens to the data? Test H is explicit about the requirement: S7 finalizes, S8 opens empty, and a late S7 event cannot mutate either board without a controlled correction.
| Strategy | Favors | Costs |
|---|---|---|
| Delete and reuse the same keys for the new season (clear the S7 sorted set, start writing new scores into the same key under an assumed 'current season' label) | Minimal storage footprint — no accumulation of old season keys | Destroys S7's own history the moment the key is cleared, directly conflicting with any historical/archival requirement (Decision 25); worse, a late-arriving S7 event has no distinct, addressable target any more — the key it would have landed on now holds S8 data, so 'late S7 event' and 'early S8 event' become indistinguishable without extremely careful, error-prone timing logic |
| Versioned, immutable season IDs (each season is its own addressable identity — game+mode+S7+scope vs game+mode+S8+scope — and a finalized season's board becomes immutable to ordinary writes rather than being deleted or reused) | A late S7 event has an unambiguous, still-addressable target (the S7 board specifically) and is rejected by the finalized-season immutability rule rather than accidentally landing in S8's fresh board; S7's data remains available for history, audits, and rewards long after S8 opens | Accumulates one board's worth of data per season per board identity over time, which is exactly the storage growth Decision 25 (archival) exists to manage |
Decision 21
Moderation: In-Place Overwrite vs Audited Correction/Rebuild
Ava's 9,800 score is invalidated by moderation and she has no earlier valid score, so she drops to unranked and everyone behind her moves up (Test G). How that correction is actually applied to the system matters as much as the outcome.
| Strategy | Favors | Costs |
|---|---|---|
| In-place overwrite (a moderator or admin tool directly mutates the stored score/rank entry to the corrected value) | Fast, immediate fix visible right away | Leaves no record of what the original value was, who changed it, or why — which is a serious problem the moment a correction is itself disputed, and it bypasses the event log entirely, meaning the log and the projection now disagree about what happened and a later replay/rebuild (Decision 24) would silently undo the correction |
| Audited correction via a versioned workflow (the moderation action is itself a durable, authored event — an invalidation or correction event, with actor, reason, and timestamp — appended to the same log, which then drives a projection update exactly like any other event) | The correction is itself part of the replayable history, so rebuilding projections from the log reproduces the corrected state, not the pre-correction one; the full chain (original score → invalidation → resulting rank change) is auditable after the fact, which matters enormously the moment a correction is challenged | Requires the moderation tooling to go through the same event-sourced path as ordinary gameplay events, rather than a quick direct-database-edit shortcut, which is more process for what might feel like a simple fix |
Decision 22
Anti-Cheat: Synchronous Strict Verification vs Asynchronous Quarantine/Review
Never trust a client-supplied score directly — but verification itself takes time and resources, and the question is whether that verification blocks the score from ever appearing on the board, or lets it appear provisionally while deeper checks run.
| Strategy | Favors | Costs |
|---|---|---|
| Synchronous strict verification (every score is fully checked — replay validation, anomaly checks, rate limits — before it is accepted at all) | A score never appears on the public board unless it has already passed every check; no window where an unverified or suspicious score is publicly visible | Adds the full verification cost to the latency of every single legitimate submission, and the deepest, most expensive checks (full replay validation against game telemetry, cross-referencing patterns across sessions) may simply not be feasible to run synchronously under submission-latency pressure |
| Asynchronous quarantine/review (lightweight synchronous checks gate initial acceptance; a score is provisionally visible while deeper asynchronous checks run, and a flagged score is quarantined/invalidated via the audited correction path from Decision 21 if it later fails) | Keeps submission latency low for the overwhelming majority of legitimate players; the expensive, thorough checks run without a latency budget constraining them | Opens a real window where a cheated score is publicly visible — on the board, potentially affecting other players' visible rank — until the deeper review catches and corrects it, which is a genuine, user-visible cost, not a hidden implementation detail |
Decision 23
Ranking Refresh: Every Score Event vs Thresholded/Personal-Best-Only Changes
Not every verified event actually changes a player's position on a highest-score board — a lower score that doesn't beat the stored best changes nothing about rank. Should the projection still process every event, or only ones that could plausibly move a rank?
| Strategy | Favors | Costs |
|---|---|---|
| Refresh on every verified event | Simplest logic — the projection consumer doesn't need board-policy-aware knowledge of whether an event could matter; uniform handling regardless of score policy (Decision 03) | Processes a meaningful share of events that provably cannot change anything (a highest-score board rejecting a lower score, Test D) as if they might, which is wasted projection work at genuinely high event volume |
| Thresholded / personal-best-only refresh (the projection only issues a sorted-set write when the event actually represents a new personal best for a highest-score board, or always for a cumulative/fastest-time board where every event matters by definition) | Skips projection writes for events that are guaranteed no-ops under the board's own score policy, directly reducing write volume on potentially hot boards | The 'does this event matter' check has to be policy-aware (Decision 03) and done correctly per board type — a bug here either silently drops a legitimate rank-changing event (treating it as a no-op when it wasn't) or does nothing but waste the optimization's point if implemented too conservatively |
Decision 24
Recovery: Database Snapshot vs Event Replay vs Combined Rebuild
When a projection is lost, corrupted, or found to have drifted from the log, how does it get back to correct?
| Strategy | Favors | Costs |
|---|---|---|
| Database snapshot only (restore the projection's backing store from a periodic backup/snapshot and call it done) | Fast to restore for a database that supports point-in-time snapshots | The snapshot is only ever as current as its last capture — any events between that capture and the failure are simply gone from the restored state unless something else replays them |
| Full event replay from the beginning (rebuild the projection by replaying the entire durable log from the start) | Always produces a provably correct projection, since it's derived fresh from the same source of truth everything else trusts | Replay time grows with total log length; for a board with a long, high-volume history, full replay-from-zero can take an unacceptably long time to restore service |
| Combined rebuild (periodic snapshot of the projection's own state plus replay of only the log tail since that snapshot) | Bounds rebuild time to roughly 'snapshot load time plus events since the snapshot,' independent of total history length — the same shape as the stock-exchange module's recovery pattern, applied here to the ranking projection instead of an order book | Needs the snapshot itself to be versioned/validated (a corrupt or stale snapshot needs a defined fallback to an older one plus more replay), and snapshot generation must not corrupt or stall the live projection while it's running |
Decision 25
Historical Data: Keep Live Index Forever vs Archive Finalized Boards
Once a season finalizes (Decision 20), its board stops changing under ordinary writes. Does it stay in the same live, fast-access tier indefinitely, or move somewhere cheaper?
| Strategy | Favors | Costs |
|---|---|---|
| Keep every season's live index forever (finalized boards stay in the same Redis/projection tier as active ones, indefinitely) | Instant access to any historical season's rank data with no extra retrieval step or latency | Memory cost (Decision 26) grows without bound as seasons accumulate, for data that by definition never changes and is read far less often than an active season's board |
| Archive finalized boards to cheaper, disk-based storage after a defined live-access window | Keeps the expensive, memory-resident tier reserved for boards that actually need fast, frequent access — active seasons — while finalized history remains fully available, just at a different latency tier | Historical queries against an archived season pay a higher latency cost than a live board would, and the archival process itself (move, verify, and only then evict from the live tier) has to be built and tested as carefully as any other data-migration path |
Decision 26
Cost/Performance: Memory-Heavy Real-Time Index vs Disk-Based Query Infrastructure
Redis sorted sets are an in-memory structure — correctness and speed come at the direct cost of holding every active board's full membership in RAM. This is a real, ongoing cost, not a one-time implementation detail.
| Approach | Favors | Costs |
|---|---|---|
| Memory-resident real-time index for all active boards and seasons | Consistently fast rank and range reads at the latency this workload's real-time budget requires, with no disk I/O on the hot path | Memory cost scales directly with total active-board membership, which is exactly why Decision 25's archival policy matters — unbounded growth here is a direct, recurring infrastructure cost |
| Disk-based query infrastructure for the same access pattern | Much cheaper per unit of stored data, and reasonable for data that is queried infrequently | Disk-backed random access for a live, high-frequency rank query does not meet the real-time latency budget this workload needs — this is the same fundamental mismatch Decision 02 already ruled out for the live path |
Decision 27
Availability: Serve Stale Rank vs Fail Reads vs Pause Updates
When the projection layer is degraded — a node is down, replication is lagging badly, a rebuild is in progress — what should reads and writes do in the meantime?
| Posture | Favors | Costs |
|---|---|---|
| Serve stale rank (continue answering rank/leaderboard queries from the last known-good state, clearly marked as stale if possible) | Keeps the player-facing leaderboard screen functional during a degraded window, which for a non-critical, informational feature is usually the better user experience than an error screen | A player sees a rank that may no longer reflect recent activity, which has to be an explicitly accepted, bounded-staleness trade rather than an unnoticed correctness gap — this is an extension of Decision 10's staleness budget into a degraded-mode, not a new invariant |
| Fail reads (return an explicit error/unavailable state rather than risk showing wrong data) | Never shows a player data that might be meaningfully wrong | Turns a projection-layer degradation into a visible outage for a feature whose typical failure mode (a few extra seconds of staleness) is usually much less severe than the cost of hiding it entirely behind an error |
| Pause updates (keep serving reads from current state, but stop accepting new score-affecting writes until the degraded component recovers) | Protects write-side correctness directly by refusing to accept events the system isn't confident it can process and order correctly | This is the kind of fail-closed posture that makes sense for an unrecoverable correctness risk (per the stock-exchange module's equivalent decision) — but for a leaderboard specifically, most read-path degradations don't actually threaten write correctness, so pausing writes over a read-path issue is often an overcorrection |
Decision 28
Pagination: Offset/Rank Range vs Versioned Cursor/Snapshot
A player scrolling through a leaderboard, or requesting a rank window around themselves (Test F: Chloe ±1 → Ben #2, Chloe #3, Dev #4), is paging through a structure that keeps changing underneath them. How pagination is implemented decides whether that scrolling stays consistent.
| Strategy | Favors | Costs |
|---|---|---|
| Offset/rank-range pagination (request 'ranks 101-150', 'ranks 151-200', etc., recomputed fresh against current state each page) | Simple to implement directly against a sorted set's native range operations (maps cleanly onto ZREVRANGE by rank window); matches Test F's rank-window query shape exactly | If the board changes between page requests (a new score lands, shifting ranks), consecutive pages can skip or duplicate a row relative to what the player would have seen in one consistent view — a minor but real glitch during active browsing of a fast-moving board |
| Versioned cursor/snapshot pagination (the first page request captures a version marker; subsequent pages are requested against that same version, either via a consistent snapshot read or by having the client carry the marker and the server serving from an equivalent-version replica) | Guarantees a stable, self-consistent view across an entire paging session, even if the live board changes underneath it — directly reuses the same version-marker idea already established for feed recovery (Decision 15) and top-N cache invalidation (Decision 13) | Needs the underlying store to actually support reading 'as of' a version, or needs a short-lived server-side snapshot per paging session, which is more machinery than stateless offset pagination |
Summary
Architecture Decision Record: The Load-Bearing Choices
Most of the 28 decisions above are genuinely workload-dependent — a different scale, a different product shape, or a different team's constraints could reasonably flip them. A smaller set is load-bearing: get these wrong and the rest of the design doesn't hold together, regardless of scale. This ADR table covers that smaller set.
| Decision | Choice | Why | Reconsider When |
|---|---|---|---|
| Source of truth | Durable event log, with Redis/SQL as derived, rebuildable projections | Every other guarantee — idempotency, auditability, correction, recovery — depends on there being one durable, replayable history that projections are derived from, not an independent source of their own | Only for a small, single-board, low-stakes system with no disputes, no season lifecycle, and no recovery requirement |
| Ranking data structure | Redis sorted sets per board, as a bounded baseline rather than an unconditionally correct tool | Matches the real-time rank/range access pattern directly at O(log N); its real limits (ties, memory, single-key throughput, cross-shard rank) are known and designed around rather than ignored | A specific board's tie-break or exact-rank requirement genuinely exceeds what score encoding and single-key sharding can deliver |
| Delivery guarantee | At-least-once delivery with idempotent, event-ID-keyed application | Exactly-once delivery is not an available network guarantee; the only real choice the system controls is whether duplicates are detected and absorbed or left to corrupt state | Never, for any system accepting untrusted, retryable client submissions — this is closer to a law than a preference |
| Partition strategy | Partition by board/scope as the default; shard a single board only when that specific board is demonstrably hot | Matches the natural unit of ranking (one rank query, one key) instead of fighting it; sharding is adopted deliberately per board, not inherited as a side effect of generic infrastructure scaling | A specific board's sustained write volume exceeds single-key throughput even after read-side mitigations (Decision 18) are exhausted |
| Exact vs approximate global rank | Exact rank on a single authoritative key as the default; a distributed rank-index merge layer or an explicit, communicated approximation only for boards that genuinely outgrow one key | Most boards at this scale fit on one key with room to spare; exactness is free there, so there is no reason to trade it away pre-emptively | A specific board's scale forces sharding (Decision 16), and the product team explicitly accepts either merge-layer complexity or a stated approximate-rank experience |
| Moderation/correction model | Every correction is itself a durable, audited event through the same log — never an in-place overwrite | An overwrite that bypasses the log is invisible to any future rebuild (Decision 24) and silently undoes itself the next time the projection is rebuilt from history | Never — this follows directly from the event-log source-of-truth choice and cannot be relaxed without reopening that choice too |
Key takeaways
- The event log as source of truth, with Redis/SQL as derived projections, is the one decision genuinely upstream of almost everything else on this page — get it right first.
- Redis sorted sets are a strong bounded baseline, not an unconditionally correct tool at every scale — their real limits (lex-order ties, single-key write throughput, broken exact rank once sharded) are design inputs, not edge cases to wave away.
- At-least-once delivery with idempotency, audited corrections instead of overwrites, and versioned immutable season identity are close to universal requirements; almost everything else here is explicitly workload-dependent and should move if the workload does.
- For the architecture these decisions assume, see Architecture Deep Dive; for what happens when these baselines are tested by real incidents, see Failure & Scale Scenarios; for how to present these trade-offs concisely under interview time pressure, see Interview Perspective.