Real-Time Gaming Leaderboard System Design: Complete Architecture

A leaderboard answers one deceptively simple question — "where do I stand?" — for a specific game, a specific mode, a specific season, and a specific scope. Getting that question right, quickly, fairly, and without being gameable, is the entire design problem.

Competitive games give players a reason to keep playing beyond finishing content: a visible ranking against other people. FlowRank is the fictional leaderboard service in this module, and Arena Clash is the fictional game it serves — a competitive game where players complete runs and submit scores. Every example in this walkthrough uses the same scope: game arena-clash, mode solo, season S7, scope global. A leaderboard's identity is that entire tuple, not just the game name — arena-clash + solo + S7 + global is a completely different leaderboard from arena-clash + solo + S7 + region:eu-west, and both are different again from next season's S8 board. Two different scopes are two different leaderboards, full stop, even though they share a game and a mode.

Score submission is not the same thing as ranking

These are two different responsibilities, and conflating them is the most common design mistake in this space. Score submission is the game server telling the leaderboard system "this player achieved this score, verified, at this time." Ranking is a derived, continuously-maintained answer to "given everyone's current scores, who is #1, who is #2, and where does this specific player sit." Submission is a write about one player; ranking is a read (or a maintained projection) over many players. A design that tries to compute rank synchronously, from scratch, on every single submission does not scale — and a design that lets ranking drift arbitrarily far behind submission breaks the product's promise of a live leaderboard. The tension between these two facts — writes must be cheap and durable, rank must be fresh and correct — drives almost everything in this module.

Who is allowed to affect a score

A leaderboard is only meaningful if the numbers on it are trustworthy. That means a player's device or client application is never the party that gets to say "I scored 9,800" and have that accepted at face value — a modified client, a replayed network request, or a tampered save file could claim anything. Instead, an authoritative game server (or an equivalent trusted verification step, such as a server-side replay validator) is the only actor whose reported score can affect a player's ranking. The client submits a *claim*; the authoritative server (or a verification pipeline sitting between the client and the leaderboard) is what turns a claim into a verified score event that FlowRank will actually apply. This single rule — never trust a client-supplied score directly — is revisited constantly throughout this module, because almost every correction and anti-cheat scenario traces back to a point where that rule was nearly violated.

Season lifecycle, introduced early

Arena Clash runs seasons — bounded competitive periods (S7 is the season used throughout this module) that open, run for some duration, and then close with a finalized, immutable result used for season-end rewards. A leaderboard's season is part of its identity: S7's global solo board and S8's global solo board track completely independent sets of scores, even though they are "the same" leaderboard from a player's point of view. Chapter 15 covers the full lifecycle state machine; for now, the important boundary is that a finalized season's results do not get silently rewritten by gameplay that happens after the season closes — a late-arriving S7 event cannot leak into S8, and it cannot mutate a published S7 result without an explicit, audited correction process.

What this design is explicitly not

FlowRank is not a matchmaking system — it does not decide who plays against whom; it only records and ranks the outcomes that matchmaking (or solo play) produces. It is not a real-time game-state synchronization system — it has nothing to do with keeping two players' positions and actions in sync during a live match; that is an entirely different latency regime (tens of milliseconds, in-memory, UDP-flavored) from a leaderboard's rank-update latency (sub-second to a few seconds is a perfectly reasonable target). It is not an anti-cheat engine in the sense of detecting aimbots or memory tampering — FlowRank consumes the *output* of verification/anti-cheat (a score has been verified, or has been invalidated), it does not perform the verification itself. Keeping these boundaries explicit is what keeps this design coherent: a leaderboard is a ranking system over verified outcomes, not a game engine, not a network layer, and not a cheat detector.

Before any architecture makes sense, a precise vocabulary is needed — what "rank" actually means, what shapes of query a leaderboard must answer, and what a tie resolves to. We define each one against the canonical Arena Clash S7 global solo board and reuse it for the rest of this module.

The canonical S7 global solo board

Query shapes a leaderboard must support

Scoring modes: highest-score, cumulative, fastest-time

Not every leaderboard ranks the same way, and these are genuinely separate modes with separate comparator logic — not variations on one theme. Highest-score mode (the mode used for the canonical S7 global solo board above) keeps each player's single best verified score and ranks descending by that value; a new submission only matters if it beats the player's current best. Cumulative mode (Test J, Chapter 09) sums verified score events over time instead of keeping a max — every valid event adds to a running total, so idempotent event handling is what prevents a retried event from being double-counted, since there is no "is this lower than my existing best" guard to fall back on. Fastest-time mode (Test K, Chapter 09) ranks ascending — a lower completion time is better — with its own explicit tie-break policy; this must not reuse the descending-score comparator by accident, since blindly sorting times descending would rank the slowest run first.

Rank semantics: deterministic unique ordering

Arena Clash uses deterministic unique ordering: every player gets a distinct rank, 1, 2, 3, 4, 5, with no gaps and no shared ranks, even when scores tie. This is different from two other schemes seen in other products — competition ranking (1, 1, 3 — tied players share a rank, and the next rank skips ahead by the group size) and dense ranking (1, 1, 2 — tied players share a rank, and the next rank does not skip). FlowRank deliberately does not use either of those: ties are broken by a secondary and tertiary key so that an unambiguous 1-2-3-4-5 ordering always exists. The tie-break policy is: highest verified score wins; among equal scores, the player who achieved that score earliest (by verified achievement timestamp) ranks higher; if scores and timestamps are both equal, the lower player ID wins, ascending. That third tie-breaker is there specifically so that rank is a pure, total function of the stored data — never dependent on insertion order, processing order, or any other incidental factor.

Freshness is a stated budget, not a promise of zero latency

"Real-time leaderboard" should never be read as "instant, perfectly consistent leaderboard." Throughout this module we distinguish three separate moments, which are covered in full in Chapter 04: the moment a score is durably acknowledged (the write is safe, even if rank hasn't moved yet), the moment the materialized ranking reflects it (the sorted-set or index a read query actually hits has caught up, possibly with a small lag), and the moment a subscribed client is notified (a push message actually reaches a device, which can itself be delayed or dropped and require a resync). A leaderboard that is honest about this three-way distinction is more trustworthy — and easier to build correctly — than one that hand-waves "real-time" as a single, atomic guarantee.

Functional requirements describe what FlowRank must do for Arena Clash. We separate the essential MVP from extensions so the core design stays focused on what actually makes a leaderboard a leaderboard.

Score submission and aggregate updates

The system must accept a verified score submission from an authoritative game server, identified by a unique event ID, and apply it to the correct player's aggregate under the correct leaderboard scope and scoring mode. For highest-score mode, a submission updates the player's current best only if it actually improves on the existing best (or if there is no existing best yet). For cumulative mode, a submission adds its value to a running total. For fastest-time mode, a submission updates the player's best time only if it is faster than the existing best (or if there is none). Every submission must be processed at most once, regardless of how many times the same event ID is retried by an at-least-once delivery mechanism upstream.

Rank queries

The system must support: top-N for a given scope (e.g., top 10 or top 100 of arena-clash/solo/S7/global); a player's own rank and score within a scope; rank-around-player — a window of players immediately above and below a given player, inclusive, for a given radius; and friends-only ranking — re-ranking a player's friend list (a small, bounded set) independently of the global board, never by filtering a precomputed global top-N (Test E and Chapter 12 cover exactly why that shortcut silently produces wrong, sometimes empty, results).

Scope discovery and live updates

A client must be able to discover which scopes exist for a game/mode (which seasons are active, which regions are offered) rather than hard-coding scope identifiers. Clients viewing a leaderboard should be able to subscribe to live updates for a scope and receive a bounded-latency notification when the board changes meaningfully, without requiring every client to poll aggressively or receive a message for every single underlying score event (Chapter 13).

Moderation, corrections, and season closure

The system must support invalidating a previously accepted score (for example, after an anti-cheat review determines it was fraudulent), which must recompute the affected player's current aggregate from remaining valid history rather than leaving a stale, now-untrusted value in place (Test G, Chapter 15). It must support a season closure workflow that freezes a season's board, finalizes an immutable ranked result snapshot (used for prize/reward qualification), and opens a new, independent season board (Test H). Historical results for finalized seasons must remain queryable after the season closes. All of this is governed by access control: only authorized game-server/service identities can submit verified scores or perform moderation actions; players can only query, never directly write to, their own or others' ranked state.

For a leaderboard, the interesting non-functional requirements are less about raw uptime and more about precisely what "fresh" and "correct" mean, and being willing to say so explicitly rather than promising an impossible instant-consistency story.

Three separate moments of "real-time"

Introduced in Chapter 02, this distinction deserves its own treatment because it is the backbone of every freshness claim in this module. Write acknowledgment: the moment the system has durably recorded that a verified score event was received and will be processed — the game server can trust this as "done," even though the leaderboard may not reflect it yet. Materialized-rank visibility: the moment the data structure that rank queries actually read from (an in-memory sorted structure, a cache, a precomputed index) has incorporated the event — this can lag the write acknowledgment by a small, bounded amount under normal load. Client notification: the moment a subscribed client's screen actually updates — this depends on push delivery, which can itself be delayed, coalesced, or missed and require a resync (Test I). A mature design states explicit targets for each of these three independently, rather than collapsing them into a single, vague "real-time" claim.

Every number in this chapter is a stated, illustrative design assumption for this walkthrough — not a measured benchmark of any real game or leaderboard product. The goal is to demonstrate the estimation process and the architectural conclusions it motivates.

Assumed workload

Assume Arena Clash has 10 million monthly active users (MAU) and 1 million daily active users (DAU). Assume each active player generates roughly 20 score events per day (runs completed, each producing a submission attempt) and issues roughly 100 leaderboard reads per day (checking their rank, viewing top-N, checking friends). Assume 5,000 active leaderboard scopes at any given time (combinations of mode, season, and region/global), and an assumed 100,000 simultaneous live-leaderboard subscribers during peak hours.

Score event throughput

1,000,000 DAU × 20 events/day = 20 million score events/day. 20,000,000 ÷ 86,400 seconds ≈ 231 events/sec average. Applying an explicitly-assumed 20× peak multiplier (to account for evening/weekend play concentration and event-driven spikes such as a new season launch or a double-XP weekend) gives an illustrative peak of roughly 4,630 writes/sec exchange-wide.

Read throughput

1,000,000 DAU × 100 reads/day = 100 million reads/day. 100,000,000 ÷ 86,400 ≈ 1,157 reads/sec average. With the same illustrative 20× peak multiplier, that is roughly 23,150 reads/sec at peak. It matters what "reads" includes here: this estimate counts explicit client requests (opening a leaderboard screen, refreshing a rank) — it does not include the bandwidth consumed by live subscription pushes to the 100,000 simultaneous subscribers, which is a separate fan-out cost covered in Chapter 13 and is driven by how often the board changes and how updates are coalesced, not by a per-client polling rate.

Why exchange-wide averages hide the real constraint

231 writes/sec and 1,157 reads/sec average, spread evenly across 5,000 scopes, would be a trivial ~0.05 writes/sec and ~0.23 reads/sec per scope — almost nothing. But traffic across scopes is never remotely even: a small number of scopes (the current season's global board, a couple of popular regional boards) absorb a hugely disproportionate share of both reads and writes, while most of the 5,000 scopes see a trickle. Because a hot scope can be concentrated behind a small number of keys in the materialized ranking layer (Chapter 09), the hot scope's peak, not the system-wide average, is what the materialized ranking layer for that scope must sustain — this is the same lesson capacity planning for any sharded, skewed system has to confront, and it directly motivates the partitioning and hot-key discussion in Chapter 14.

Event storage and retention

Assume each durable score-event record averages 250 bytes (event ID, player ID, scope fields, score/time value, achievement timestamp, submitting server identity, schema version). At 20 million events/day, that is 20,000,000 × 250 bytes ≈ 5 GB/day of raw event volume, or roughly 150 GB/month before any compaction or tiering. This is a modest, entirely manageable volume for a durable event log with a sensible retention policy (for example, keeping full per-event history for the active season and recent seasons, and compacting or archiving older seasons) — it is not, by itself, a reason to avoid durable event logging.

Network egress for live updates

Assume a push update message (a rank-change or board-delta notification) averages 150 bytes, and assume, illustratively, that an actively-watched hot board emits on the order of 50 update messages/sec during peak contention. Fanning that out to a subscriber base concentrated on a few hot scopes — say 20,000 of the 100,000 simultaneous subscribers watching the single hottest scope — gives 20,000 × 50 × 150 bytes/sec ≈ 150 MB/sec of egress if every subscriber required an independent, uncoalesced stream of every single update. This is precisely why Chapter 13 treats update coalescing and snapshot/delta design as a first-class concern for live leaderboard delivery, not an afterthought — broadcasting every individual score event to every subscriber does not scale, but broadcasting a coalesced, versioned board delta a few times a second does.

The data model has to cleanly separate three things: the immutable record of what actually happened (verified score events), the derived per-player aggregate that ranking is computed from, and the materialized ranking structure itself. Confusing these three is a common design mistake.

Sample verified score event — Test A

{
  "event_id": "evt-101",
  "player_id": "p-dev",
  "leaderboard_scope": "arena-clash:solo:S7:global",
  "value": 9700,
  "achieved_at": "2026-10-13T10:05:00Z",
  "submitted_by": "game-server-07",
  "received_at": "2026-10-13T10:05:00.412Z",
  "status": "accepted"
}

Sample player-score aggregate after Test A

{
  "player_id": "p-dev",
  "leaderboard_scope": "arena-clash:solo:S7:global",
  "current_value": 9700,
  "achieved_at": "2026-10-13T10:05:00Z",
  "last_event_id": "evt-101",
  "aggregate_version": 2
}

Uniqueness constraints and why they matter

event_id must be unique per leaderboard scope at minimum (globally unique is simpler and recommended) — this is the mechanism behind Test C: reprocessing evt-101 a second time is detected via this uniqueness constraint (or an explicit dedup check against last_event_id/a processed-events set) before it can re-apply its effect. (player_id, leaderboard_scope) is the natural key for the aggregate table — exactly one current aggregate row per player per scope, never more than one, which is also what guarantees each ranked player appears at most once per leaderboard snapshot.

FlowRank exposes two very different kinds of surface: a trusted, server-to-server ingestion API for verified score submission, and a player-facing, read-mostly query API. Blurring these two into one general-purpose API is a common source of security and consistency bugs.

Trusted score submission (server-to-server only)

This endpoint is never reachable directly by a player's client — only by an authenticated, authorized game-server or verification-service identity, over a trusted channel. A client-facing API, by design, has no path that lets a player directly write an arbitrary score.

POST /internal/v1/score-events
Authorization: Bearer {trusted-service-token}
Idempotency-Key: evt-101

{
  "event_id": "evt-101",
  "player_id": "p-dev",
  "leaderboard_scope": "arena-clash:solo:S7:global",
  "value": 9700,
  "achieved_at": "2026-10-13T10:05:00Z"
}

Response 202 Accepted:
{
  "event_id": "evt-101",
  "status": "ACCEPTED",
  "aggregate_version": 2
}

Top-N query

GET /v1/leaderboards/arena-clash/solo/S7/global/top?limit=10&cursor=
Authorization: Bearer {player-session-token}

Response 200 OK:
{
  "leaderboard_scope": "arena-clash:solo:S7:global",
  "board_version": 46,
  "entries": [
    { "rank": 1, "player_id": "p-ava",  "value": 9800 },
    { "rank": 2, "player_id": "p-dev",  "value": 9700 },
    { "rank": 3, "player_id": "p-ben",  "value": 9500 },
    { "rank": 4, "player_id": "p-chloe","value": 9500 },
    { "rank": 5, "player_id": "p-eli",  "value": 8700 }
  ],
  "next_cursor": null
}

Own rank and rank-around-player

GET /v1/leaderboards/arena-clash/solo/S7/global/players/p-chloe/rank
Authorization: Bearer {player-session-token}

Response 200 OK:
{
  "player_id": "p-chloe",
  "rank": 3,
  "value": 9500,
  "board_version": 40
}

GET /v1/leaderboards/arena-clash/solo/S7/global/players/p-chloe/nearby?radius=1

Response 200 OK:
{
  "board_version": 40,
  "entries": [
    { "rank": 2, "player_id": "p-ben",   "value": 9500 },
    { "rank": 3, "player_id": "p-chloe", "value": 9500 },
    { "rank": 4, "player_id": "p-dev",   "value": 9100 }
  ]
}

Friends leaderboard

GET /v1/leaderboards/arena-clash/solo/S7/global/friends-of/p-ava
Authorization: Bearer {player-session-token}

Response 200 OK:
{
  "board_version": 12,
  "entries": [
    { "rank": 1, "player_id": "p-ben", "value": 9500 },
    { "rank": 2, "player_id": "p-dev", "value": 9100 },
    { "rank": 3, "player_id": "p-eli", "value": 8700 }
  ]
}

Season results and live subscription

GET /v1/leaderboards/arena-clash/solo/S7/results
Authorization: Bearer {player-session-token}

Response 200 OK (only valid once the season is finalized):
{
  "leaderboard_scope": "arena-clash:solo:S7:global",
  "season_state": "FINALIZED",
  "finalized_at": "2026-11-01T00:00:00Z",
  "snapshot_version": 9981,
  "entries": [ { "rank": 1, "player_id": "p-ava", "value": 9800 } ]
}

GET /v1/leaderboards/arena-clash/solo/S7/global/subscribe?since_version=42
Authorization: Bearer {player-session-token}
Upgrade: websocket

# Server replies with contiguous deltas 43..46, or a fresh snapshot
# if that range's retention window has already expired (Test I).

Moderation / correction (trusted, internal)

POST /internal/v1/moderation-actions
Authorization: Bearer {trusted-moderation-token}
Idempotency-Key: mod-5521

{
  "moderation_id": "mod-5521",
  "action_type": "INVALIDATE",
  "target_player_id": "p-ava",
  "leaderboard_scope": "arena-clash:solo:S7:global",
  "target_event_id": "evt-004",
  "reason": "anti-cheat review: score rejected"
}

Response 200 OK:
{
  "moderation_id": "mod-5521",
  "status": "APPLIED",
  "resulting_aggregate_version": 3,
  "player_new_rank": null
}

Error semantics and pagination

Rank and top-N queries use a board_version field on every response so a client can detect whether two separate calls were served from the same underlying snapshot — rank-around-player and pagination are only stable within one version (concurrent writes between two calls can legitimately produce different ranks, and that is expected, not a bug). Top-N pagination uses an opaque cursor, never a raw numeric offset, since a raw offset is not stable across concurrent rank changes. A submission for an unknown leaderboard_scope, a replay of an event_id already marked invalidated, or a request from an identity lacking the right entitlement each return a distinct, explicit error code rather than a generic failure — this matters operationally, since "duplicate, already applied" and "rejected, never applied" need to be distinguishable by the caller.

The smallest correct version of FlowRank is a single relational table and an indexed ORDER BY query. It is a genuinely reasonable starting point, and understanding exactly where it struggles is more valuable than skipping straight to a distributed design.

A player_scores table holds one row per (player_id, leaderboard_scope): player_id, leaderboard_scope, current_value, achieved_at, last_event_id. A unique index on (player_id, leaderboard_scope) enforces exactly one aggregate row per player per scope. A composite index on (leaderboard_scope, current_value DESC, achieved_at ASC, player_id ASC) supports both top-N and rank lookups directly, in the exact tie-break order defined in Chapter 02.

Top-N and rank lookup as SQL

-- Top-N, using the composite index directly
SELECT player_id, current_value, achieved_at
FROM player_scores
WHERE leaderboard_scope = 'arena-clash:solo:S7:global'
ORDER BY current_value DESC, achieved_at ASC, player_id ASC
LIMIT 10;

-- A single player's rank: count how many rows strictly outrank them
SELECT 1 + COUNT(*) AS rank
FROM player_scores
WHERE leaderboard_scope = 'arena-clash:solo:S7:global'
  AND (current_value, achieved_at, player_id) > (
    SELECT current_value, achieved_at, player_id
    FROM player_scores
    WHERE leaderboard_scope = 'arena-clash:solo:S7:global' AND player_id = 'p-chloe'
  );

Write path: upsert, not insert

For highest-score mode, applying a verified event is an upsert guarded by a comparison, not a blind insert: UPDATE player_scores SET current_value = :value, achieved_at = :achieved_at, last_event_id = :event_id WHERE player_id = :player_id AND leaderboard_scope = :scope AND :value > current_value (with an INSERT-on-no-existing-row path for a player's first score in that scope). This single guard clause — :value > current_value — is what makes Test D (Dev's delayed, lower 9,200 arriving after his 9,700 is already recorded) a correct no-op: the UPDATE simply matches zero rows, and nothing changes.

Why this works at modest scale

For a comparatively modest write rate and a modest number of concurrently-active scopes, a single well-indexed relational table handles both the write-with-guard pattern and the ORDER BY-based read pattern correctly and reasonably efficiently. The rank-lookup query above is O(log n) against the index for the COUNT, which is perfectly fine at moderate row counts per scope.

Where it struggles — and why this is a caching problem, not "SQL is bad"

The real pressure point is not write throughput — a guarded upsert on an indexed row is cheap. It is read volume against a value that changes relatively rarely but is queried extremely often: every top-N request and every rank request re-runs an ORDER BY or a COUNT against the same hot rows, even though the underlying ranking for a popular scope only actually changes when someone's aggregate updates. At the estimated peak of tens of thousands of reads/sec concentrated on a handful of hot scopes (Chapter 05), re-executing an indexed query from scratch for every single read becomes the bottleneck, and it gets worse under concurrent writes to that same scope, since the database has to maintain index consistency while serving a flood of concurrent readers. This is precisely the kind of workload — infrequent writes relative to read volume, over a data shape that supports being held entirely in a fast, purpose-built structure — that motivates introducing a materialized ranking layer on top of (not instead of) the relational source of truth, which is exactly what Chapter 09 does.

A Redis sorted set (ZSET) is an excellent baseline for a bounded leaderboard: it keeps members ordered by score automatically and answers rank and range queries in logarithmic time. It is not, however, a universal answer to every leaderboard scale problem — and treating it as one causes real, specific bugs.

The core commands, with correct semantics

ZADD arena-clash:solo:S7:global 9700 p-dev sets (or updates) Dev's score to 9,700 in that scope's sorted set. ZREVRANK arena-clash:solo:S7:global p-dev returns Dev's zero-based rank in descending-score order (add 1 for a human-facing 1-based rank). ZREVRANGE arena-clash:solo:S7:global 0 9 WITHSCORES returns the top 10 members with their scores, in descending order. ZSCORE arena-clash:solo:S7:global p-dev returns Dev's current score directly. ZREVRANGE ... WITHSCORES with an offset around a computed rank implements rank-around-player efficiently.

The tie-break gap — the sharpest limitation

A native Redis sorted set ranks strictly by its numeric score field. When two members have equal scores, Redis's default behavior falls back to lexicographic ordering of the member name — which has nothing to do with "earlier achievement timestamp," Arena Clash's actual tie-break rule. Relying on this default would make Chloe (p-chloe, 9,500) and Ben (p-ben, 9,500) order by string comparison of p-ben vs p-chloe (which happens to match the desired order here only by coincidence of naming, not by any real correctness guarantee) rather than by who actually achieved 9,500 first. The fix is to encode the tie-break into the score itself: instead of storing the raw value, store a composite numeric score that combines the primary value with an inverted, bounded encoding of the achievement timestamp, so that a single numeric ZSET score fully and correctly captures the two-key sort (value, then earlier-timestamp-wins) — the player-ID tertiary tie-break can then be handled at the read layer for the vanishingly rare case of an exact timestamp collision, or encoded similarly if sub-millisecond precision is a concern. This composite-score encoding is explained in depth in the Architecture Deep Dive; the point here is simply that Redis does not give you the correct tie-break for free, and assuming it does is a real bug, not a theoretical one.

Memory cost, atomicity, and read freshness

Every member tracked in a ZSET consumes memory proportional to the member count, for as long as it is tracked — which is exactly why Chapter 15 tiers active-season boards in Redis but archives finalized-season results elsewhere rather than keeping every season's full membership resident in memory forever. ZADD and ZREVRANK/ZREVRANGE are atomic, single-command operations against one key, which is what makes a single sorted set internally consistent under concurrent updates. Read freshness from a ZSET is exactly as fresh as the most recent successful ZADD against it — there is no inherent additional lag beyond normal command-processing time, but that also means the ZSET is only ever as correct as whatever wrote to it, which is why it cooperates with, rather than replaces, the authoritative aggregate described next.

Single-key bottlenecks and hot boards

Redis is effectively single-threaded on the data plane for operations against any one key — a single extremely hot leaderboard key under heavy concurrent write and read load can become a throughput bottleneck specifically for that key, even while the rest of a Redis instance (and other keys on it) have capacity to spare. This is the hot-board problem introduced in Chapter 05's capacity estimate, and it is Redis-specific in flavor: the fix is not "add more Redis nodes" by itself, because...

...splitting one board across shards breaks exact global rank

If a single hot scope's membership is split across multiple Redis shards to spread load (for example, partitioning players by a hash of player_id into several sub-ZSETs), a plain ZREVRANK against any one shard only returns that shard's local rank — it has no visibility into the members on other shards, so it is not the true global rank. Producing an exact global rank from sharded sorted sets requires an explicit aggregation step across shards (merging and re-ranking, or maintaining a coarser secondary structure), which is real additional work, not something ZREVRANK gives you automatically once a board is split. Chapter 14 covers this trade-off, including when an approximate rank is an acceptable alternative to paying for exact cross-shard aggregation on every query.

How Redis cooperates with the authoritative store

Redis is a materialized, disposable, accelerating projection of the authoritative aggregate and event history — never a replacement for them. The relational (or equivalent durable) aggregate store from Chapter 08 remains the system of record for "what is this player's current best, and from which event did it come"; Redis is rebuilt from that source of truth (or from replaying verified events) whenever it needs to be — after a restart, after a bug, after a cache-layer outage. If the ZSET and the authoritative aggregate ever disagree, the authoritative aggregate wins, by construction, and the ZSET is corrected to match it, never the other way around.

Cumulative and fastest-time modes in Redis

ZINCRBY arena-clash:cumulative:S7:global 100 p-ava is the natural fit for Test J's cumulative mode — Redis's increment is atomic, so two concurrent increments do not race against each other inside Redis. But ZINCRBY has no concept of event identity: it will happily apply the same retried +50 twice if called twice, which is exactly why Chapter 10 requires deduplication to happen before the increment is issued, not inside the ZSET — Test J's correct total of 150 (not 200) depends on the duplicate +50 (same event ID as the first +50) being recognized and dropped upstream of Redis. For fastest-time mode (Test K), scores are stored and compared ascending — ZADD plus ZRANGE (not ZREVRANGE) with MIN conflict semantics (keep the lower, better time) — a deliberately distinct code path from the descending highest-score comparator, so that the two modes can never accidentally share logic and invert one another's meaning.

The path from "a player finished a run" to "the leaderboard reflects it" has several stages, each with a specific correctness job. Skipping or reordering any of them is where real leaderboard bugs come from.

The pipeline, stage by stage

Game server: completes verification (replay validation, server-side run simulation, or an equivalent trust mechanism) and emits a verified score event with a server-assigned achieved_at and a unique event_id. Auth: the ingestion endpoint authenticates the calling service identity and checks it is entitled to submit for this game/scope — a request without a valid trusted-service credential is rejected before anything else happens. Validation: structural checks — is the scope known and currently accepting writes (not a finalized season), is the value plausible for the game, is the event well-formed. Durable ingestion: the event is written to a durable, ordered log or equivalent durable store and acknowledged to the game server — this acknowledgment is the write-ack moment from Chapter 04, and it happens before the aggregate or ranking has necessarily been updated. Dedup: before the event is allowed to affect any aggregate, the system checks whether this exact event_id has already been applied; if so, the event is accepted as a no-op duplicate rather than reprocessed (Test C). Authoritative aggregate update: the guarded upsert from Chapter 08 applies — only if the event actually improves (or, for cumulative mode, validly adds to) the player's current aggregate. Materialized ranking update: the Redis (or equivalent) structure is updated to match the new aggregate, and the scope's board_version is incremented.

Best-score vs. cumulative: different idempotency anchors

Highest-score and fastest-time modes get a natural, almost accidental idempotency property from their comparison guard: reapplying the same event twice is harmless because the second application compares the same value against itself and the guard (> current_value or < current_value) fails on the second attempt even without an explicit dedup check — though an explicit event_id dedup check is still required practice, since two *different* events with the same value could otherwise be confused. Cumulative mode has no such natural protection: ZINCRBY-style addition has no idea whether it has seen a given increment before, so correctness depends entirely on the explicit event_id dedup step happening before the increment is issued, never on the arithmetic itself. This is precisely the distinction Test J is designed to expose: +100, +50, and a duplicate +50 (same event_id as the first +50) must total 150, which only happens if the duplicate is caught by ID, not by any property of addition.

Out-of-order events

At-least-once delivery combined with retries and network variance means a lower, older score for a player can arrive and be processed after a higher, newer one is already recorded — Test D. The guarded upsert handles this correctly by construction: the comparison is against the *current authoritative value*, not against arrival order, so a late 9,200 arriving after a recorded 9,700 is evaluated as 9,200 > 9,700, which is false, and the update is a correct no-op. Out-of-order arrival is normal and expected in a distributed ingestion path — the guard, not strict delivery ordering, is what keeps the aggregate correct.

Corrections are a distinct, audited path

A moderation action (Chapter 06) does not go through the normal guarded-upsert path at all — it explicitly marks one or more prior events as invalidated and triggers a recomputation of the affected player's aggregate from the remaining valid event history, rather than attempting to reverse-apply a delta. This is what makes Test G correct in both of its variants: if no earlier valid score exists, recomputation yields "no current aggregate" (unranked); if an earlier valid score does exist, recomputation correctly restores it as the new current aggregate, rather than leaving a stale, now-invalid value in place or just zeroing the player out regardless of their actual remaining history.

At-least-once plus dedup, never exactly-once delivery

No stage of this pipeline claims exactly-once network delivery — retries, timeouts, and at-least-once semantics between the game server and the ingestion endpoint are assumed as the normal case. What the system does guarantee is the *effective* outcome of exactly-once application, achieved through the combination of at-least-once delivery and event_id-based deduplication at the aggregate-update stage. This distinction — "we don't prevent duplicates arriving, we prevent duplicates from having effect" — is the correct way to describe this guarantee, and claiming true exactly-once delivery would be inaccurate.

When does the server acknowledge success?

The game server receives its 202 Accepted acknowledgment once the event is durably ingested (the durable-ingestion stage above), not once the aggregate or the materialized ranking has been updated. This is a deliberate, stated boundary: it keeps the game server's write path fast and decoupled from however long aggregate/ranking propagation takes, at the cost of the aggregate and rank genuinely lagging the acknowledgment by a small, bounded amount — exactly the write-ack vs. materialized-visibility distinction from Chapter 04.

Tracing Test A end to end through every component makes the write-ack / materialized-visibility / client-notification distinction concrete, rather than abstract.

Starting state: the canonical S7 global solo board, Dev at rank #4 with 9,100. Dev completes a run in Arena Clash and achieves a verified score of 9,700 at 10:05:00 UTC.

Step by step

1. Verification: game-server-07, the authoritative server for Dev's match, validates the run server-side and constructs the verified score event: event_id: evt-101, player_id: p-dev, value: 9700, achieved_at: 10:05:00 UTC. 2. Submission: game-server-07 calls the trusted ingestion endpoint with this event and its own service credential. 3. Auth and validation: the ingestion layer confirms game-server-07's credential is valid and entitled for arena-clash:solo:S7:global, confirms the scope is open (S7 is active, not frozen), confirms the event is well-formed. 4. Durable ingestion: evt-101 is written to the durable event log and acknowledged — game-server-07 receives its 202 Accepted here. This is the write-ack moment: Dev's submission is now safe, even though nothing about the ranking has changed yet. 5. Dedup check: evt-101 has not been seen before, so it proceeds. 6. Aggregate update: the guarded upsert evaluates 9700 > 9100 (Dev's current aggregate) — true — and updates Dev's aggregate to current_value: 9700, achieved_at: 10:05:00, last_event_id: evt-101, aggregate_version: 2. 7. Materialized ranking update: the Redis sorted set for arena-clash:solo:S7:global is updated — Dev's composite score changes, and ZREVRANK for Dev now evaluates to the position corresponding to rank #2 (ahead of Ben and Chloe's 9,500, behind Ava's 9,800). This is the materialized-visibility moment — any rank query issued after this point sees Dev at #2. The scope's board_version increments from, say, 45 to 46. 8. Notification: the live-update fan-out layer (Chapter 13) picks up the board-version change, computes a coalesced delta describing Dev's move, and pushes it to subscribed clients currently watching this scope. This is the client-notification moment, and it can lag step 7 by the push-delivery budget from Chapter 04.

What a concurrent reader sees during the lag between step 6 and step 7

If a rank query for Dev happens to land in the brief window after the authoritative aggregate update (step 6) but before the materialized ranking has been updated (step 7), that query is served from the materialized layer and will still report Dev's old rank (#4, 9,100) — the materialized layer has not caught up yet. This is expected, bounded staleness, not a correctness bug, provided the lag stays within the stated budget from Chapter 04 and the system never claims the materialized answer is more current than it actually is. A query for the *authoritative* aggregate directly (bypassing the materialized layer) would already reflect 9,700 at this point, which is exactly why the two are tracked as separate concerns with separate freshness guarantees.

Board-version propagation summarized

Each read shape introduced in Chapter 02 has a different optimal implementation. Treating them all as "just query the sorted set" works for some and silently produces wrong answers for others.

Top-N

ZREVRANGE scope 0 N-1 WITHSCORES directly, and this result is highly cacheable at an HTTP/CDN or application-cache layer for a short, bounded TTL, since it does not depend on who is asking — only on the current board state. Cache key: the scope plus the board version, so a cached response is naturally invalidated the instant the underlying version changes, without needing a separate explicit invalidation signal.

Own rank

ZREVRANK scope player_id (or the authoritative-aggregate path if a slightly larger freshness window is acceptable and a database round-trip is preferable to a Redis dependency for this specific read). Not cacheable across players, since it is inherently player-specific, but cheap enough per-call that it rarely needs to be.

Rank-around-player

Compute the player's rank via ZREVRANK, then issue ZREVRANGE scope (rank-radius) (rank+radius) WITHSCORES — a windowed range read, cheap in a sorted-set structure. Test F's result (Ben #2, Chloe #3, Dev #4 around Chloe with radius 1) depends on these two calls using a consistent snapshot of the same board version — if the board changes between the rank lookup and the range read, the window could be computed against a stale rank. A single versioned read (or wrapping both calls against one board-version check) avoids that race.

Friends-only — the one that must not take the global-top-N shortcut

A friends-only leaderboard must be computed directly from the player's (small, bounded) friend list: fetch each friend's current aggregate/score (a handful of ZSCORE calls, or a multi-get), then sort that small in-memory set using the same tie-break rule as the main board. It must never be implemented as "filter the global top-100 (or top-N) down to just the player's friends" — Test E demonstrates exactly why that shortcut is wrong: from the original board state, none of Ava's friends (Ben, Dev, Eli) need to even be in a global top-100 for a much larger player base, and filtering a precomputed top-N for friends who aren't in it would silently return an empty or incomplete result, even though a perfectly valid friends-ranking (Ben #1, Dev #2, Eli #3 among themselves) exists. The friend subset is always small and bounded by a player's friend-list size, which is exactly why computing it directly, every time, is both correct and cheap — there is no scaling argument for taking the shortcut.

Regional/scope filters

A regional leaderboard is not a filtered view of the global board — it is its own leaderboard scope (arena-clash:solo:S7:region:eu-west) with its own sorted set, populated by events tagged with that region at ingestion time. This keeps regional queries exactly as fast as global ones (a direct ZREVRANGE against that scope's own key) rather than requiring a filter-and-re-rank operation over the full global population on every regional query.

Cursor pagination and cache invalidation

Paginating beyond the first page of a top-N uses an opaque cursor encoding the last returned rank (or score+tie-break tuple) plus the board version it was computed against, rather than a raw numeric offset — a raw offset silently produces skipped or duplicated entries if ranks shift between page requests. Cache invalidation for any of these reads follows directly from the board-version scheme: a cache entry keyed by (scope, board_version, ...) for any of the above never needs to be explicitly evicted on a write — it is naturally superseded the moment a read requests the new version.

Getting a rank change in front of a player quickly is a fan-out and coalescing problem, not simply a matter of picking WebSockets over polling. The wrong instinct — broadcast every individual score event to every subscriber — does not scale and is not what players actually need.

Polling vs. SSE vs. WebSocket

Polling (the client periodically re-fetches top-N or its own rank) is simple, stateless, and entirely adequate for a leaderboard screen that is not the player's main focus at a given moment — it trades some freshness for operational simplicity, and it degrades gracefully under server-side issues since it is just a normal read request. Server-Sent Events (SSE) gives a one-way, server-to-client push stream, a good fit since leaderboard updates are inherently server-initiated and clients rarely need to push anything back over the same channel. WebSocket gives a full bidirectional channel and is reasonable when a client already has a WebSocket open for other purposes (for example, in-match communication) and can multiplex leaderboard updates onto it cheaply. None of these is a universally correct choice — the decision depends on what else the client already maintains and how central the leaderboard view is to the current screen.

Subscription fan-out and board version

A subscriber registers interest in a specific leaderboard_scope and receives: an initial snapshot (current top-N or the player's own neighborhood, plus the current board_version), followed by a stream of deltas, each tagged with the board_version it moves the client to. The client only ever needs to track one number — its last_applied_board_version — to know exactly how current its view is and to detect a gap.

Coalescing: why not broadcast every event

Pushing a message for every individual verified score event would mean a hot scope's subscriber count multiplied by its event rate in outbound messages per second — exactly the ~150 MB/sec worst-case egress estimate from Chapter 05. Instead, the delivery layer coalesces: it batches the board-version changes that occurred within a short window (illustratively, a few hundred milliseconds to a couple of seconds) into a single delta message describing the net effect (which ranks moved, to what), and advances board_version to the latest value covered by that batch. A player who briefly moved from #5 to #4 and back to #5 within one coalescing window may see no visible flicker at all — which is the correct, desired behavior, not a missed update, since the net effect truly is unchanged.

Snapshot vs. delta, and reconnect/resync — Test I

When a client reconnects after a gap (a dropped connection, a backgrounded app resuming), it presents its last_applied_board_version (42 in Test I). If the delta log retains a contiguous record of changes from 43 through the server's current version (46), the server replays just that range — the client applies four small deltas and catches up precisely. If that range has already aged out of retention (a longer disconnect, or a scope with very high churn and a short delta-retention window), the server instead sends a fresh, full snapshot at the current version, and the client discards its old state entirely rather than attempting a partial, now-impossible replay. Deciding between these two paths is driven entirely by the version gap and retention window, never by how long the client was disconnected in wall-clock time taken alone — wall-clock time is a reasonable proxy but the actual decision criterion is whether the specific version range is still retained.

Authoritative update vs. display refresh

It is worth restating plainly: the moment an update is authoritative (materialized-visibility, Chapter 04/11) and the moment a specific client's screen reflects it (client-notification) are different moments, and a gap between them — even a connected client might have a brief coalescing-window delay — is expected and acceptable, not a bug, as long as it stays within the stated delivery budget and the client's board_version tracking makes the gap detectable and resolvable rather than silently inconsistent.

Never apply an out-of-order delta

A client must validate that an incoming delta's base version matches its own last_applied_board_version before applying it — using the version number as the source of truth, never the order in which messages happened to arrive over the network (which is not guaranteed, especially across reconnects or multiple delivery paths). A delta that doesn't chain cleanly from the client's current version is a signal to resync, not to apply anyway and hope for the best.

Most leaderboard scopes are small enough to live comfortably on a single materialized structure. The interesting scaling problem is the handful of scopes that aren't — and the honest trade-offs involved in handling them.

Partitioning by game, season, and scope

The natural, low-risk partitioning axis is the leaderboard scope key itself — arena-clash:solo:S7:global and arena-clash:solo:S7:region:eu-west already live on logically independent sorted sets (potentially on different physical Redis nodes/shards), since they never need to be queried jointly in a way that requires cross-scope atomicity. This gets most scopes most of the way to acceptable scale without touching the hard problem at all — the hard problem is what to do with a single scope that is, by itself, too hot for one key.

The hot-board problem

Per Chapter 05's estimate, a small number of scopes (the live season's global board chief among them) can absorb a disproportionate share of both read and write traffic. Because Redis is effectively single-threaded on the data plane per key (Chapter 09), this traffic cannot be addressed simply by adding more Redis capacity elsewhere — the hot key itself is the constraint.

Why splitting one board across shards breaks exact global rank

The tempting fix — shard the hot scope's membership across several sorted sets, for example by hashing player_id — spreads write and read load, but breaks the simple, single-command ZREVRANK guarantee: each shard only knows its own members, so a shard-local ZREVRANK is a shard-local rank, not the player's true position among the full scope's population. An exact global rank from sharded data requires an explicit cross-shard step: either merging each shard's relevant neighborhood at query time (practical for rank-around-player and for a single player's rank, more expensive for top-N across many shards), or maintaining a smaller, separately-updated aggregation structure that tracks enough information across shards to answer global queries without a full merge every time. Neither is free, and both add real latency or staleness compared to a single unsharded ZSET — this cost is the actual price of handling a board too hot for one key, not an incidental implementation detail.

Tiered top-N aggregation

A pragmatic middle ground for the specific case of top-N (as opposed to every player's exact rank): each shard maintains its own small, cheap top-K locally; a lightweight aggregation step periodically merges the shards' top-K lists into the scope's global top-N. This gives a fast, scalable top-N (which is the most commonly requested view anyway) at the cost of that merge step lagging slightly behind the absolute latest writes — an explicit, bounded staleness trade, consistent with Chapter 04's framing, rather than a hidden one.

Approximate rank as an explicit, named option

For a player far from the top of an extremely large, sharded, hot scope, an exact rank may not be worth its cross-shard cost — an approximate rank (a bucketed or estimated value, clearly labeled as approximate to the client) can be an entirely reasonable, explicitly-chosen trade-off for that specific query shape, while top-N and a player's own precise standing near a meaningful threshold (for example, a prize cutoff) continue to warrant exact computation. The key discipline is that this is a conscious, stated design choice per query type — never a silent degradation that looks exact but isn't.

Multi-region reads vs. authoritative writes

Read replicas of the materialized ranking layer, placed closer to players in different regions, reduce read latency for top-N and rank queries without changing where writes are authoritatively applied — writes still flow through the single authoritative aggregate-update path described in Chapter 10, and replicas simply lag that authoritative state by ordinary replication delay, which is just another instance of the materialized-visibility staleness budget from Chapter 04, now with a network-distance component added to it.

A season is not just a date range — it is a versioned state machine with explicit transitions, and getting the boundaries right is what makes prize qualification, historical results, and anti-cheat corrections all trustworthy.

The state machine

Open: S7 exists as a scope, configured, but not yet accepting gameplay scores. Active: S7 accepts verified score submissions normally — this is the state for every example in this module up to this chapter. Freeze: a short, explicit window just before closure during which new submissions are rejected (or queued for post-closure reconciliation review) so that the final ranking can be computed against a stable, non-moving population — without a freeze window, "the moment the season ended" is ambiguous down to network-timing races. Finalize: the frozen aggregate state is used to produce an immutable leaderboard snapshot (Chapter 06) — this is the artifact used for prize/reward qualification, and once created, it is never edited in place. Publish: the finalized snapshot becomes queryable via the season-results endpoint (Chapter 07). Archive: the active-season materialized structures (the hot Redis sorted set) are torn down or moved to cold storage, since the season no longer needs low-latency live ranking — the durable aggregate and event history remain queryable, just not through the hot path. A new season (S8) then opens independently, with its own empty scope, sharing no aggregate state with S7 whatsoever.

Late events — Test H

A verified score event for S7 that is submitted (due to network delay, an offline client syncing late, or a delayed verification pipeline) after S7 has finalized must not be silently applied to S7's now-immutable result, and it must not be misrouted into S8 either — S8 began with an empty board and has no relationship to S7's gameplay. The correct handling is an explicit late-event queue: such an event is captured, flagged, and routed to a review process that can decide whether it warrants a formal, audited correction to the finalized S7 snapshot (producing a new, explicitly versioned correction) — never an automatic, silent mutation of the published result.

Prize qualification and audit

Because the finalized snapshot is immutable and versioned, prize qualification ("who was in the top 100 when S7 closed") can always be re-derived exactly, even long after the fact, by reading the snapshot rather than re-running live ranking logic against data that may have since changed. Every moderation action that touches a finalized season's standing is itself an audited record (Chapter 06) linked to a specific snapshot version, so a dispute ("why did my final rank change after the season ended") has a traceable, reconstructable answer.

Anti-cheat review and correction/disqualification — Test G, revisited across the season boundary

Test G's invalidation logic (recompute from remaining valid history, not a blind drop) applies identically whether the correction happens mid-season (affecting the live, active board immediately) or post-finalization (affecting a published, immutable result through an explicit, separate correction record that supersedes the original snapshot for qualification purposes while leaving the original snapshot itself intact as a historical artifact of what was published at the time). The two cases share the same correctness rule; what differs is *where* the corrected result is recorded and how visibly it is distinguished from the original publication.

Historical rebuild

Because every accepted and invalidated event is retained in the durable event log, any season's aggregate state and ranking outcome can, in principle, be deterministically reconstructed by replaying its valid event history from scratch — this is the same replay-based disaster-recovery property described in Chapter 16, and it is also what makes a moderation-driven recomputation (rather than an ad hoc manual edit) possible and auditable in the first place.

A leaderboard's failure modes are less dramatic than a payments outage, but no less real — a wrong rank, a stuck feed, or an exploitable submission path all directly damage a competitive product's core promise.

Event duplication and lag

Duplicate events (Test C) are an expected, handled condition, not an incident — provided the dedup rate stays within a normal baseline. A *rising* duplicate rate is itself a useful early signal of an upstream retry storm or a misbehaving game-server deployment, worth monitoring even though each individual duplicate is harmless. Materialized-visibility lag (Chapter 04/11) becoming unusually large — beyond its stated budget — is a genuine degradation worth alerting on, distinct from the normal, expected small lag.

Redis outage and stale reads

If the materialized ranking layer becomes unavailable, the authoritative aggregate store (Chapter 08) remains intact and correct — the system can fail over to serving rank/top-N queries directly against it (at higher latency and without the sorted-set's O(log n) convenience) rather than serving nothing, while clearly flagging responses during this period as degraded. Once the materialized layer recovers, it is rebuilt from the authoritative store — never assumed to still be correct after an unclean recovery — consistent with Chapter 09's "Redis is disposable and rebuildable" framing.

Feed disconnect

Covered in depth in Chapter 13/Test I: a disconnected or reconnecting client resyncs via versioned delta replay or a fresh snapshot, never by assuming its last-known state is still accurate or by applying deltas without checking the base version they chain from.

Cheating and abuse

FlowRank's job is not to detect cheating itself (Chapter 01), but it must provide the primitives anti-cheat processes need: every accepted score is attributable to a specific verifying server and event, every invalidation is audited, and recomputation from remaining valid history is correct (Chapter 10/15). A submission path that cannot be traced back to a specific authoritative verifying identity is itself a security gap worth treating as a finding, not a minor gap.

Season-rollover races

Covered in Chapter 15/Test H: the freeze state exists specifically to remove the ambiguity around "which season does this event belong to" at the rollover boundary. A design without an explicit freeze window is exposed to exactly this race.

Hot keys and disaster recovery

Hot-scope overload (Chapter 14) and full materialized-layer loss both resolve the same way at the limit: rebuild from the durable, replayable event history and the authoritative aggregate store. This is the concrete reason replay-determinism is treated as a correctness requirement throughout this module, not a nice-to-have — it is the actual disaster-recovery mechanism.

Concrete metrics worth tracking

The final architecture earns every component by solving a specific, demonstrated problem from the chapters above — nothing here is speculative, and the boundaries between its four paths are what make the whole system reason about correctly.

The four paths, and who owns what

Evolution, summarized

Correctness boundaries, restated plainly

This design guarantees: at-most-once effective application of each verified event; deterministic, fully tie-broken rank ordering; a finalized season that does not silently change; and a materialized ranking layer that is always rebuildable from durable, authoritative state. It does not guarantee zero-latency rank visibility, instantaneous client notification, or exact global rank at unbounded scale without an explicit cross-shard cost — each of those is instead a stated, bounded, monitored trade-off.

Remaining trade-offs worth naming

This design treats single-authoritative-write-path-per-scope as the default rather than a true multi-writer, globally-distributed write model, since Arena Clash's scopes do not require it; it treats Redis sorted sets as the default materialization choice for active, bounded scopes while explicitly archiving finalized ones rather than keeping every season resident forever; and it treats approximate rank as an explicit, opt-in trade-off rather than a silent default for any specific query. Every numeric figure anywhere in this module — the capacity estimates, the latency budgets — is an explicitly stated design assumption for teaching purposes, never a verified benchmark of any real product.