Real-Time Leaderboard System Design — Failure and Scaling Scenarios

FlowRank's correctness contract starts at ingestion: every score event must be applied at most once, and only a verified higher score (in highest-score mode) or a genuinely new, non-duplicate increment (in cumulative mode) can ever change a player's standing. The four incidents below are all shapes of the same underlying question: what happens when the same event, or an out-of-order one, arrives more than — or later than — its logical due.

Scenario 01 — Duplicate score event after producer retry

Trigger: Arena Clash's game server emits an achievement event (evt-101: Dev's personal best of 9,700 at 10:05:00) to FlowRank's ingestion endpoint, but the acknowledgment is lost on the way back — a timeout, a network blip, a load-balancer failover. The game server's own retry logic, which cannot distinguish 'my event never arrived' from 'the response just didn't arrive,' resends the identical evt-101 payload moments later. Immediate technical effect: ingestion now sees the same event ID twice. Every event carries a producer-assigned, globally unique event ID that the ingestion layer checks against a durable dedup ledger before admitting it to ranking, so the second evt-101 is recognized as already-applied rather than treated as a new candidate score change.

User-visible effect: this is Test A followed by Test C. After evt-101 is first applied, standings become Ava #1 (9,800), Dev #2 (9,700), Ben #3 (9,500), Chloe #4 (9,500), Eli #5 (8,700); the duplicate evt-101 arriving afterward produces zero change to Dev's score, zero change to rank order, and no second notification implying a second personal best. How to detect it: dedup-ledger hit rate per event ID, with an alert on an abnormally high duplicate rate from one producer, since that usually signals an over-aggressive client-side retry loop rather than a one-off network blip.

Containment: no halt — this is resolved inline, per event; unrelated players' events keep flowing without interruption. Recovery sequence: none needed — the dedup check is itself the recovery; for callers that care about the outcome, ingestion returns the same already-applied result the first request produced. How to prevent recurrence: require every event ID to be unique and stable per emitted achievement, keep the dedup lookback window at least as long as the producer's maximum retry window, and treat 'same event ID, different payload' as a hard ingestion error rather than silently accepting whichever payload arrived most recently.

Scenario 02 — Duplicate cumulative increment would inflate totals

Trigger: in a cumulative-scoring mode (total points earned across a session, not a single personal best), a player's client emits an increment event, the acknowledgment is lost, and the producer retries with the same event ID — structurally identical to Scenario 01, but applied to an additive operation instead of a replace. Immediate technical effect: Test J's exact sequence — +100 applied (evt-C1), +50 applied (evt-C2), then a duplicate of evt-C2 (same event ID, same +50 payload) arrives. The dedup ledger recognizes evt-C2 has already been applied and discards the duplicate rather than adding another +50 on top.

User-visible effect: the player's total reads 150, never 200 — cumulative mode is exactly as vulnerable to duplicate-inflation as highest-score mode is to a duplicate personal best, just with additive instead of replace semantics. How to detect it: a per-player cumulative-total reconciliation job that sums every applied, deduplicated event ID and compares the result against the materialized total, flagging any drift immediately.

Containment: no halt; rejected inline, exactly like Scenario 01. Recovery sequence: none — the correct total already stands once dedup fires; if a bug ever let a duplicate through, the fix is a replayable, audited correction (Scenario 08's pattern), never a silent decrement applied in place. How to prevent recurrence: cumulative mode must dedup on event ID exactly like highest-score mode does — a common implementation mistake is assuming 'add' operations are inherently safer to repeat than 'set' operations, when in fact an additive apply without dedup is strictly more dangerous under at-least-once delivery, since every redelivery compounds the error instead of simply re-confirming the same final value.

Scenario 03 — Out-of-order older personal-best event

Trigger: Dev's game client buffers an achievement event locally (a 9,200 result) during a brief connectivity loss, then flushes it to ingestion after connectivity resumes — arriving after evt-101 (Dev's 9,700) has already been applied. Immediate technical effect: this is Test D. Ingestion compares the incoming 9,200 against Dev's current best (9,700) and rejects the incoming event as a no-op, since 9,200 is lower than 9,700 — a value-comparison rejection specific to highest-score mode's monotonic-improvement rule, distinct from Scenario 01's event-ID dedup rejection, since this is a different event ID entirely.

User-visible effect: no change — Dev remains #2 at 9,700, and no notification is sent implying a regression in his standing. How to detect it: a 'rejected — lower than current best' rate tracked per player and per producer; a spike can indicate a client-side buffering bug or a systemic delivery-ordering issue worth investigating even though each individual rejection was handled correctly.

Containment: no halt; a per-event rejection. Recovery sequence: none — this is the system working exactly as designed; if a current best is later found to be erroneous and genuinely needs to be replaced by a lower, previously-rejected value, that can only happen through the audited correction workflow (Scenario 08), never through this automatic comparison path, which is a one-way, can-only-improve ratchet by design. How to prevent recurrence: always compare an incoming candidate against the current committed best read from the durable ranking source, never a possibly-stale cache, so that the score itself — not network arrival order or event timestamp — is the gate that decides whether an update applies.

Scenario 04 — Two simultaneous new high scores for one player

Trigger: Dev's client reports two distinct achievement events from two different matches finishing within milliseconds of each other — say a 9,300 and a 9,650 — both genuine improvements over Dev's prior best at the moment they were earned, submitted via two different game-server instances handling concurrent match-end callbacks. Immediate technical effect: both events race to perform the same read-compare-write operation against Dev's current best. Without a per-player serialization point, both reads could see the same stale 'current best,' both could conclude they're an improvement, and whichever write lands last could even persist the lower of the two scores.

User-visible effect without the fix: Dev's rank could transiently — or, if the race condition is a persistent bug rather than a one-off, permanently — reflect the lower of the two eligible scores instead of the higher one. With the fix in place, the user deterministically ends up at the higher of the two, regardless of which event's request happened to complete first at the network layer. How to detect it: a per-player optimistic-concurrency version-mismatch counter, and a correctness alarm on any applied score for a player that is lower than another event already known to exist for that player within the same short window.

Containment: no halt — resolved per player via a per-player serialization mechanism, such as a versioned compare-and-set against the player's current best record, or routing all of one player's events through a single partition/queue so the two events apply as a strict sequence rather than concurrently. Recovery sequence: whichever event is evaluated second is compared against the already-updated value, not the stale one it originally read — so it correctly becomes a no-op if it's now lower, or still applies if it's still higher than both, using exactly the same comparison logic as Scenario 03, just operating within a tighter time window. How to prevent recurrence: serialize all mutations to one player's ranking record — never a global lock, just a per-player one — using optimistic concurrency (an expected-version compare-and-set) or a single-writer queue keyed by player ID.

FlowRank never trusts a client-supplied score directly — every event has to prove its legitimacy before it can move a player's rank, and nothing it presents, including the global #1 position, is unconditionally final. The next four incidents walk the trust boundary from the moment a game server submits a report through the moment a champion's win is formally revoked.

Scenario 05 — Game server submits malformed or unauthorized event

Trigger: a misconfigured or compromised game-server instance sends an ingestion request missing required fields (no player ID, no match ID), carrying an invalid or expired service credential, or targeting a game+mode+season+scope combination that producer is not authorized to write to. Immediate technical effect: schema validation and authorization checks run before the event ever reaches the dedup ledger or the ranking comparison — a malformed or unauthorized payload fails at this gate and is never evaluated as a candidate score change.

User-visible effect: none for players — the event simply never happened as far as any leaderboard is concerned. The originating producer sees an explicit rejection with a reason code (schema violation, auth failure, or scope mismatch) in its own logs. How to detect it: rejection-reason metrics broken out by producer/service identity; a sustained spike in schema or auth failures from one producer identity is itself worth escalating, independent of whether any single rejected event mattered on its own.

Containment: no halt; a per-event rejection isolated to the offending producer, with no effect on healthy producers. Recovery sequence: none needed beyond the producer fixing its payload or credential and resubmitting, if the underlying event was genuine. How to prevent recurrence: strict schema validation and service-to-service authentication/authorization on every ingestion path, scope-level write authorization so a producer for arena-clash:solo can never write to a different game's board even by accident, and treating ingestion as a hostile boundary regardless of whether the caller is a first-party game server.

Scenario 06 — Cheating client forges an impossible score

Trigger: a tampered game client reports an achievement event with a score outside Arena Clash solo mode's physically achievable range, or a sequence of achievements inconsistent with the mode's rules — an impossible completion time, or progression that skips required stages. Immediate technical effect: a server-side plausibility and anti-cheat check — independent of the client's self-reported value — flags the event rather than accepting it as a verified score. 'Verified' in FlowRank's ranking policy specifically means passed this check, not merely 'received.'

User-visible effect: the forged score never appears on the public leaderboard. Depending on the severity of the flag, the event is either hard-rejected immediately or routed to quarantine (Scenario 07) when the evidence is ambiguous rather than clearly conclusive. How to detect it: plausibility-bound violations per player and producer, anomaly scoring on achievement velocity against the observed population distribution, and correlation with known-cheat-tool fingerprints where available.

Containment: no halt; the offending player's further events can be routed through stricter validation or held for review without affecting any other player's standing. Recovery sequence: none needed for a hard-rejected forged score; if a forged score had already been accepted before detection improved, it follows the moderation-invalidation path in Scenario 08, never a silent deletion. How to prevent recurrence: server-authoritative scoring wherever feasible (the server, not the client, computes the final score from verifiable match events), plausibility bounds tuned against the real population of legitimate top scores, and layered anti-cheat signals rather than a single score-ceiling check.

Scenario 07 — Suspected score is quarantined pending review

Trigger: an event passes basic schema and authorization validation but trips a softer anti-cheat heuristic — not a hard-impossible value, but statistically unusual enough to warrant review, such as a score improvement far larger than the player's own historical variance. Immediate technical effect: the event is written durably (nothing is lost) but held in a quarantine state — applied to neither the public leaderboard nor the player's best — pending a review decision, rather than either accepted optimistically or rejected outright.

User-visible effect: the player does not see the new rank yet. If review later approves it, the player's rank updates at that point rather than being backdated into history as if it had always been visible — keeping every past, already-published leaderboard state consistent with what was actually shown at the time. If rejected, the event is marked invalid and never surfaces. How to detect it: quarantine-queue depth and average time-in-quarantine as explicit operational metrics; an item aging past a review SLA is itself an alert, since an un-reviewed item isn't 'safe,' it's 'undecided.'

Containment: no halt to other traffic; only the specific player's pending change is held. Recovery sequence: a reviewer — human or an automated secondary check — resolves the quarantined event to accepted or rejected; once resolved, the normal ranking-comparison or no-op path applies exactly as if the event had just arrived. How to prevent recurrence: tune the quarantine threshold so the queue stays small enough to review promptly, keep a durable, timestamped audit record of every quarantine decision and its rationale, and give players visibility that a pending review exists where appropriate, rather than silence.

Scenario 08 — Moderation invalidates the current #1 score

Trigger: after review — via Scenario 07's quarantine path, a player report, or a retroactive anti-cheat sweep — Ava's 9,800 score, currently global #1, is found to be invalid (confirmed cheating, a scoring-service bug that overcounted, or forged telemetry) and is ordered invalidated. Ava has no other valid score on record for this leaderboard. Immediate technical effect: this is Test G. The invalidation is applied as a new, audited, versioned correction event — never a silent in-place row edit — that revokes the event that produced Ava's 9,800 and removes its contribution to her best.

User-visible effect: Ava becomes unranked on this board, and the leaderboard re-sorts to Ben #1 (9,500), Chloe #2 (9,500, later timestamp than Ben so she stays below him under the tie-break), Dev #3 (9,100), Eli #4 (8,700). Every affected player's rank shifts up by exactly one position; no player is skipped or duplicated in the new ordering. How to detect it: this is a deliberate, operator-triggered action rather than something auto-detected — the signal worth monitoring is the correction-events-applied rate, ensuring every correction has a linked case/ticket ID in the audit log, never an unattributed edit.

Containment: no system-wide halt; this board continues serving reads throughout, with the correction applied as one more versioned event in the stream — reads briefly before and after the correction simply see two different, each internally self-consistent, leaderboard versions. Recovery sequence: none needed beyond the correction itself; if the invalidation is later found to be wrong on appeal, a second audited correction event reinstates the score — forward-only corrections, never rewriting history to pretend the first correction never happened. How to prevent recurrence: require every rank-altering moderation action to go through the same versioned-event pipeline as a legitimate score submission, just with revoke semantics and a mandatory case reference, never a direct update against a materialized rank table.

FlowRank keeps the durable record of what happened — the event log — deliberately separate from the fast, queryable view of current standings, typically Redis sorted sets. That separation is what makes deterministic recovery possible, but it also means the two can transiently disagree. These eight incidents are about the gap between 'the event is safely recorded' and 'the rank index reflects it.'

Scenario 09 — Score committed to DB but ranking projection fails

Trigger: evt-101 (Dev's 9,700) is durably appended to the event log successfully, but the downstream process that updates the Redis sorted-set projection crashes, throws, or times out before applying it. Immediate technical effect: the source of truth — the durable event log — is correct and complete; the materialized rank index is now missing one update and is stale by exactly one event for Dev.

User-visible effect: a leaderboard read served from the stale projection still shows Dev at his old position (#4, 9,100) for a bounded window, until the projector catches up — a transient lag, not a permanently incorrect state. How to detect it: a projector-lag metric comparing the last-applied event offset against the durable log's latest offset, crossing a staleness threshold; a projector crash also triggers the same process-supervision alerts as a service crash anywhere else.

Containment: reads continue to be served from whatever the projection currently holds — FlowRank does not halt reads during this gap, but does label this as a known, bounded staleness window rather than claiming the projection is always instantaneously correct. Recovery sequence: the projector restarts (or a replacement instance takes over) and resumes consuming the durable log from its last successfully committed offset, applying the missed event through the same idempotent-apply logic as live traffic. How to prevent recurrence: make projection consumption resumable from a durable offset — never 'start from latest, skip whatever was missed' — monitor projector lag continuously, and run the projector under fast-restart supervision so gaps stay short.

Scenario 10 — Rank index updated but acknowledgment/notification lost

Trigger: the projector successfully applies evt-101 to the Redis sorted set — Dev now shows 9,700, rank recalculated — but the downstream notification (a 'new personal best' push, or the producer's ingestion acknowledgment) is dropped on the network before delivery. Immediate technical effect: ranking state is fully correct and durable; only a notification about that state failed to arrive.

User-visible effect: Dev's rank is already correctly updated and visible to anyone else querying the board; Dev's own client, if waiting on that specific push, might not immediately know — but a normal leaderboard fetch would already show the new state. How to detect it: notification delivery failure/retry metrics tracked separately from ranking-correctness metrics, since a gap here never implies any doubt about the underlying rank.

Containment: no halt; a purely delivery-reliability concern layered on top of already-correct state. Recovery sequence: standard notification retry/resend logic, or the client simply re-fetches current state on its next poll or reconnect — nothing in the ranking system itself needs to recover anything, since nothing there was ever wrong. How to prevent recurrence: make notifications derivable and re-fetchable from current durable state rather than a one-shot fire-and-forget, keeping ranking correctness and notification delivery as cleanly separate concerns with separate monitoring.

Scenario 11 — Redis node crashes and leaderboard index disappears

Trigger: the Redis instance (or unpromoted primary node) hosting the arena-clash:solo:S7:global sorted set crashes — an out-of-memory kill, a host failure, or an operator-initiated restart without a replica ready. Immediate technical effect: the in-memory sorted set disappears instantly; if persistence (RDB/AOF) was enabled, some recent state may be recoverable from disk, but anything written after the last persisted point is not recoverable from Redis alone.

User-visible effect: leaderboard reads for this board fail or return explicitly unavailable until the index is restored — a visible, bounded outage, never silently wrong data; a missing index must never be served as an empty leaderboard that could be mistaken for a real, if unlikely, state. How to detect it: Redis health checks and liveness probes fail immediately, and read-path error rate for this specific board spikes to match.

Containment: reads return an explicit 'temporarily unavailable' response (or are served from a caught-up replica if one exists) rather than a silently empty leaderboard. Recovery sequence: promote a replica if one exists and is caught up — the fastest path — otherwise rebuild the sorted set from scratch by replaying the durable event log from the beginning, or from the last good projection snapshot, since the event log, not Redis, is the authoritative history. How to prevent recurrence: run Redis with replication so a replica can be promoted instead of a full rebuild, enable persistence as a second line of defense, and keep the event-log-replay rebuild path tested and fast enough to meet a stated recovery-time assumption.

Scenario 12 — Redis memory limit, eviction or hot-key overload

Trigger: the sorted set backing a very large or very active board grows past the Redis instance's configured memory ceiling, or a single extremely active board becomes a hot key receiving far more traffic than one instance can sustain, since write ordering on one key is effectively single-threaded. Immediate technical effect: under memory pressure, a configured eviction policy could reclaim memory by evicting keys — a real risk of silently losing part of a leaderboard if eviction isn't deliberately scoped away from ranking keys; under hot-key overload, the single key becomes a serialization bottleneck and latency for that one board climbs while other boards are unaffected.

User-visible effect: worst case for memory pressure is an evicted board appearing empty or incomplete — recoverable, but alarming; for a hot key, that one board's updates and reads lag noticeably while every other game/mode/season board stays fast. How to detect it: per-instance memory-utilization alarms well before the configured max-memory ceiling, and per-key command-rate and latency metrics isolating one key as disproportionately hot, separate from aggregate instance load.

Containment: ranking keys are explicitly excluded from any eviction policy — a leaderboard is never cache, so it should never be eligible for LRU-style eviction; a hot key is throttled or admission-controlled at the write path, scoped to that one board, the same control used in Scenario 19's burst handling. Recovery sequence: if a key was evicted, rebuild via Scenario 11's index-rebuild-flow; if hot-key latency degraded without data loss, recovery is simply load subsiding or the key being split across shards (Scenario 17's trade-offs apply). How to prevent recurrence: size Redis memory capacity against worst-case cardinality assumptions, configure max-memory policy to never evict ranking keys, and plan sharding/partitioning decisions ahead of the pressure, not during an incident.

Scenario 13 — Durable event broker becomes unavailable

Trigger: the durable, replayable event broker that ingestion writes to and the projector reads from becomes unreachable — a cluster outage, a network partition, or a full disk on the broker's storage. Immediate technical effect: new score events cannot be durably recorded; accepting an event without a durable record would mean a crash or restart could silently lose it.

User-visible effect: new achievement events from the game cannot be acknowledged as accepted during the outage; existing leaderboard reads continue working fine from the last-known projection state, since reads don't depend on the broker being up. How to detect it: broker health, write-latency, and error-rate alarms, plus the ingestion layer's own 'event not durably recorded' error rate spiking to match.

Containment: ingestion fails closed — new events are rejected rather than accepted into an in-memory-only buffer a crash could lose; reads are unaffected and continue serving the last consistent projection. Recovery sequence: once the broker is reachable and healthy again, ingestion resumes accepting and durably recording new events; any producer-side buffered events submitted afterward are processed through the normal idempotent-dedup path, handling a post-outage retry flood exactly like Scenario 01, just at higher volume. How to prevent recurrence: run the broker with its own replication and durability guarantees, monitor broker health independently of FlowRank's own services, and ensure producers use sane local retry/backoff so recovery doesn't trigger a thundering-herd retry storm.

Scenario 14 — Consumer backlog causes stale visible ranks

Trigger: the ranking projector falls behind the event log's production rate for a sustained period — a slow downstream dependency, insufficient consumer parallelism, or a burst of events (Scenario 19) that outpaces normal projector throughput. Immediate technical effect: the gap between the log's latest offset and the projector's last-applied offset grows instead of staying bounded, and the projection reflects an increasingly old moment.

User-visible effect: players see their leaderboard position lag noticeably behind their actual latest achievement — not wrong, just late, with the lag itself worth surfacing explicitly rather than presenting stale data as current. How to detect it: projector lag, converted to an estimated time lag, tracked continuously, not just checked during incidents.

Containment: no halt to ingestion — events keep being durably recorded regardless of projector speed, since the log and projection are deliberately decoupled; if the backlog threatens to breach a stated staleness budget, it's escalated as a capacity incident, not hidden. Recovery sequence: scale out projector consumer parallelism (partitioned so a given player's events stay in order), or address whatever slow dependency caused the lag; the projector catches up by processing its backlog in order, with no event skipped or lost. How to prevent recurrence: provision projector throughput with headroom above assumed peak event rate, monitor lag continuously against the stated staleness budget, and design the projector to scale horizontally across partitions without breaking per-player event ordering.

Scenario 15 — Poison message repeatedly fails projection

Trigger: one specific event in the durable log has a payload the projector cannot process without throwing — a schema-evolution mismatch, an edge-case value, or a bug triggered only by that exact combination of fields. Immediate technical effect: a naive projector that simply retries the failing event forever gets stuck — it cannot advance past the poison message, so every later event in that partition's order stalls behind it, even though those later events are perfectly processable on their own.

User-visible effect: without a poison-message strategy, the entire board (or that player's ordering partition) appears frozen — no rank updates apply at all, a much larger blast radius than the one bad event deserves. How to detect it: a specific event's retry count exceeding a threshold with no forward progress; projector lag growing unbounded even though the broker is healthy is the distinguishing signature versus Scenario 14's general backlog.

Containment: after a bounded number of retries, the poison message is moved to a dead-letter queue — durably preserved, never discarded — so the projector can advance past it and keep processing every later, valid event, isolating the blast radius to exactly the one unprocessable event. Recovery sequence: engineers inspect the dead-lettered event, fix the gap that made it unprocessable, and replay that specific event back through the pipeline once a fix is verified, applied exactly once via the same idempotent event-ID dedup as any other event. How to prevent recurrence: schema versioning with backward-compatible evolution rules, defensive validation in the projector with clear dead-letter semantics from day one, and alerting on dead-letter-queue depth so a poison message never sits unnoticed for long.

Scenario 16 — Ranking projection applies the same event twice

Trigger: unlike Scenario 01's ingestion-level duplicate, this is a bug inside the projector itself — a consumer-group rebalance causing reprocessing from an earlier offset, or a bug in the projector's own 'mark as applied' bookkeeping that lets it reapply an event it already committed. Immediate technical effect: the projector calls its apply logic twice for the same event ID; if the projector's own internal idempotency check — a separate applied-event-ID set, checked before every apply, not just ingestion's dedup ledger — is missing or buggy, the event's effect could be applied twice, doubling a cumulative increment or re-triggering a stale notification.

User-visible effect without the internal check: a cumulative total could silently inflate — the Scenario 02 failure mode, but originating inside the projector instead of at ingestion; with the internal check in place, the second apply call is recognized as already-done and is a no-op. How to detect it: per-event apply-count tracking inside the projector itself, and a reconciliation job comparing the sum of all durable event effects against the materialized projection state to catch any drift that slips through.

Containment: no halt; caught and corrected by the projector's own idempotency layer before it reaches a reader, assuming that layer is implemented — if a bug bypasses it, the affected player's record is flagged for reconciliation. Recovery sequence: if a double-apply slipped through, recompute the affected player's state by replaying their full event history from the durable log — the index-rebuild-flow pattern, scoped to one player rather than the whole board — and overwrite the projection's value with the correct, freshly-derived one. How to prevent recurrence: the projector must maintain its own applied-event-ID bookkeeping independent of ingestion-level dedup, since at-least-once broker delivery means the projector itself will see redelivered events as a matter of course, not as an edge case.

A single Redis sorted set gives an exact global rank almost for free via ZREVRANK — until the board is too large or too hot for one key to hold. These five incidents are about the real limits of that convenience: what breaks when one board is split across shards, when an exact rank becomes too expensive to ask for honestly, and when read or fan-out traffic outgrows what a single structure can serve.

Scenario 17 — Different ranking shards return inconsistent views

Trigger: the arena-clash:solo:S7:global board has grown large enough that it's split across multiple Redis shards, commonly by a hash of player ID, to avoid the single-hot-key ceiling from Scenario 12. A client queries for a global top-N or a specific player's global rank. Immediate technical effect: each shard can report its own shard-local top members and a shard-local rank via ZREVRANK, but no single shard has a global, cross-shard total order — a player's shard-local rank is not their actual global rank, and merging each shard's local top-N is an approximation of the true global top-N, never an exact computation.

User-visible effect: two different queries could disagree slightly on where the line between rank 99 and rank 101 falls, or a player near a shard boundary could see an approximate rank off by a handful of positions from the mathematically exact one — an inherent property of sharding a total-order structure, not a bug to silently paper over. How to detect it: a reconciliation job periodically computing a true exact rank for a sample of players (via a full cross-shard merge, or against the durable event log) and comparing against the approximate, shard-merged answer, tracking the measured deviation rather than expecting a binary pass/fail.

Containment: no halt; the product decision is to clearly label cross-shard results as approximate rather than present a shard-merge result with false precision, and to offer an explicit, more expensive exact-rank path (Scenario 18) for cases that truly need it. Recovery sequence: not applicable as an incident to recover from — this is a standing architectural trade-off; what does get fixed is drifting approximation quality, for example rebalancing shard boundaries if one shard grows disproportionately and skews the merge error. How to prevent recurrence: choose a sharding key that distributes load evenly, keep per-shard top-N merge logic consistent and well-tested, and be explicit in the product surface about which queries are exact and which are approximate.

Scenario 18 — Exact global rank query becomes too expensive

Trigger: a client requests a precise, not-approximate global rank on a board large enough that computing it cheaply via a single sorted set's ZREVRANK is unavailable, because the board is sharded (Scenario 17) or the single-structure approach has grown past a comfortable size. Immediate technical effect: computing a truly exact global rank now requires either a full cross-shard merge-and-count operation, whose cost scales with shard count and board size, or maintaining a separate, more heavyweight exact-rank index with added write cost on every score change to keep it current.

User-visible effect: if served synchronously on every request, exact-rank queries could introduce noticeably higher latency than a simple top-N fetch, especially under load. FlowRank's policy is to offer exact rank as a deliberately, differently provisioned capability — possibly rate-limited or slightly staler — rather than silently degrading the fast path for everyone to support it. How to detect it: latency and resource-cost metrics specifically on the exact-rank code path, tracked separately from standard top-N and around-player queries.

Containment: rate-limit or cache exact-rank requests more aggressively than standard reads, and keep the exact-rank computation isolated in a separate resource pool so a surge in exact-rank requests cannot degrade the fast top-N and around-player paths most traffic actually uses. Recovery sequence: not an outage-style recovery — this is capacity and architecture tuning; if exact-rank demand grows, the fix is investing in a purpose-built exact-rank index, not forcing the existing fast structure to serve a fundamentally different cost profile. How to prevent recurrence: decide up front, as an explicit product and architecture choice, whether exact global rank is worth its real cost at this scale, or whether an honestly-labeled approximate rank with a stated error bound is the better trade-off.

Scenario 19 — Sudden 100× game tournament traffic spike

Trigger: Arena Clash runs an in-game tournament event, causing both match volume and the resulting score-submission rate to spike roughly 100× above baseline across a short window, simultaneously with elevated read traffic from players checking live standings. Immediate technical effect: ingestion, the event broker, the projector, and the read path all see proportionally higher load at once — the write-side counterpart to Scenario 14's backlog risk and Scenario 20's read-side surge, both stressed simultaneously.

User-visible effect: without admission control, rank updates could lag far behind the stated staleness budget, and read latency for standings could degrade broadly across every player checking the tournament board, not just the heaviest submitters. How to detect it: aggregate ingestion rate and projector lag crossing high-water marks together, plus read-latency percentiles scoped to the specific tournament board, since other, unrelated boards should stay unaffected.

Containment: admission throttling/backpressure on ingestion for the specific overloaded board — never silently dropping events, since a throttled producer gets an explicit retryable rejection that preserves the event rather than risking a silent loss — and read-path caching/rate-limiting to absorb the surge without destabilizing the write path. Recovery sequence: once tournament traffic subsides, any backlog allowed to form within the staleness budget drains, and normal admission limits relax; a post-incident reconciliation confirms every accepted event was eventually applied exactly once. How to prevent recurrence: provision capacity headroom above an assumed peak multiplier for known high-traffic events (see Scale Walkthroughs), pre-scale ahead of a scheduled tournament rather than reacting during it, and keep write and read paths independently scalable.

Scenario 20 — Top-100 read traffic surges after a popular match

Trigger: a high-profile match or streamed event drives a large number of spectators and players to simultaneously query the same global top-100 view within a short window — a read-side surge concentrated on one specific, popular query shape rather than spread evenly across all possible queries. Immediate technical effect: if every one of those reads hits the live Redis sorted set directly, read QPS on that one board could spike well past normal levels, competing with the write path's own access to the same structure.

User-visible effect: without caching, read latency for the top-100 view could degrade for everyone checking it during the surge; with a short-TTL cache or precomputed snapshot layer in front of the live structure, reads stay fast and the live structure is shielded entirely. How to detect it: read QPS broken out per query shape — top-N specifically, separate from around-player or friends queries — and cache hit-rate dropping or cache-miss latency climbing as the surge begins.

Containment: serve top-100 reads from a short-TTL cache or a periodically refreshed snapshot, with explicitly bounded staleness stated as part of the staleness budget, rather than hitting the live sorted set on every request; if the cache itself is overwhelmed, scale the cache tier, not the underlying ranking store. Recovery sequence: once the surge subsides, cache refresh intervals and capacity return to steady-state; no ranking data was ever at risk, since this is purely a read-amplification concern layered in front of already-correct state. How to prevent recurrence: default to serving popular, read-heavy query shapes from a cache/snapshot layer sized against assumed popular-event read multipliers, and keep that caching layer easy to scale independently of the live ranking store.

Scenario 21 — Friends-ranking query fans out across too many users

Trigger: a player with an unusually large friends list triggers a friends-subset rank computation that has to gather current scores for every friend before it can rank them against each other. Immediate technical effect: Test E's correctness requirement — Ava's friends-subset rank (Ben #1, Dev #2, Eli #3, using the original canonical state) must reflect only the eligible friends subset, never be derived by filtering the global top-N, since a friend could easily rank far outside any reasonably sized global top-N and still be #1 among friends. That requirement means the query has to explicitly fetch each friend's current score, and a very large friends list means a correspondingly large fan-out for one query.

User-visible effect: for players with large friends lists, a friends-rank query could be noticeably slower than a global top-N query, since it genuinely requires gathering more individual data points rather than reading one small, pre-sorted range. How to detect it: friends-query latency correlated with friends-list size, and a fan-out-size distribution tracked so an unusually large list — or a bug inflating apparent list size — is visible before it causes a latency outlier.

Containment: cap the fan-out size per query and/or compute friends-rank asynchronously with a short-lived cached result for very large lists, rather than letting one pathological query size degrade response time unpredictably — this never changes the friends-ranking correctness rule, only how the computation is scheduled. Recovery sequence: not an outage — a latency-tail concern; if a specific fan-out proves consistently too slow, the fix is capping, caching, or precomputing a per-player friends-subset materialized rank, updated incrementally as friends' scores change, rather than recomputing from scratch on every query. How to prevent recurrence: design the friends-ranking data path to scale with friends-list size from the start, and keep the eligible-subset-directly rule enforced in the query logic itself, so a performance shortcut can never accidentally reintroduce filtering-the-global-top-N as an optimization.

Players watching a live leaderboard expect their view to track reality closely, but the delivery channel carrying rank-change updates — a WebSocket or SSE feed — is exactly the kind of best-effort, disconnect-prone path that needs its own resync discipline, independent of whatever is happening in ranking storage itself.

Scenario 22 — WebSocket/SSE disconnect and missed versions

Trigger: a player's live-leaderboard client holds a WebSocket/SSE connection that silently drops — a mobile network handoff, a server restart, an idle timeout — without the client immediately noticing, leaving the client's state at version 42 while the server-side feed has since advanced to version 46. Immediate technical effect: this is Test I. On reconnect, the client reports its last-known version (42); the server compares this against its current version (46) and the configured delta-retention window.

User-visible effect: if deltas 43 through 46 are still retained, the server replays exactly that range and the client's view catches up incrementally, in order, with every intermediate rank change visible; if retention for that range has already expired, the server instead sends a fresh full snapshot at the current version, and the client discards its stale state entirely rather than trying to patch an unbridgeable gap. How to detect it: client-reported version-gap size on reconnect, and delta-retention-window exhaustion rate — how often reconnects fall back to a full snapshot instead of a cheap delta replay.

Containment: no server-side halt; this is a per-client resync handled entirely within the reconnect handshake, with no effect on other connected clients or on the underlying ranking store. Recovery sequence: the delta-replay-or-snapshot decision above is the recovery itself — by the end of the reconnect handshake, the client is guaranteed to be at a known, correct current version, never left silently behind. How to prevent recurrence: size the delta-retention window against realistic reconnect-latency assumptions, and make every feed message carry an explicit version number so gap detection is mechanical, never inferred from timing.

Scenario 23 — Feed messages arrive out of order or duplicated

Trigger: the live feed's own fan-out/transport layer — not the underlying ranking store — delivers two consecutive version updates out of order, or redelivers the same version update twice, due to network retries or a multi-path distribution layer. Immediate technical effect: the client's version-sequence check, the same mechanism from Scenario 22, detects either a duplicate version number already applied, or an out-of-order version — a gap, or a version lower than already applied.

User-visible effect: with correct client-side handling, none — a duplicate version update is a no-op, and an out-of-order one is buffered briefly and reordered, or triggers Scenario 22's gap-resync path if reordering isn't resolved quickly; without correct handling, a naive client could flicker between an older and newer rank state. How to detect it: duplicate-version and out-of-order-version rates on the feed delivery layer, tracked as feed-health metrics separate from ranking-correctness metrics.

Containment: no ranking-side action needed; purely a feed-transport quality issue, resolved entirely client-side via version checks. Recovery sequence: the client discards duplicates, buffers and reorders small out-of-order gaps, or falls back to Scenario 22's full resync path for larger gaps. How to prevent recurrence: provision feed fan-out capacity with headroom, and keep the feed distribution tier isolated from the ranking store's own write path so congestion or retries in fan-out can never backpressure ranking writes.

Scenario 24 — Slow subscriber causes backpressure

Trigger: one specific connected client's consumption rate falls behind the feed publisher's production rate — a slow mobile connection, a backgrounded app, or a client-side processing bottleneck — while every other subscriber keeps up normally. Immediate technical effect: if the publisher buffers per-subscriber outbound messages for a slow reader without a bound, that buffer grows, risking memory pressure on the publisher that could, if unmanaged, degrade delivery to every other, healthy subscriber sharing that publisher instance.

User-visible effect: the slow subscriber itself falls further behind with an increasingly large version gap; no other subscriber should experience any degradation at all — fan-out isolation means one slow reader's problem stays that reader's problem. How to detect it: per-subscriber outbound backlog/buffer depth on the publisher side, with a configured threshold marking a subscriber slow.

Containment: the publisher disconnects a subscriber once its backlog crosses the bound, rather than letting one slow connection degrade delivery to everyone else on the same fan-out path. Recovery sequence: the disconnected client reconnects and goes through Scenario 22's resync handshake — its version gap is now large, likely triggering a full snapshot rather than a cheap delta replay — and resumes receiving live updates from there. How to prevent recurrence: bounded per-subscriber buffers as a hard architectural rule, and where feasible, tiered delivery so subscribers with different capabilities aren't forced onto one path.

Scenario 25 — Cache serves outdated top-N after a rank change

Trigger: a short-TTL cache or precomputed snapshot in front of the live ranking store — the one introduced in Scenario 20 to absorb read surges — has not yet refreshed at the moment a rank-changing event, such as evt-101, is applied to the live store underneath it. Immediate technical effect: reads served from the cache during the gap between the live store's update and the cache's next refresh reflect the pre-update standings — the explicit, bounded staleness the cache layer was designed to trade for read scalability, not an unexpected bug.

User-visible effect: a player refreshing the top-100 view during this narrow window might briefly see Dev still at his pre-evt-101 position before the next cache refresh catches up, bounded by the cache's TTL/refresh interval and stated as part of the system's staleness budget, never presented as instantaneous. How to detect it: periodic cache-vs-live-store divergence checks confirming divergence never exceeds the expected TTL-bound window, and cache refresh-latency monitoring.

Containment: no halt; accepted, bounded behavior, unless the divergence window exceeds its stated bound, in which case it's treated as a cache-refresh-pipeline problem scoped to the cache tier, similar to Scenario 14's lag. Recovery sequence: the next scheduled cache refresh — or an explicit invalidation triggered by significant rank-changing events, if that optimization is in place — brings the cache back in line with the live store. How to prevent recurrence: choose a cache TTL/refresh interval deliberately against the stated staleness budget, and consider event-driven cache invalidation for especially rank-sensitive views rather than waiting out a fixed TTL blindly.

A season boundary is one of the few moments where FlowRank deliberately changes which board a score is even allowed to affect. Getting that transition wrong risks either losing a legitimately earned late score or letting it bleed into a season it was never part of. These three incidents live right at that boundary.

Scenario 26 — Season rollover races with late score submissions

Trigger: Season S7 reaches its scheduled finalization instant and Season S8 opens, but a legitimate Arena Clash match that started just before the boundary finishes and emits its score event a few seconds after S7 is marked finalized — genuine and timely from the player's perspective, but arriving on the wrong side of a now-closed boundary. Immediate technical effect: this is Test H. S7 is finalized (immutable to ordinary writes) and S8 has opened empty; the late event cannot be silently applied to either season automatically — applying it to S7 would mutate a finalized season without authorization, and applying it to S8 would misrepresent a match unambiguously played under S7's rules and window.

User-visible effect: the event is held, not discarded, pending a controlled decision. By policy, matches are defined to belong to the season they started in, and a short, explicitly bounded grace window held open right after the official finalization instant exists specifically for this class of in-flight match, before the season is truly sealed. How to detect it: a late-event-after-finalization rate tracked per season boundary, and the age of the oldest unresolved late event relative to the grace window's expiration.

Containment: no halt to either season's normal traffic; the late event is routed into a grace-window-pending state, closely related to Scenario 07's quarantine but gated by time rather than suspicion, rather than either silently applied or silently dropped. Recovery sequence: if within the grace window and the match's start time qualifies it for S7, the event is applied to S7 as a controlled, audited correction after the fact — finalized season immutable to ordinary writes, but an audited correction process is explicitly not an ordinary write. If the grace window has elapsed, the event is rejected with an explicit reason and escalated for manual resolution rather than quietly discarded. How to prevent recurrence: define and publish the season-boundary grace-window policy before the season even starts, and make the season state machine itself enforce which mutation paths are available in each state — open, grace-window, finalized — so no code path can apply an ordinary write to a finalized season.

Scenario 27 — Season finalized before all accepted events were processed

Trigger: the season-finalization trigger fires — a scheduled time, or an operator action — while the ranking projector still has a non-empty backlog of durably accepted S7 events that haven't yet been applied to the live projection, a Scenario 14-style lag, unlucky specifically in its timing relative to the season boundary. Immediate technical effect: if finalization is implemented as 'snapshot whatever the live projection currently shows and call it final,' the final standings could omit the effect of events that were already durably accepted — and therefore rightfully part of S7 — but simply hadn't been projected yet, a correctness gap distinct from Scenario 26's late-arrival case.

User-visible effect without the fix: a player whose event was accepted before the boundary but not yet reflected in the live rank view could be finalized at the wrong standing. How to detect it: an explicit pre-finalization check querying whether the projector is fully caught up to the last event accepted before the boundary, rather than assuming it is.

Containment: finalization is held — the season stays in a brief 'finalizing' state, not yet immutable — until the backlog of pre-boundary events is confirmed fully applied, a short, bounded wait since the backlog at a single point in time is finite. Recovery sequence: once the backlog drains and every pre-boundary event is confirmed applied, finalization proceeds against the now-accurate projection; if finalization had already incorrectly completed before this check existed, the fix is the same audited-correction path as Scenario 26 — recompute true final standings from the durable event log and apply the difference as a versioned correction, never a silent re-snapshot. How to prevent recurrence: make 'finalize' an explicit state-machine transition that requires projector-lag-at-boundary to be zero, or below a tiny, explicitly accepted threshold, as a precondition, never a simple read-and-freeze operation.

Scenario 28 — Prize winner is disqualified after finalization

Trigger: after S7 is finalized and prizes or recognition have been determined from the final standings, a moderation review — Scenario 08's process — concludes that the season's #1 finisher's winning score was obtained through cheating, and the win must be revoked. Immediate technical effect: a finalized season is immutable to ordinary writes, but a disqualification is explicitly not an ordinary write — it must go through the same audited, versioned correction workflow used for any other post-hoc correction, applied against the already-finalized S7 standings rather than the currently active S8 board, since the correction belongs to the season the score was earned in, never the currently active one.

User-visible effect: the historical S7 standings are updated via a versioned correction — the disqualified player is marked accordingly and every other finalist's position recalculates up by one where applicable — and this correction is itself timestamped and auditable, so anyone reviewing S7's history afterward sees both the original finalized state and the correction, never a silently rewritten record. How to detect it: this is operator/moderation-triggered, not automatically detected; what's monitored is that every post-finalization correction carries a case reference and that the correction workflow explicitly supports operating on seasons beyond just the currently active one.

Containment: no impact on S8 or any other season; the correction is scoped exactly to S7's historical record. Recovery sequence: the correction is applied, and any affected prize or recognition process is notified through its own non-FlowRank workflow to handle real-world consequences, explicitly out of FlowRank's scope beyond providing the corrected, authoritative historical standings. How to prevent recurrence: design the correction workflow from the outset to be season-addressable, able to amend any past season's finalized record, and retain season history per a stated retention policy specifically so a disqualification months after finalization remains possible to apply correctly.

A leaderboard's identity — which game, which mode, which season, which scope (global, a region, a guild, a tournament bracket) — is as important to get right as the scores themselves; mixing two boards' data, even briefly, is a correctness failure just as serious as a wrong score. These six incidents cover identity, cardinality, replication, disaster recovery, corruption, and privacy.

Scenario 29 — Region/mode/scope routing mixes two boards

Trigger: a routing or key-construction bug — a missing scope qualifier in a cache key, a misconfigured regional endpoint, or a deploy that changes key-naming conventions inconsistently across services — causes some reads or writes intended for one leaderboard identity, say arena-clash:solo:S7:global, to be routed against a different one, say arena-clash:solo:S7:region-eu, or worse, a different game entirely. Immediate technical effect: scores or reads cross a boundary that should be hard — a player's global achievement could appear on, or be evaluated against, a regional board it was never meant to affect, or vice versa.

User-visible effect: players could see an unfamiliar name on 'their' board, a score appear to vanish from the board they expect while surfacing on another, or rank computations that make no sense relative to the board a player believes they're viewing — confusing and trust-damaging even when no score value itself is wrong. How to detect it: every event and read carries its full leaderboard-identity tuple (game, mode, season, scope) explicitly, and a validation layer checks that the identity used to write matches the identity declared by the producer or request, treating any mismatch as a hard, loud error, never silently corrected by guessing the intended identity; cross-board anomaly detection — a score appearing on a board its producer was never authorized for, per Scenario 05's check — is a second, independent catch.

Containment: the specific misrouted path is halted or rejected the instant it's detected — writes to the wrong identity stop immediately, and reads from a misconfigured endpoint fail closed rather than silently continuing to serve the wrong board. Recovery sequence: identify every event misapplied to the wrong board using the durable event log's recorded identity metadata, since every event is logged with its full identity tuple regardless of where it ended up being projected, remove its effect from the wrong board's projection, and correctly apply it to its intended board — a correction-and-reconcile sequence closely related to Scenario 08's, just driven by a routing bug rather than a moderation decision. How to prevent recurrence: make the full identity tuple a mandatory, explicit part of every key, event, and query — never inferred from an endpoint URL or a configuration default that could silently drift — and add automated tests that specifically assert cross-board isolation.

Scenario 30 — Leaderboard key/cardinality explosion from unbounded scopes

Trigger: a new scope dimension is introduced, for example a per-guild or per-tournament-bracket leaderboard, without a bound on how many distinct scope values can exist, and the number of distinct small leaderboards created grows unexpectedly large — far more individual sorted-set keys than were planned for. Immediate technical effect: total memory consumption across the Redis fleet grows roughly linearly with the number of distinct scope values, even though each individual board might be tiny — a cardinality problem distinct from Scenario 12's single-hot-key-at-large-size problem; here no single board is overloaded, but the sheer count of boards threatens total capacity.

User-visible effect: if unmanaged, this could eventually degrade service broadly through fleet-wide memory pressure rather than any one board behaving badly — much harder to attribute to a specific cause without key-cardinality monitoring in place. How to detect it: total key count and aggregate memory usage attributable to a given scope dimension, such as 'all guild-scoped boards combined,' tracked as its own capacity metric distinct from any single board's health.

Containment: apply a bound on the scope dimension itself where the product allows it — archive or evict genuinely inactive guild or tournament boards after their relevant window closes, rather than retaining every one forever at full size — and alert well before aggregate capacity is threatened, not after. Recovery sequence: archive, to cheaper, colder storage that's always reconstructible from the durable event log, boards for scopes that are no longer active, freeing live Redis capacity without losing any history. How to prevent recurrence: design every new scope dimension with an explicit lifecycle — active to archived — and a capacity model from day one, rather than treating 'just add a new scope' as free.

Scenario 31 — Multi-region replication lag or region failure

Trigger: FlowRank replicates ranking data across regions for read locality and disaster-readiness, and either normal replication lag widens beyond its usual bound under load, or an entire region hosting a replica — or a regional write path for a region-scoped board — becomes unreachable. Immediate technical effect: for lag, reads served from a lagging regional replica reflect an older moment than the authoritative region's current state, the same kind of bounded, explicit staleness as Scenario 25's cache lag, just at the replication-topology level; for a full region failure, any board whose authoritative write path lived only in that region cannot accept new writes until a failover decision is made.

User-visible effect: players reading from a lagging region see slightly stale standings, bounded by the stated replication-lag budget; players whose board depends on a failed region's write path see new submissions rejected or delayed until failover completes, never silently accepted into a region that can't make them durable and visible. How to detect it: replication-lag metrics per region pair, and region-health/reachability monitoring from outside the affected region, since an unreachable region can't reliably report its own health.

Containment: for lag, no halt — reads continue, labeled with their known staleness bound; for a region failure, the affected board's write path fails closed rather than accepting writes a now-unreachable region can't durably record, and promotion, if a DR region exists, follows the same fencing discipline as any other failover — only one region is ever authoritative for a given board's writes at a time. Recovery sequence: for lag, resolved once the replication pipeline catches back up on its own; for a region failure, promote a healthy region's replica to authoritative for the affected board under a new epoch, reconcile its last-known state against the durable event log, and reopen writes once verified consistent. How to prevent recurrence: state explicit replication-lag and failover-time assumptions up front, rehearse region-failover as a drill, and keep the durable event log itself replicated across regions so no single region's loss can lose data, only temporarily lose availability.

Scenario 32 — Disaster recovery from event history and snapshots

Trigger: a catastrophic loss affecting both the live ranking projection and its usual fast-recovery path — replicas, recent snapshots — simultaneously, for example a broader infrastructure incident taking out a whole availability zone's worth of ranking infrastructure at once, well beyond a single Scenario 11-style node crash. Immediate technical effect: there is no fast 'promote a replica' option left; the only remaining path to a correct ranking state is a full rebuild from the durable, replicated event log, combined with an older verified projection snapshot if one survived, to reduce how far back replay has to start.

User-visible effect: affected boards are unavailable for reads and writes for a genuinely longer window than a routine single-node recovery — an honest, bounded outage while the rebuild runs, never a silent partial or incorrect leaderboard served in the meantime. How to detect it: broader infrastructure monitoring identifying loss of an entire zone's worth of ranking infrastructure, confirmed by independent, out-of-zone monitoring.

Containment: affected boards are marked unavailable explicitly while the rebuild proceeds; unaffected boards on unaffected infrastructure continue operating normally, since identity-scoped isolation limits the blast radius to exactly the infrastructure actually lost. Recovery sequence: confirm which boards' projections were actually lost versus merely unreachable; for each, load the most recent verified snapshot that survived elsewhere, such as a cross-region snapshot copy; replay the durable event log from that snapshot's point forward, applying every event exactly once via the same idempotent-apply logic as live traffic; run full invariant checks — no player duplicated, deterministic sort including ties, totals matching the event history — before reopening the board; reopen and communicate the exact window of unavailability. How to prevent recurrence: replicate both the durable event log and periodic projection snapshots across failure domains, rehearse full rebuild-from-log drills regularly, and size snapshot frequency against a stated recovery-time assumption.

Scenario 33 — Partial data corruption or rank/score divergence

Trigger: a bug, a hardware memory error, or an undetected partial write causes the live ranking projection for a board to quietly diverge from what the durable event log actually implies — no crash, no error, just a projection that's silently wrong, for example a player's displayed score not matching what their event history supports, or a duplicate entry for one player ID in a supposedly deduplicated sorted set. Immediate technical effect: the board keeps serving reads and accepting writes, but against subtly incorrect state — the most dangerous class of incident precisely because nothing visibly fails; a subsequent rank computation could show an impossible ordering without any component reporting an error.

User-visible effect without detection: inconsistent-looking standings — a player appearing at two positions, or a score that doesn't match what the player knows they achieved — eroding trust well before anyone identifies a root cause. How to detect it: continuous invariant checks against the live projection — no player ID appears twice on one board, scores are non-decreasing where highest-score mode guarantees monotonic improvement per player, deterministic sort order including ties holds — plus periodic reconciliation comparing a sample of the live projection against an independently replayed state from the durable event log; any violation is a hard alert, never a soft warning.

Containment: the affected board, or affected player records if isolable, is flagged, and if the divergence is severe enough to risk visibly wrong rankings, reads fail over to a freshly rebuilt view rather than continuing to serve known-corrupted state. Recovery sequence: discard the suspect live projection state for the affected scope, reload the last verified snapshot, and deterministically replay the durable event log forward from there — identical in spirit to Scenario 11/32's rebuild, just triggered by silent corruption rather than an outright crash; resume serving the board only once invariant checks pass clean. How to prevent recurrence: run invariant checks continuously in production, not only during recovery, and run a periodic shadow-replay reconciliation job that independently rebuilds a sample of the projection from the durable log and diffs it against live state as an early-warning system.

Scenario 34 — Player deletion/privacy request affects historical boards

Trigger: a player exercises a data-deletion or privacy right, such as an account-deletion request, requiring removal or anonymization of their personally identifying data, while that player has scores recorded on one or more current and historical — including already-finalized — leaderboards. Immediate technical effect: the request creates tension between honoring the deletion for personal data and preserving the historical accuracy of a finalized season's standings, since removing a top finisher from a finalized board outright would incorrectly imply no one ever achieved that rank, shifting everyone below them up in a way that misrepresents what actually happened.

User-visible effect: FlowRank's resolution is to anonymize the identifying display information for the deleted player's entries while preserving the rank/score slot itself in historical, finalized standings — for example, showing a generic deleted-player label at the position they actually earned — rather than either refusing the request or silently collapsing history to pretend the position was never earned. For a current, non-finalized board, policy may allow full removal from ongoing standings, since no finalized historical record is yet at stake. How to detect it: this is a request-driven workflow, not something auto-detected; what's monitored is that every deletion or privacy request is processed through a defined pipeline touching every board, current and historical, the player appears on.

Containment: no halt to any board; the anonymization or removal is applied as a scoped, audited update affecting only that player's identifying fields, never other players' scores or rank positions. Recovery sequence: not applicable as an incident — a routine, request-driven data-handling procedure; if a board is later found to have been missed, it's corrected via the same audited-correction workflow used elsewhere, applied retroactively to that specific gap. How to prevent recurrence: design the deletion/privacy pipeline up front to enumerate every board a player's data could appear on, keep the anonymize-identity-but-preserve-earned-position policy explicit and documented, and treat the pipeline as a first-class, tested workflow rather than a manual one-off.

The last two incidents are about the ranking policy's own definition — the comparator that decides who's actually ahead of whom — and what happens when the inputs that comparator depends on, a mid-season rule change or a clock, aren't as stable as the rule assumes.

Scenario 35 — Ranking comparator/tie-break changes mid-season

Trigger: an operator or product decision changes the tie-break rule, or some other aspect of the ranking comparator, partway through an active season — for example moving from 'earlier achievement timestamp wins ties' to a different tie-break policy, applied while S7 is still open and accumulating scores. Immediate technical effect: every already-recorded score pair previously ordered under the old comparator could, in principle, re-sort differently under the new one — a mid-season comparator change is not a small, local edit, it's a potential re-ordering of the entire board.

User-visible effect without careful handling: players could see their rank shift without having earned or lost anything themselves, purely because the rule for breaking ties with someone else's score changed underneath them — confusing and trust-damaging if not clearly communicated and deliberately scoped. How to detect it: this is a planned, operator-initiated change, not something to detect as an anomaly; what's monitored is the magnitude of the resulting re-sort, how many players' positions actually move, so the operational impact is known before the change goes live, not discovered after.

Containment: the comparator change is deployed as an explicit, versioned event, closely related to how Scenario 08's corrections are versioned, applied atomically across the whole board — never a gradual rollout where some reads use the old comparator and others use the new one simultaneously, which could produce a transiently self-inconsistent ordering with no change to either player's score. Recovery sequence: not applicable if done correctly — the board re-sorts once, deterministically, under the new comparator, and that becomes the new deterministic sort going forward; if a non-atomic rollout did cause a transient inconsistent-ordering window, the fix is confirming every read path now agrees and that no rank-change notification was sent based on the transiently inconsistent state. How to prevent recurrence: avoid changing the comparator mid-season wherever possible, scoping rule changes to season boundaries per Scenarios 26–28's lifecycle discipline; where unavoidable, apply it as a single atomic, versioned, clearly communicated transition, never a gradual or silently rolled-out one.

Scenario 36 — Clock skew affects achievement-time tie-break

Trigger: the tie-break rule depends on earlier achievement timestamp winning, and the wall clock on a game-server instance producing achievement events has drifted out of sync with the fleet — an NTP failure, container clock skew — so the timestamp it attaches to an event doesn't accurately reflect when the achievement actually happened relative to other servers' events. Immediate technical effect: Ben and Chloe's tie at 9,500 is broken today by Ben's earlier timestamp (10:01:00 versus 10:02:00); if the server that recorded one of those timestamps was meaningfully skewed, the recorded order between two genuinely close achievements could be wrong relative to the real-world sequence in which they happened, even though FlowRank's own tie-break logic is applied correctly to whatever timestamps it was given.

User-visible effect: a tie-break result that doesn't match what either player believes actually happened first — distinct from Scenario 03's late-delivery case, since here the event arrived promptly; the timestamp attached to it at the moment of achievement was simply wrong. How to detect it: continuous clock-synchronization monitoring, NTP offset per game-server instance, with alerting past a configured tolerance, plus an audit check flagging tie-break outcomes between two achievements recorded unusually close together in time, where clock-skew-driven mis-ordering is most plausible, for closer scrutiny.

Containment: no halt and no automatic re-ordering based on suspicion alone — a flagged close tie-break is surfaced for review via the same quarantine-style holding pattern as Scenario 07, not silently reversed, since reversing it without evidence risks being wrong in the opposite direction. Recovery sequence: resynchronize the drifted server's clock via NTP/PTP immediately to stop the problem from recurring; for the specific tie-break already decided, only correct it if an actual investigation, such as independently corroborated match-completion telemetry, establishes the true order — otherwise the originally recorded timestamps stand, since mere suspicion of clock drift is not by itself grounds for a correction. How to prevent recurrence: require all event producers to run continuous, monitored time synchronization as a baseline operational requirement, and consider recording a secondary, monotonic sequence hint alongside the wall-clock achievement timestamp specifically to make tie-breaks more robust to this failure mode.

The 36 scenarios above describe what breaks; this closing chapter puts illustrative numbers on how the design behaves as load grows. Every number below is a stated assumption for this educational exercise, not a verified production benchmark for Arena Clash, FlowRank, or any real game or leaderboard service.

10× DAU growth, a 20× daily peak multiplier, and a 100× tournament burst

Assumption: start from an illustrative baseline of 1 million daily active players, each submitting on average 5 score-relevant events per day — roughly 58 events/sec sustained average, spread across a 24-hour day. Growing DAU 10× to 10 million players scales that sustained average proportionally to roughly 580 events/sec. Within any single day, traffic is never flat; assume a 20× peak-to-average multiplier for a normal evening prime-time peak, putting peak load at 10× DAU around 11,600 events/sec. A tournament burst (Scenario 19) compounds on top of normal peak, illustratively modeled here as 100× the *average* baseline rate for a short window — about 5,800 events/sec at the original 1 million DAU baseline, or roughly 58,000 events/sec if the same tournament lands after 10× DAU growth has already happened.

A single board with 10 million ranked players

Assumption: a Redis sorted-set member costs roughly 80–100 bytes of overhead once score, skip-list pointers, and a player-ID string are accounted for — an illustrative range, not a measured figure for any specific Redis build or configuration. At 10 million ranked players, that's roughly 10,000,000 × 90 bytes ≈ 900 MB for the sorted set itself, before replication; with one replica for durability, total footprint for that one board is roughly 1.8 GB. Importantly, raw size by itself is not the limiting factor here — ZREVRANK on a skip list is O(log N), and log₂(10,000,000) is only about 23, so a 10-million-member board is still fast to query exactly. What actually forces sharding (Scenario 17) at this scale is the single-threaded write-ordering ceiling on one hot key (Scenario 12), not the key's depth.

Many small boards vs. one extremely hot board

Assumption: the same aggregate write rate behaves very differently depending on how it's distributed across keys. A single hot instrument-style board absorbing, illustratively, 50,000 events/sec is bounded by that one partition's single-writer ceiling, regardless of how much total hardware exists elsewhere in the fleet — the core lesson of Scenario 12. The same 50,000 events/sec spread evenly across 10,000 small, independent boards (5 events/sec/board on average, such as per-guild boards) is comfortably within any one key's headroom, and the system scales this balanced case roughly linearly by adding more independent shards or keys. The architectural takeaway mirrors every partitioned system: horizontal scale works across keys, never within one key.

Read-dominant vs. write-dominant workload shapes

Assumption: under normal conditions, a consumer-game leaderboard is typically read-dominant — illustratively, players check standings on the order of 50–100× more often than they set a new score worth writing. That shape is exactly why Scenarios 20 and 25's caching discipline matters more day-to-day than raw write-path optimization. A tournament burst (Scenario 19) can transiently flip this ratio toward write-dominant, since everyone is both setting new scores and checking live standings at once — both the write path and the read path spike together, which is precisely why that scenario calls out ingestion, broker, projector, and read-path pressure simultaneously rather than treating it as a pure write problem.

Memory and storage sizing, and durable log growth

Assumption: beyond the live projection's memory footprint, the durable event log itself grows with raw event volume. At the 10 million DAU / 5 events-per-day assumption above, and an illustrative 200 bytes per durably logged event (payload plus metadata), that's roughly 10,000,000 × 5 × 200 bytes ≈ 10 GB/day of raw log growth. Over a 90-day season, that's roughly 900 GB of raw, pre-compaction log data for one season's worth of events across the whole player base — a number that makes compaction and cold-storage tiering a first-class design concern, not an afterthought, for any season that runs this long at this scale.

Projection lag under load

Assumption: if a projector sustains a throughput of T events/sec and a burst arrives at rate B > T, the backlog (Scenario 14) grows at roughly (B − T) events/sec for the duration of the burst, and draining it afterward takes roughly backlog ÷ T. Illustratively, a projector sustaining 20,000 events/sec facing the 58,000 events/sec tournament-burst assumption above accumulates backlog at roughly 38,000 events/sec; a 5-minute burst would leave a backlog of roughly 11.4 million events, which the same 20,000 events/sec projector would need roughly 9.5 minutes to drain afterward — directly motivating Scenario 19's admission throttling, since letting the backlog grow unchecked would otherwise keep pushing the recovery tail further out.

Subscriber fan-out cost

Assumption: live-feed fan-out scales independently of the ranking/matching workload, since publishing one rank-change update once and fanning it out to many subscribers is a delivery-tier concern, not a ranking-store concern. If a feed publishes, illustratively, 1,000 updates/sec at 150 bytes/update to 10,000 concurrently connected subscribers, that's roughly 1,000 × 150 × 10,000 ≈ 1.5 GB/sec of aggregate egress bandwidth. Growing subscribers 10× to 100,000 grows aggregate egress roughly 10× to about 15 GB/sec — purely a fan-out/network-capacity problem addressed by more publisher replicas and distribution-tier capacity, with zero additional load placed on the ranking store or the projector.

Exact rank vs. approximate rank at scale

Assumption: once the 10-million-player board from above is sharded across, illustratively, 16 shards to relieve the single-hot-key ceiling, a standard cross-shard top-N merge (Scenario 17) becomes an approximation whose error grows modestly with shard count and skew. Maintaining a genuinely exact global-rank index instead (Scenario 18) means either a full merge-and-count on every exact-rank query, whose cost scales with shard count and total size, or a dedicated exact-rank structure updated on every single write — illustratively adding a second write (and its own durability/ordering cost) to every score-changing event, roughly doubling write-path cost specifically to support the exact-rank feature. That doubled write cost, paid on every event regardless of whether any player ever requests an exact rank that day, is the concrete number behind the Trade-offs & Decisions discussion of when exact rank is actually worth it.

Season archival storage growth

Assumption: using the ~900 GB-per-season raw log figure above, retaining every finalized season's full raw event log indefinitely across, illustratively, 20 completed seasons would grow to roughly 18 TB of raw, uncompacted historical data — before any compaction, deduplication, or cold-storage tiering is applied. In practice, a finalized season's authoritative recovery need is its final standings snapshot plus its event log for audit/correction purposes (Scenario 28), so compacting older, rarely-accessed seasons into cheaper cold storage, while keeping only the current and immediately prior season's log readily warm, is the lever that keeps this growth from becoming unbounded operational cost.