System Design · 10 chapters · 26 named scenarios · ~36 min read
Feature Flag System Failures: Outages, Staleness & Scale
Twenty-six named failure and scale scenarios, each walked through with a concrete timeline and the mechanism that actually recovers it. All tenant names, user IDs, version numbers, and traffic figures are fictional illustrative placeholders for this module (Northstar Cloud), never measured production data.
Scenario Group 01
Deployment & Release Process Failures
Before any distributed-systems failure exists, the process-level coupling between deploying code and releasing behavior is its own source of incidents.
Scenario: redeploy-for-one-tenant blocks an unrelated hotfix
With a build-time flag, enabling the new dashboard for Tenant Atlas alone requires a full redeploy. That redeploy happens to also ship an unrelated, half-finished change that was merged to the same branch earlier that day — now live for every tenant, not just Atlas.
Recovery / prevention
Separate the deploy pipeline from the release decision entirely: ship code continuously behind flags defaulted off, and let a flag change — not a deploy — be the action that changes user-visible behavior.
Scenario: an all-tenants-simultaneous release causes a shared-dependency spike
A build-time flag flip goes live for all 200 tenants at the exact same moment the new deployment finishes rolling out. The new dashboard calls a downstream analytics service that was sized for gradual adoption, not an instant step-function in traffic.
Recovery / prevention
A percentage-rollout-capable flag platform lets a release ramp from 1% to 100% over hours or days, giving downstream dependencies time to be observed and, if necessary, scaled before full exposure.
Scenario Group 02
Central Database Failures (Naive Design)
Scenario: flags database outage mid-traffic
The naive V1's flags database has a brief outage — a failed maintenance operation. Every request that synchronously checks a flag now also fails, turning a configuration-store blip into a full application outage within seconds.
Recovery / prevention
Not a retry or failover at the database layer — the structural fix is moving evaluation off the synchronous per-request path entirely, onto a local SDK snapshot that doesn't depend on the database being reachable at evaluation time.
Scenario: flag-check volume degrades unrelated application queries
As traffic grows, the flags database — shared with other application tables in this naive design — starts seeing enough flag-check load that connection pool exhaustion causes unrelated application queries to slow down too, even though those queries have nothing to do with feature flags.
Recovery / prevention
Isolate the flags database from application data stores, and more importantly, stop putting flag checks on the same per-request hot path as application queries at all — local evaluation removes this coupling entirely.
Scenario Group 03
Snapshot Staleness Failures
Scenario: a stale SDK serves the wrong behavior after a confirmed admin change
An admin publishes a change and sees it reflected in the admin UI immediately. A support ticket comes in minutes later: a specific user still sees the old behavior. The relevant app instance's SDK hasn't polled or received a push yet.
Recovery / prevention
This is expected, bounded behavior, not a bug — as long as the propagation window is within the documented SLO. The fix for the support workflow is visibility: expose each SDK instance's currently-active config version so support can distinguish 'still propagating' from 'genuinely broken.'
Scenario: a stuck poller never refreshes at all
A bug in one SDK instance's polling loop causes it to silently stop polling after a transient network error, rather than retrying. That instance serves an increasingly stale snapshot indefinitely, with no automatic recovery.
Recovery / prevention
A hard staleness ceiling: if an SDK's snapshot age exceeds a documented maximum (far beyond the normal propagation SLO), it should emit a loud, monitorable health signal — and in some deployments, fail toward a safe default rather than silently keep serving an arbitrarily old configuration.
Scenario Group 04
Tenant Isolation Failures
Scenario: cross-tenant credential probe
A request authenticated as Tenant Birch attempts to read Tenant Cedar's production configuration through the admin API — possibly a bug in an integration, possibly deliberate.
Recovery / prevention
The control plane rejects it with a 403 based on the verified credential's actual tenant scope, server-side, before any row is read — tenant identity must never be derived from a client-supplied value.
Scenario: a shared cache key collides across two tenants
Two different tenants independently create a flag named new-dashboard. A distribution-layer cache keyed only on flag key — not the full tenant/project/environment tuple — serves one tenant's rules to the other's SDK.
Recovery / prevention
Every cache and CDN key in the distribution path must include the full namespace tuple, never just the flag key. This has to be verified deliberately — it's exactly the kind of isolation gap that passes every authorization test while still leaking data.
Scenario Group 05
Operator & Process Failures
Not every incident is a distributed-systems problem — some of the most damaging ones are a human with valid credentials doing the wrong thing, or a lifecycle step nobody owned.
Scenario: an operator publishes to production while believing they're in staging
An admin-console tab left open from an earlier staging session is used to publish a change — the environment selector looks similar enough across tabs that the operator doesn't notice until customer reports arrive. The change was correct for staging and actively wrong for production.
Recovery / prevention
Require an explicit, visually distinct confirmation naming the target environment and tenant before any production publish — not a generic 'are you sure?' — plus the approval gate (Architecture Deep Dive, Chapter 06) as a second independent check that would catch this even if the first operator didn't notice. Recovery is the same rollback mechanism as any bad publish: a new version restoring the last known-good state, never a rewrite.
Scenario: a compromised admin credential attempts an immediate production rollout
A leaked admin API token is used to attempt publishing a flag change directly to production, bypassing the normal review process by moving fast before anyone notices.
Recovery / prevention
RBAC scoping (credential → specific environment/project, never blanket admin) limits blast radius even if a token leaks. The approval gate structurally prevents a single credential — compromised or not — from publishing to production alone. The immutable, tenant-scoped audit log (Architecture Deep Dive, Chapter 03) is what makes the attempt visible and attributable after the fact, and credential revocation plus a forced key rotation closes the hole once detected.
Scenario: a flag is deleted while an old, un-upgraded app version still evaluates it
A flag is removed from the control plane as part of routine cleanup once a rollout reaches 100% and the old code path is believed to be fully retired. An old, not-yet-redeployed instance of the application still calls evaluate('old-flag-key', ...) and now hits a flag that no longer exists in any snapshot.
Recovery / prevention
Evaluation returns the caller-supplied safe default with reason FLAG_NOT_FOUND (Architecture Deep Dive, Chapter 04) rather than erroring — this is exactly the invariant that makes flag removal safe for instances that haven't redeployed yet. The real prevention is process, not mechanism: a documented flag-retirement policy requiring confirmation that no deployed version still references a key before deletion, not just that the rollout reached 100%.
Scenario Group 06
Targeting & Bucketing Failures
Scenario: naive per-request randomness flickers a user's experience
A rollout implemented as Math.random() < 0.10 on every request means the same user can see the dashboard appear and disappear across consecutive page loads within the same session.
Recovery / prevention
Replace per-request randomness with a deterministic hash of a stable identity — the fix is architectural, not a patch; there's no safe way to make per-request randomness consistent without removing the randomness from the hot path entirely.
Scenario: a salt rotation accidentally reshuffles an active rollout's cohort
An unrelated edit to a flag's metadata also regenerates its bucketing salt as a side effect of a careless admin-tool implementation. Every user's bucket membership changes at once — including users who were already relying on being consistently ON.
Recovery / prevention
Salt changes must be a rare, explicit, separately audited action — never an incidental side effect of editing something else. Treat salt as part of the flag's stable identity, not its mutable configuration.
Scenario: wrong bucketing unit chosen for a B2B feature
A ten percent rollout bucketed by user ID puts some of Tenant Cedar's employees on the new dashboard and others on the old one, inside the same company workspace — confusing for a collaborative feature where consistency across a tenant's whole team matters.
Recovery / prevention
Switch the bucketing key to tenant ID for this specific flag, so the rollout decision is consistent for every user inside a given tenant, at the cost of a coarser statistical sample.
Scenario: a missing or malformed targeting key produces random-looking assignment
A client library upgrade accidentally stops populating the context's bucketing key for a subset of requests — anonymous sessions before login, say. stableHash(..., bucketingKey=null) either throws or hashes a constant placeholder, and every affected request either errors or lands in the same bucket, looking statistically broken to anyone monitoring rollout distribution.
Recovery / prevention
Evaluation must treat a missing bucketing key as a defined, explicit case — fail to a documented default with a distinct reason (e.g. MISSING_BUCKETING_KEY), never silently hash a null or constant placeholder into the normal bucket space, which corrupts the statistical validity of the rollout without any visible error.
Scenario: Java and JavaScript SDKs disagree on bucket membership for the same user
A backend service (Java SDK) and a server-side rendering layer (Node SDK) evaluate the same flag for the same user and get different variations — because one SDK normalizes the hashed tuple's string encoding slightly differently (trailing whitespace, numeric-to-string formatting) than the other, producing different hash inputs for logically identical data.
Recovery / prevention
A canonical serialization specification (exact field order, exact string encoding rules, no locale-dependent number formatting) must be part of the bucketing contract itself, not left to each SDK implementation's discretion — and a cross-language conformance test suite, run against every SDK with identical fixture inputs, is the only reliable way to catch a normalization mismatch before it reaches production.
Scenario Group 07
Distribution & Versioning Failures
Scenario: out-of-order version delivery
Version 13 activates on an SDK instance. Moments later, a delayed retry from an earlier publish attempt delivers version 12 to the same instance.
Recovery / prevention
The SDK compares version numbers and rejects anything older than what's already active — enforced by the SDK itself, since the delivery path's ordering can't be trusted.
Scenario: a delta applied against the wrong base
An SDK that missed an earlier update is on version 11. A delta arrives assuming a base of version 12. Applying it blindly would produce a logically inconsistent configuration.
Recovery / prevention
The SDK checks the delta's declared base against its current version; on mismatch, it falls back to fetching the full snapshot instead of risking silent corruption.
Scenario: a malformed configuration is published
An edit introduces a cyclic prerequisite between two flags — flag A requires flag B, which requires flag A. If this reached an SDK unvalidated, evaluation could recurse indefinitely.
Recovery / prevention
The snapshot builder validates structurally before publishing — cyclic prerequisites, duplicate keys, invalid weights are all rejected at the source, so no SDK ever receives a structurally broken snapshot.
Scenario: duplicate outbox delivery triggers repeated snapshot activation
The outbox consumer (Architecture Deep Dive, Chapter 06) redelivers the same publish event twice — a consumer restart mid-processing, or an at-least-once delivery guarantee doing exactly what it promises. A naive snapshot builder that isn't idempotent could build and publish two snapshots for what was logically one config change, incrementing the version counter twice for a single edit.
Recovery / prevention
The snapshot builder keys its own processing on the outbox event's unique sequence number and is idempotent against redelivery — reprocessing the same event number is a no-op if that version has already been built. SDK-side monotonic activation (Architecture Deep Dive, Chapter 08) provides a second independent layer of protection: even if a duplicate snapshot were somehow published, an SDK already running the resulting version simply ignores the redundant delivery.
Scenario Group 08
Outage & Recovery Scenarios
Swipe to pan
Scenario: control-plane outage during a warm fleet
The control plane and config database go fully offline for an extended maintenance window. Every already-running application instance keeps serving traffic normally from its last-loaded snapshot.
Recovery / prevention
No action needed on the data plane — this is the design working as intended. The actionable gap is the next scenario: what happens to a brand-new instance that starts during this exact window.
Scenario: cold start during a control-plane outage
An autoscaling event launches a new instance while the control plane and distribution layer are both unreachable. The new SDK has never loaded a snapshot and has nothing to fall back on.
Recovery / prevention
Enforce the documented per-flag cold-start default policy (fail closed, fail open, or block startup) rather than leaving the behavior to whatever an empty map lookup happens to return.
Scenario: a rollout elevates error rates, requiring rollback
A twenty percent rollout correlates with a measurable increase in application error rates. An operator needs to safely revert.
Recovery / prevention
Publish a new version restoring the last known-good configuration — never delete or rewrite the problematic version — and monitor fleet-wide version adoption to confirm the rollback has actually reached production traffic before declaring the incident resolved.
Swipe to pan
Scenario: a regional kill switch has observable propagation delay
An emergency kill switch publishes immediately. Region East activates it within a few seconds. Region West, on a separate distribution path, is still serving the previous version for a few seconds longer.
Recovery / prevention
This is not a bug — it's bounded eventual consistency that must be measured, monitored, and documented as an explicit SLO, never silently assumed to be instantaneous.
Scenario: a CDN edge serves a stale snapshot after an emergency revocation
An emergency kill switch publishes a new version immediately, but a CDN edge node — caching the snapshot response by URL, with a longer TTL than anticipated — continues serving the previous version to SDKs in its region for several minutes past the intended propagation window.
Recovery / prevention
Snapshot URLs are versioned (the version number is part of the cache key, not just a path that gets overwritten), so a CDN caching by URL is structurally caching an immutable artifact — it can't serve stale content for a version-specific URL. An SDK's 'is there a newer version' check must query a cache-bypassed or very-short-TTL endpoint even if the versioned snapshot payload itself is aggressively cached, and emergency kill-switch publishes should explicitly issue a CDN invalidation as a belt-and-suspenders measure alongside the version check, bounding the window rather than trusting the version check alone.
Scenario Group 09
Scale Scenarios
Scenario: 20,000 SDK instances polling one origin
At illustrative fleet scale — 20,000 backend SDK instances, each polling every 10 seconds — the distribution layer sees roughly 2,000 requests per second, a completely different and far smaller number than the roughly 10 million local evaluations per second happening fleet-wide.
Mitigation
This is manageable without special measures at this scale; it becomes a reason to add a regional relay/CDN tier only once origin fan-out itself becomes the bottleneck at a materially larger fleet size.
Scenario: a reconnect storm after a regional network blip
A brief regional network interruption drops every open polling/push connection simultaneously. If every one of 20,000 instances retries at the same instant, the resulting thundering herd can itself cause a second, self-inflicted outage on the relay tier.
Mitigation
Exponential backoff with jitter spreads reconnection attempts over several seconds instead of one spike — the same mitigation any reconnect-storm problem needs, paired with rate limiting and backpressure on the relay tier itself as a second line of defense.
Scenario: regional distribution bounds origin fan-out
As fleet size grows well past the illustrative 20,000-instance example, a single origin serving every instance directly becomes the actual bottleneck, regardless of how efficient each individual poll is.
Mitigation
A regional relay/CDN layer bounds the origin's fan-out to a small, fixed number of regions; each region absorbs its own local instance population independently.
Scenario: a hot tenant's snapshot grows large enough to spike SDK memory and bandwidth
One large tenant accumulates several thousand flags, each with multiple targeting rules, over years of product growth. That tenant's full snapshot grows to tens of megabytes — every SDK instance in that namespace now holds that much more in memory, and every poll (even a 304-avoided one) has a larger payload the one time it does need to fetch fresh content.
Mitigation
A documented per-snapshot size limit, enforced at publish time (rejecting or warning on a snapshot exceeding it) surfaces the problem before it becomes an incident, not after. For genuinely large tenants, differential/incremental fetch (Architecture Deep Dive, Chapter 08) bounds the common-case payload to what actually changed rather than the full snapshot on every update; flag archival for long-unused flags bounds growth at the source.
Scenario Group 10
Worked Capacity Example (Illustrative)
A single illustrative worked example, to make the local-vs-central traffic split concrete. All figures are hypothetical planning numbers, not measured production data.
| Quantity | Formula | Result |
|---|---|---|
| SDK instances (illustrative) | given | 20,000 |
| Flags evaluated per instance per second (illustrative) | given | 500 |
| Total local evaluations / sec | 20,000 × 500 | 10,000,000 / sec |
| Poll interval (illustrative) | given | 10 seconds |
| Central poll requests / sec | 20,000 ÷ 10 | 2,000 / sec |
| Ratio: local evaluation vs. central traffic | 10,000,000 ÷ 2,000 | 5,000× more local than central |
The number that has to survive a central, synchronously-reachable service (2,000 req/s) is orders of magnitude smaller than the number that must stay entirely local (10 million evaluations/s). A design that routes the larger number through a central service at every request — the naive V1 from Complete Design — was never going to scale, independent of how well-provisioned that central service is.