System Design · 20 chapters · ~34 min read
Feature Flag System Design: Trade-offs & Decisions
Twenty concrete decision records — each with the options actually on the table, honest pros and cons, and the reasoning that would hold up when a reviewer pushes back. None of these are the only correct answer; they're the defensible answer for the requirements this module set out in Complete Design.
Decision 01
Bucketing Unit: User, Tenant, or Device?
The percentage-rollout bucketing key changes what '10%' actually means.
Swipe to pan
| Option | Pros | Cons |
|---|---|---|
| User ID | Broad behavioral sampling across every tenant; good for general UX validation | A single large tenant can have a very different effective exposure than intended if their users dominate the sample |
| Tenant ID | Every user inside a tenant gets a consistent experience; easy to reason about for B2B support tickets | Coarser statistical sample — fewer independent data points for the same percentage |
| Device/session ID | Useful for anonymous, pre-login flows | Same user can land in different buckets across devices — can read as flaky if surfaced cross-device |
Decision 02
Distribution: Poll-Only, Push-Only, or Both?
How SDKs learn about a new published version.
Swipe to pan
| Option | Pros | Cons |
|---|---|---|
| Poll-only | Simple, stateless, no persistent connection infrastructure, degrades predictably | Freshness strictly bounded by interval; shorter intervals cost more request volume |
| Push-only | Lowest typical latency for a change to reach a connected SDK | A missed or dropped notification has no fallback — an SDK can silently stay stale indefinitely |
| Poll + push (chosen) | Push gives low typical latency; polling guarantees eventual correction if a push is missed | Slightly more implementation surface — two code paths that both have to agree on activation rules |
Decision 03
Bounded Staleness vs. a Promised Instant Kill Switch
Could the design promise an instantaneous global kill switch instead of a bounded propagation window?
| Option | Pros | Cons |
|---|---|---|
| Promise instant global kill | Simpler story to tell stakeholders | Not actually achievable without a synchronous call on every evaluation — which reintroduces the exact outage risk this design exists to avoid |
| Bounded, measured propagation window (chosen) | Honest, testable SLO; preserves local-evaluation availability guarantees | Requires documenting and monitoring an actual propagation p99, and accepting a real (if small) window where regions disagree |
Decision 04
Snapshot Delivery: Full Snapshot vs. Delta
How much of the configuration does an SDK refetch on each update.
Swipe to pan
| Option | Pros | Cons |
|---|---|---|
| Full snapshot every time | Simple, self-healing — an SDK can never apply a delta on the wrong base | Wasteful bandwidth at scale, especially for large configs with small changes |
| Delta with full-snapshot fallback (chosen) | Much smaller payloads in the common case; falls back safely when base version doesn't match | More implementation complexity — base-version checking and a fallback path must be tested, not assumed |
Decision 05
Salt Lifetime: Fixed Forever vs. Rotatable
Should a flag's bucketing salt ever change?
| Option | Pros | Cons |
|---|---|---|
| Fixed salt for the flag's lifetime | Preserves monotonic-expansion guarantees automatically; predictable support/debugging story | A rollout that needs a genuinely fresh random sample (e.g., after a rollback) can't get one without a new flag |
| Rotatable salt, explicit and audited (chosen) | Supports a deliberate full reshuffle when genuinely needed | Breaks monotonic expansion the moment it's used — must be a rare, explicit, logged action, not an incidental side effect of an unrelated edit |
Decision 06
Cold-Start Default: Fail Closed, Fail Open, or Block?
What a brand-new SDK instance does before it has ever loaded a snapshot.
| Option | Pros | Cons |
|---|---|---|
| Fail closed (flag default = off) | Safe for risky, unreleased behavior — worst case is a feature doesn't show up yet | Wrong choice for a flag gating already-proven, low-risk behavior — can cause unnecessary regressions on cold start |
| Fail open (flag default = on) | Appropriate for mature, low-risk flags where 'on' is the expected steady state | Dangerous default for a flag still being validated |
| Block startup until a snapshot loads (bounded timeout) | Guarantees no evaluation happens against an undefined state | Adds startup latency and a new failure mode (what happens if the timeout is hit) |
Decision 07
Distribution Topology: Single Origin vs. Regional Relay
How the published snapshot physically reaches tens of thousands of SDK instances.
| Option | Pros | Cons |
|---|---|---|
| Single central origin | Simplest to operate and reason about at moderate scale | Fan-out to every instance worldwide becomes the origin's own direct load; cross-region latency affects every poll |
| Regional relay/CDN layer (chosen at scale) | Origin only serves a handful of regions; each region serves its own local population — bounded, predictable fan-out | An additional operational tier; introduces its own regional propagation delay to reason about (see the kill-switch decision above) |
Decision 08
RBAC & Approval: Centralized Control Plane vs. Per-Tenant Admin
Where authentication, authorization, and approval workflows live.
| Option | Pros | Cons |
|---|---|---|
| Per-tenant, self-hosted admin | Maximum tenant autonomy | Duplicated governance logic; inconsistent audit quality; harder to guarantee isolation across independently-built admin surfaces |
| Centralized control plane (chosen) | One well-tested authorization and audit implementation shared by every tenant; consistent approval workflow | A single component whose correctness every tenant depends on for administration — though not for evaluation, by design |
Decision 09
Browser SDK: Ship Full Rules vs. Pre-Evaluated Values
What a browser actually receives for client-side evaluation.
Swipe to pan
| Option | Pros | Cons |
|---|---|---|
| Ship full targeting rules to the browser | Browser can re-evaluate locally without a round trip for every context change | Leaks segment definitions, rollout percentages, and other tenants' targeting logic to anyone opening dev tools |
| Ship only pre-evaluated values for this context (chosen) | No sensitive targeting logic ever reaches an untrusted environment | Any context change (e.g., user logs in) needs a fresh bootstrap call to get updated evaluated values |
Decision 10
Telemetry: Synchronous Confirmation vs. Fire-and-Forget
Whether an evaluation call waits for its exposure event to be recorded.
| Option | Pros | Cons |
|---|---|---|
| Synchronous telemetry write | Guarantees every exposure event is recorded before evaluation returns | Makes evaluation latency and availability depend on the telemetry pipeline — directly violating the local-fast-evaluation requirement |
| Async, non-blocking telemetry (chosen) | Evaluation latency and availability are completely independent of the telemetry pipeline's health | A telemetry outage can silently drop exposure events — acceptable, since exposure data is for analysis, not authoritative state |
Decision 11
Flag Prerequisites: Allow Chains, or Forbid Them?
Should one flag be allowed to depend on another flag's value?
| Option | Pros | Cons |
|---|---|---|
| No prerequisites allowed | Every flag evaluates fully independently — simplest mental model, no cycle risk | Forces duplicated targeting logic across related flags (e.g., a 'beta UI' flag and a 'beta UI advanced settings' flag that should only ever both be on together) |
| Prerequisites allowed, cycle-checked (chosen) | Expresses real dependency relationships directly; avoids duplicated rules | Requires explicit cycle detection and a documented max-depth, or risks infinite recursion on a misconfigured graph |
Decision 12
Validation Timing: At Edit Time, Publish Time, or SDK-Side?
Where a malformed configuration gets caught.
| Option | Pros | Cons |
|---|---|---|
| Trust each SDK to validate on load | No extra central validation step | Every SDK implementation (potentially in different languages) must independently implement identical validation — drift is likely, and a bad config can reach production traffic before any SDK rejects it |
| Validate centrally at publish time (chosen) | One authoritative validator; a malformed version never reaches any SDK in the first place | Requires the snapshot builder to understand every rule type an SDK might need to evaluate — validation logic has to stay in sync with evaluation logic |
Decision 13
Flag Types: Boolean-Only vs. Multivariate
Whether flags can return more than true/false.
| Option | Pros | Cons |
|---|---|---|
| Boolean-only | Simpler evaluation model, simpler admin UI, covers most on/off release use cases | Can't express A/B/C experiments or gradual numeric rollouts (e.g., a rate limit threshold) without stacking multiple boolean flags awkwardly |
| Multivariate — boolean, string, number, JSON (chosen) | One mechanism covers release toggles, experiments, and remote-configurable parameters | Larger payloads per flag; targeting rules and percentage splits need to handle N variations instead of just two |
Decision 14
Audit Trail: Coarse Event Log vs. Full Field-Level Diff
How much detail the audit trail captures per change.
| Option | Pros | Cons |
|---|---|---|
| Coarse event log ("flag X was edited by Y at time T") | Cheap to store and query | Not enough detail to answer 'what exactly changed' during an incident review without separately diffing snapshot versions |
| Full field-level diff per change (chosen) | Supports precise incident review and compliance reporting directly from the audit record | More storage and more care needed to ensure the diff itself doesn't leak data across tenant boundaries |
Decision 15
SDK-Local vs. Server-Side Remote Evaluation
Where the actual evaluate() call runs — inside the caller's own process, or as a request to a remote service.
| Option | Pros | Cons |
|---|---|---|
| SDK-local evaluation (chosen, default) | Sub-millisecond, no network dependency on the hot path, survives a control-plane outage entirely | Every SDK language needs its own correct, kept-in-sync implementation of the evaluation and bucketing algorithm |
| Server-side remote evaluation (POST /evaluate) | No SDK to install or keep updated; works from any language or environment via plain HTTP | Reintroduces a synchronous network dependency and a central point of latency/availability risk on every evaluation — exactly what local evaluation exists to avoid |
Decision 16
Relational Source of Truth vs. Document/KV Store
What the control plane's durable system of record actually is, underneath the outbox and snapshot layers.
| Option | Pros | Cons |
|---|---|---|
| Relational (chosen) | Compound uniqueness constraints (tenant/project/environment/key) and foreign-key integrity are enforced by the database itself, not application code; transactions make the outbox pattern straightforward | Schema migrations for new rule types or metadata fields require more upfront design than a schemaless store |
| Document / KV store | Flexible schema is convenient for rapidly evolving rule shapes; often pairs naturally with the snapshot's own JSON-like structure | Enforcing tenant-scoped uniqueness and referential integrity (a rule pointing at a segment that must exist) becomes application-level discipline instead of a database guarantee — exactly the kind of invariant most likely to silently drift |
Decision 17
Per-Environment Credentials vs. Broad Project-Wide Tokens
Whether an API credential is minted per environment or issued once per project and trusted everywhere inside it.
| Option | Pros | Cons |
|---|---|---|
| Project-wide credential | Fewer credentials to manage and rotate; simpler onboarding | A single leaked credential exposes every environment at once, including production — a staging-only integration test leaking its token becomes a production incident |
| Per-environment, per-scope credential (chosen) | A leaked staging key can't touch production; a read-only SDK key can't publish even if compromised | More credentials to issue, track, and rotate — needs real tooling (not a spreadsheet) to stay manageable at scale |
Decision 18
Sticky Assignment Persistence vs. Deterministic Recomputation
Once a user is bucketed, is that assignment stored, or recomputed from the same hash every time?
| Option | Pros | Cons |
|---|---|---|
| Store the assignment (sticky, persisted) | Assignment survives any future salt or algorithm change by construction, since it's just a stored row | Requires a datastore lookup (or cache) at evaluation time — reintroduces exactly the kind of per-request dependency local evaluation is designed to avoid, at scale |
| Recompute deterministically every time (chosen) | Zero storage, zero lookup — the hash function itself *is* the persistence mechanism, consistent with fully local evaluation | A salt change (Decision 05) or hash-algorithm change silently reshuffles every assignment at once, since nothing was ever actually stored |
Decision 19
Single Snapshot Format vs. Dual-SDK Compatibility Window
When the snapshot schema itself needs to evolve (a new rule type, a new field), how old and new SDK versions coexist during the rollout of that change.
| Option | Pros | Cons |
|---|---|---|
| Single format, synchronized SDK upgrade | Simpler to reason about — one schema version live at any time | Requires coordinating every SDK across every team and language to upgrade before the new schema can ship, which is rarely realistic at a 20,000-instance fleet scale |
| Dual-compatibility window (chosen) | New snapshot fields are additive and ignored by old SDKs that don't understand them; the snapshot builder can also down-level a snapshot for environments still running old SDKs, if needed | The control plane must track which SDK versions exist in the fleet and avoid publishing features that would silently break an old SDK — real operational overhead, not free |
Decision 20
CDN Caching vs. Rapid Revocation Freshness
Aggressive CDN caching minimizes origin load and latency; an emergency kill switch needs the opposite — near-immediate freshness.
| Option | Pros | Cons |
|---|---|---|
| Long CDN TTL on snapshot responses | Minimal origin load at huge fleet scale; cheap and fast for the overwhelmingly common case where nothing urgent is happening | A naive long TTL keyed by a non-versioned path can serve a stale snapshot well past an emergency revocation's intended window |
| Versioned URLs + short-TTL version-check endpoint (chosen) | The snapshot payload itself can cache aggressively and safely, because its URL is immutable per version; only the small 'what's the latest version' check needs to stay fresh | Two different cache policies to operate and reason about instead of one uniform rule |