Feature Flag System Architecture: Evaluation, Bucketing & Distribution

A flag key, a type (boolean, string, number, JSON), a default value, an ordered list of targeting rules, and a status (active/archived). Immutable identity, mutable configuration.

An ordered entry: a match condition (segment membership, explicit user/tenant list, attribute predicate), a percentage rollout definition (variations + weights + salt), and the value or variation to serve when it matches.

A named, reusable set of match conditions — 'beta testers,' 'enterprise plan,' 'EU region' — referenced by many flags' targeting rules instead of duplicating the same predicate everywhere.

A flag's identity (key, type, the set of possible variations) is shared across every environment — the same flag shouldn't silently change type between staging and production. Its configuration (is it on, what rules apply, what's the default) is environment-specific and expected to differ: a flag can be 100% on in staging and 1% in production simultaneously. Collapsing these into one row per flag would force every environment to share configuration, which defeats the purpose of having separate environments at all.

An SDK never reasons about individual flag_environment_config rows changing — it only ever activates a complete published_config identified by a single monotonic config_version. This is what makes monotonic activation (Chapter 08) checkable with one integer comparison instead of reconciling a set of independent row-level timestamps.

Every administrative request's tenant scope comes from the verified credential, never a client-supplied tenant ID or project slug in the URL alone. A credential scoped to Tenant Birch requesting Tenant Cedar's configuration gets a 403, server-side, before any row is read.

A snapshot cache key or CDN path built from flag key alone (without tenant/project/environment) will silently serve one tenant's configuration to another's SDK the moment two tenants happen to share a flag key — a subtle, easy-to-miss isolation bug that has nothing to do with the authorization layer being correct.

Every mutation — who changed what, when, with what diff — is itself namespace-scoped, so a tenant's compliance review of their own audit history never surfaces another tenant's activity.

A flag carries two distinct resting values, not one shared 'default': offVariation is what every targeted user gets while the flag is deliberately turned off, and fallthroughVariation is what an enabled flag returns when a user matches no targeting rule. Collapsing these into a single defaultValue field is a common modeling mistake — it means an operator can't turn a flag off to a safe value (say, the old code path) while separately choosing a different fallthrough for enabled-but-unmatched traffic (say, the new code path for everyone not yet in a rollout segment). Two fields, two independent operator decisions.

Every evaluation returns not just a value but a machine-readable reason string — which rule matched, or that none did. This is what lets a support engineer explain any outcome without guessing or re-deriving the logic by hand.

A flag can depend on another flag's value (a prerequisite). Before evaluating one, the engine checks for a cycle in the dependency graph and fails safe to the default rather than recursing forever.

The tuple hashed — tenant, project, environment, flag key, bucketing unit, subject ID, salt — must serialize to identical bytes regardless of which SDK language computes it. A Python SDK and a Java SDK that format the same logical tuple differently will silently disagree on bucket membership for the same user.

The salt is what makes two different flags' rollouts for the same user statistically independent. Changing a flag's salt deliberately reshuffles its bucket membership — useful when a rollout needs a genuinely fresh random sample, but it breaks the monotonic-expansion guarantee for that flag, so it should be a rare, explicit, audited action, not a side effect of an unrelated edit.

This is deterministic bucketing into a fixed-size space for percentage targeting, a different problem from consistent hashing's goal of minimizing key remapping when nodes join or leave a ring. Don't conflate the two in an interview — they solve different problems with superficially similar-sounding mechanics.

A production-environment change requires a second reviewer before it can publish. Cheap insurance against both accidental and malicious changes, and the natural place to log who approved what and why.

The durable config write and the publish-event write happen in the same database transaction (see Outbox Pattern for the general mechanism), so a crash between 'save the edit' and 'notify downstream' can never silently drop a change the admin believes succeeded.

A snapshot builder consumes the outbox event and produces a new version number — never mutates a previous one. Version 12 will always mean exactly what it meant the moment it was built; a further edit produces version 13.

The snapshot builder rejects a config with duplicate keys, invalid rollout weights, cyclic prerequisites, or unsupported rule operators before it is ever published — an SDK should never need to defend itself against a structurally broken snapshot.

A credential that can PATCH a flag is not automatically allowed to POST /publish — separating 'propose a change' from 'make it live' is what makes the approval gate (Chapter 06) structurally enforceable rather than a convention engineers are trusted to follow.

Not every caller can embed an SDK — a lightweight webhook handler or a language without a maintained SDK still needs a flag decision. Remote evaluation trades the data plane's latency and availability guarantees (Chapter 09) for convenience; it's explicitly the exception path, not the primary evaluation model this design optimizes for.

An ETag or version tag lets most polls return a cheap 304 Not Modified. Freshness is bounded by the poll interval for a healthy, connected instance — acceptable for most flags, not sufficient alone for an emergency kill switch.

A push/pub-sub channel meaningfully improves typical freshness, but delivery isn't guaranteed — treat a push as 'go check for a new version,' with polling retained as the safety net that catches a dropped notification.

The SDK compares incoming version numbers against the one it's running and refuses to activate anything older — enforced by the SDK itself, because the delivery mechanism underneath it (retries, multiple relay paths, redelivery) cannot itself guarantee ordering.

An incremental delta is only valid relative to a known base version. If an SDK's current version doesn't match the delta's assumed base — because it missed an earlier update — it rejects the delta and falls back to fetching the full snapshot instead of risking silent corruption.

Authentication, RBAC, validation, the approval workflow, publish orchestration, and audit. Can tolerate seconds of latency per operation; correctness and auditability matter far more than raw throughput here.

Distribution (CDN/relay), the SDK's local snapshot cache, and the in-process evaluator. Must tolerate huge evaluation volume with sub-millisecond latency and must keep functioning when the control plane is unreachable.

The data plane depends on the control plane only through versioned, asynchronously distributed snapshots — never a live synchronous call from an SDK back into the control plane on the evaluation hot path.

Runs inside a trusted environment. Holds the full targeting rule set for its namespace in memory and evaluates locally for every incoming request or job.

Runs in an untrusted environment. Receives a bootstrap payload containing only this specific session's already-evaluated flag values — never the underlying segment definitions, rollout percentages, or other tenants' rules, which would otherwise be visible to anyone opening developer tools.

Each evaluation can emit an exposure event (flag, variation, reason, anonymized/derived bucketing key) to an async telemetry pipeline — fire-and-forget from the evaluation call's perspective, batched and sent on a background interval.

Telemetry and targeting context both carry the minimum attributes needed, favoring derived or hashed identifiers over raw PII wherever a derived attribute suffices for correct bucketing or segment matching.

An evaluation happening is not the same claim as a user genuinely being exposed to a variation's effect — a flag evaluated during a health check, a retry, or a background job inflates evaluation counts without representing real user exposure. Exposure metrics used for experiment analysis need to filter to evaluations that actually reached user-visible behavior, or they systematically misstate sample size.

If exposure telemetry is sampled or dropped under backpressure (Chapter 11), that sampling must be uniform across variations — a bug that disproportionately drops one variation's events will make a rollout look like it's underperforming when the difference is actually in the data pipeline, not the product.