System Design · 8 chapters · ~30 min read
Multi-Tenant Feature Flag System Interview Guide
How to scope, whiteboard, and defend this design in forty-five minutes — the opening framing, the build-up order, twelve questions an interviewer is likely to push on, how this contrasts with remote config/A-B testing/permissions/deployment controls, an evaluation rubric, a closing checklist, and the mental model that ties it all together.
Chapter 01
Scope This in Five Minutes
The strongest opening move is separating 'what gets stored and governed' from 'what gets evaluated and how fast' before drawing anything.
Open with: 'A feature flag system has two very different halves — a governed, versioned configuration pipeline, and a local, deterministic evaluation engine. Before I draw boxes, let me scope which half this conversation cares about most.'
- Ask: boolean-only flags, or multivariate (string/number/JSON)?
- Ask: backend evaluation only, or does a browser SDK need to evaluate too?
- Ask: is an emergency kill switch a hard requirement, and if so, what propagation latency is acceptable?
- Ask: illustrative scale — how many tenants, how many service instances, how many evaluations per second?
- State explicitly what's out of scope (full admin-console UX, analytics dashboards) so the interviewer can redirect you
- Jumping straight to 'I'll put flags in Redis' before establishing whether evaluation needs to survive a Redis outage at all
- Treating this as 'just a config table with a cache in front of it' — that framing misses tenant isolation, deterministic rollouts, and versioned safe distribution entirely
- Promising an instantaneous global kill switch before establishing whether that's even physically achievable given the chosen distribution mechanism
Chapter 02
Whiteboard Build-Up Order
Draw in this order so each box's reason for existing is already on the board before you draw the next one.
Swipe to pan
| Step | What to draw | What to say |
|---|---|---|
| 1 | Client + SDK/Evaluator | "Evaluation happens here, in-process, with zero network calls on the hot path." |
| 2 | Control Plane | "This is where humans edit flags — auth, RBAC, and an approval gate before anything reaches production." |
| 3 | Config DB + Outbox | "A commit and a publish-event land in the same transaction, so a crash can't silently drop a change." |
| 4 | Snapshot Builder | "Turns the change into an immutable, versioned, validated artifact — never mutates a previous version." |
| 5 | Distribution | "Poll-plus-push delivers it to every SDK; the SDK enforces monotonic activation so it never goes backward." |
Chapter 03
Twelve Questions an Interviewer May Push On
Each of these targets a specific invariant from Architecture Deep Dive. Name the invariant, then the mechanism.
Q
How do you make a 10% rollout actually stay at 10% for the same user across requests?
Strong answer angle
Name the failure first — per-request randomness flips the same user between ON and OFF. The fix: hash a stable tuple (tenant, flag, user ID, salt) into a fixed bucket space (e.g., 0–99,999) deterministically, so the same inputs always produce the same bucket.
Likely follow-ups
- What happens when you raise it to 20%? — Monotonic expansion — moving the threshold from 10,000 to 20,000 only adds newly-eligible buckets; anyone already below 10,000 stays included.
- Does this work the same across your Python and Java SDKs? — Only if the canonical input serialization is identical byte-for-byte across languages — a subtle, easy-to-miss correctness requirement.
Q
Your central flags database goes down. What happens to evaluation?
Strong answer angle
Nothing, for already-warmed SDK instances — evaluation reads only from the local in-memory snapshot, never a live database call. The outage blocks new publishes and administration, not evaluation.
Likely follow-ups
- What about a brand-new instance starting during the outage? — It has no snapshot to fall back on — this needs an explicit, documented per-flag cold-start default policy, not an accidental default from an empty lookup.
Q
How do you prevent Tenant A from ever seeing Tenant B's flag configuration?
Strong answer angle
Isolation enforced at every layer config passes through — authorization checks against a verified credential's real tenant scope, DB queries scoped to the full tenant/project/environment tuple, and critically, cache and CDN keys that include the full namespace, not just the flag key.
Likely follow-ups
- Which layer is easy to miss? — Cache/CDN keys — if two tenants happen to create a flag with the same name and the cache key is just the flag key, one tenant's config can silently leak to the other's SDK.
Q
Can you guarantee an emergency kill switch takes effect everywhere instantly?
Strong answer angle
No, and say so directly — distribution is asynchronous, so even a kill switch has a measured, bounded propagation window per region. The honest answer documents and monitors that window rather than promising something the architecture can't deliver.
Likely follow-ups
- How would you shrink that window? — A push-based notification channel reduces typical latency; polling remains the safety net. Neither makes the window exactly zero.
Q
An SDK receives configuration version 12 after it's already running version 13. What should it do?
Strong answer angle
Reject it — enforce monotonic activation: an SDK never activates a version older than the one it's already running, because the delivery path (retries, multiple relay routes) can't guarantee ordering on its own.
Q
Why not just send incremental deltas for every config update, to save bandwidth?
Strong answer angle
Deltas are fine as long as the SDK validates the delta's assumed base version against its own current version first. If an SDK missed an earlier update and the base doesn't match, it must fall back to a full snapshot fetch instead of applying the delta blindly and risking corruption.
Q
How would you estimate the load on your config-distribution service at scale?
Strong answer angle
Separate evaluation volume from distribution volume explicitly — 20,000 instances evaluating 500 flags/sec each is 10 million evaluations/sec, entirely local and invisible to the distribution service. The same fleet polling every 10 seconds is only about 2,000 requests/sec against distribution — more than three orders of magnitude smaller.
Q
Is a feature flag an acceptable way to gate access to an admin-only operation?
Strong answer angle
No — a flag decides what a user sees, not what they're authorized to do. A protected operation needs an independent, server-side permission check regardless of flag state; using a flag as the only gate is a common, security-relevant mistake.
Q
What does your browser SDK actually receive — the full rule set, or something smaller?
Strong answer angle
Only its own pre-evaluated result for the current context, never the underlying targeting rules, segment definitions, or rollout percentages. Shipping full rules to an untrusted browser leaks business logic to anyone opening dev tools.
Q
A rollout causes an error-rate spike. Walk me through recovery.
Strong answer angle
Publish a new version that restores the last known-good configuration — never delete or rewrite the problematic version, so the audit trail stays honest. Then monitor fleet-wide version adoption to confirm the rollback has actually reached production traffic before declaring the incident resolved.
Q
How does your telemetry pipeline affect evaluation latency if it's overloaded?
Strong answer angle
It shouldn't, at all — exposure events are emitted asynchronously and non-blockingly; evaluation never waits on, or fails because of, telemetry delivery. A telemetry outage degrades analytics, not application behavior.
Q
Why bucket by tenant ID for some flags and by user ID for others?
Strong answer angle
The bucketing unit changes what the rollout percentage actually means — user ID spreads a sample across individual end users regardless of tenant, while tenant ID keeps every user inside one tenant consistent, which matters for a collaborative or B2B feature where a split experience inside one company would be confusing.
Chapter 04
Common Mistakes Under Pressure
These are the specific slips that cost points even from candidates who clearly understand the domain.
- Describing evaluation as a remote API call instead of a local, in-memory function — this single framing error undermines the entire design's scalability story
- Promising an instantaneous global kill switch without naming the propagation mechanism that would have to make that true
- Treating a feature flag as sufficient authorization for a sensitive operation
- Conflating deterministic percentage bucketing with consistent hashing — they solve different problems
- Describing rollback as 'deleting the bad version' instead of 'publishing a new version that restores the old content'
- Forgetting that cache/CDN keys need the full tenant/project/environment tuple, not just the flag key
Chapter 05
Feature Flags vs. Everything That Sounds Similar
A useful follow-up is whether you can draw the boundary between a feature flag and four adjacent concepts that get casually conflated in everyday engineering conversation.
| Concept | What it actually controls | How it differs from a feature flag |
|---|---|---|
| Feature flag | Whether a code path is active, and for whom, evaluated locally and deterministically | The baseline concept this module is about — governed centrally, evaluated at the edge. |
| Remote config | Tunable runtime parameters (a timeout value, a page-size limit, a UI color) with no targeting logic | Usually no rules, no bucketing, no variations — just a centrally-managed value. A feature flag is remote config plus targeting, rollout, and an explicit on/off semantic. |
| A/B testing | A statistically-designed experiment comparing variants to measure an effect on a metric | A feature flag can *carry* an A/B test's variant assignment, but the flag system itself doesn't do experiment design, statistical power analysis, or significance testing — that's a separate analytics layer consuming the flag's exposure events. |
| Permissions / entitlements | What a specific, authenticated user or role is allowed to do | A permission is an authorization decision with compliance and security consequences; a flag is a release-management decision with none. Using a flag as a permission check is a named anti-pattern in this module's invariant I7. |
| Deployment controls (canary, blue/green) | Which running *version of the binary* receives traffic | Deployment controls operate at the infrastructure/routing layer and are coarse-grained (a whole instance or pod). A flag operates inside one already-deployed binary and can target a single user — deploying and releasing are deliberately decoupled, not the same lever. |
Chapter 06
Evaluation Rubric & the Closing Checklist
What actually separates a pass from a strong pass, by level — and a printable sequence to walk through out loud if the conversation stalls.
| Level | Correctness bar | Operational bar |
|---|---|---|
| Mid-level | Explains local evaluation vs. central config correctly; can describe deterministic bucketing when prompted | Names one or two failure modes when asked directly |
| Senior | Volunteers the control/data-plane split unprompted; explains monotonic activation and why delivery order can't be trusted | Proactively raises staleness windows, isolation enforcement points, and the flag-is-not-authorization distinction without being asked |
| Staff / Principal | Reasons about cross-language hash normalization, snapshot size limits, and the operator/process failure class (not just distributed-systems failures) as first-class design concerns | Discusses how the design changes for regulated tenants, offline mobile SDKs, and browser-only apps; names the specific metric (Architecture Deep Dive, Chapter 12) that would catch each failure mode in production before a customer reports it |
- Requirements — boolean vs. multivariate, backend-only vs. browser, kill-switch latency, illustrative scale
- Key invariant — evaluation is local and deterministic; configuration is centrally governed
- First solution — a conditional in code, and precisely why it stops working at the second tenant
- Local evaluation — SDK snapshot cache, atomic swap, and the cold-start/warm-outage distinction
- Targeting — deterministic bucketing, rule precedence, segments, prerequisites and cycle safety
- Distribution — poll plus push, monotonic version activation, delta-vs-full fallback
- Security — tenant isolation at every layer, RBAC, browser trust boundary, flags are not authorization
- Failures — at least two concrete incidents narrated proactively, with detection and recovery
- Trade-offs — name the alternative, its cost, and the condition that would flip the decision
Chapter 07
Defending Your Trade-offs Out Loud
An interviewer pushing on a trade-off is usually checking whether you understand the cost of your own decision, not looking for a different answer.
If pushed on 'why not just make evaluation synchronous and always current,' the honest answer names the actual cost: a synchronous per-request dependency on a remote service makes that service's availability a hard ceiling on the whole application's availability — which is the exact failure mode the naive central-database version demonstrated.
If pushed on 'why accept any staleness window at all,' the honest answer is that local evaluation and perfect real-time consistency are in direct tension — you can have one or the other on the hot path, and for a system whose primary job is enabling safe releases without downtime, availability and speed win, with staleness bounded and monitored rather than eliminated.
Chapter 08
The Mental Model to Leave With
Two sentences carry this entire design.
“Deployment and release are different decisions, on different schedules. Configuration is centrally governed; evaluation is local.”
— The closing principle from this module's Visual Design Journey
Key takeaways
- 01Evaluation must be local, in-memory, and deterministic — never a per-request network dependency
- 02Configuration is centrally governed, versioned, validated, and distributed asynchronously
- 03Tenant isolation has to hold at every layer config passes through, not just the first auth check
- 04A percentage rollout needs determinism, uniformity, and monotonic expansion — not raw randomness
- 05An SDK must enforce monotonic version activation itself, because delivery order can't be trusted
- 06A warmed SDK survives a control-plane outage; a cold one needs an explicit default policy
- 07A feature flag decides what a user sees — never whether they're authorized to do something