Stock Exchange System Design — Failure, Recovery & Scaling Scenarios

These four incidents share one theme: a core component Flow Exchange depends on for correctness — the matching engine process, the pre-trade risk service, the journal disk, or a snapshot file — becomes unavailable or untrustworthy. In every case, the fix is the same shape: stop, verify the last durable position, and resume only from a position you can prove is correct.

Scenario 01 — Matching engine process crashes

Trigger: the single-writer matching engine process for a partition (say, the engine owning ACME) dies — an unhandled exception, an out-of-memory kill, a host reboot, or an operator-initiated restart. Immediate technical effect: the in-memory order book for that partition disappears instantly; nothing else in the process's address space survives. The durable command journal still holds every accepted command up to the last committed sequence number. Impact on orders/trades: anything journaled before the crash is safe and will be replayed; anything in flight — received by the gateway but not yet sequenced and committed — is a lost client response (the client sees no acknowledgment), not necessarily a lost order, since the gateway may still hold it for retry. No already-reported trade is ever reversed.

How to detect it: process supervision (systemd, a Kubernetes liveness probe, or a watchdog) detects the exit within its health-check interval; downstream consumers notice the partition's committed-sequence counter stall while inbound traffic keeps queuing at the sequencer. Containment: the partition is marked halted for new commands the instant the crash is detected — Flow Exchange never keeps accepting orders against a book that no longer exists in memory. Other partitions are unaffected, since each is single-writer and independently owned; inbound commands for the dead partition queue at the gateway up to a bounded backlog limit, then reject with a retryable error rather than growing unbounded.

Recovery sequence: 1) load the most recent verified snapshot for the partition; 2) replay every journaled command from that sequence number onward, deterministically rebuilding the book and any implied executions; 3) confirm the rebuilt committed sequence matches the journal's last fsynced offset; 4) re-fence the partition under a new epoch so no stale writer can resume matching; 5) resume accepting new commands only once replay completes and the book passes invariant checks. How to prevent recurrence: bound memory allocations per order/book to reduce OOM risk, run engines under fast-restart supervision, and keep snapshot intervals short enough that replay time stays inside the recovery-time target (see Scale Walkthroughs for the snapshot-interval trade-off).

Scenario 02–03 — Crash after journal commit but before acknowledgment, and duplicate order due to network retry

These two incidents are one coin with two faces, handled by the same mechanism. Trigger: either the engine commits a command to the journal and crashes (or the ack is dropped on the network) before the client sees a response, or a client — seeing no response — retries the same order under its own timeout logic. Immediate technical effect: in the first case the command *was* durably committed; only the acknowledgment is missing. In the second, a second copy of the same client order arrives at the gateway. Impact on orders/trades: without deduplication, the retry would be treated as a brand-new order, producing a second resting order or a second execution for what the client believes is one order — a genuine duplicate execution, the one outcome that must never happen.

How to detect it: every inbound command carries a client order ID scoped to the member's session; the gateway/engine keeps a dedup table (or checks the journal directly) mapping client order ID to exchange order ID and last known state, and alerts on an abnormal rate of client-order-ID collisions per session. Containment: no halt — this is resolved inline per command, and unrelated orders are never blocked while a duplicate is resolved.

Recovery sequence: on receiving a client order ID that already exists for that session, the gateway does not re-submit it to the sequencer; it looks up the durable outcome of the original command and returns that same outcome — accepted, rejected, partially filled, or filled — as an idempotent response. For the crash-after-commit case, this lookup-and-replay-the-response path is exactly how the client eventually learns its true outcome, with no second matching attempt ever occurring. How to prevent recurrence: require client order IDs to be unique per session for its lifetime, reject reused IDs that don't match a known prior command as a protocol violation, and keep the dedup lookback window at least as long as the client's maximum retry/timeout window.

Flow Exchange's core promise is that every command for a given book is serialized by the sequencer into one deterministic order. The hardest incidents to reason about are the ones where a human expects a different, more 'intuitive' resolution than whatever the sequencer actually decided — most visibly, a cancel racing a fill.

Scenario 04 — Cancel races with fill

Trigger: a member sends a cancel for a resting order at nearly the same wall-clock instant that an incoming marketable order would fill it. Immediate technical effect: both commands arrive at the sequencer, which assigns each a strict position in the per-book command stream; only one resolution exists once sequencing happens, even though the two events looked simultaneous to the member. Impact on orders/trades: whichever command the sequencer processes first wins. Using the canonical book (S1 $101.00×50 seq10, S2 $101.00×30 seq11, S3 $101.50×70 seq12, B1 $100.50×40 seq13, B2 $100.00×80 seq14): Test D sends cancel(S1) racing an incoming BUY 60 @ $101.00. If cancel(S1) is sequenced first, S1 is gone and the buy can only reach S2 — it fills 30 against S2 at $101.00, fully exhausting that level; the remaining 30 shares of the incoming order rest at $101.00 as a new resting bid, and S3 at $101.50 is never touched because a $101.00 buy limit can never execute above $101.00. If the buy is sequenced first, S1 fills 50, S2 fills 10 (remaining 20), and the cancel then finds S1 already filled, so it is rejected as 'nothing left to cancel' rather than retroactively undoing the fill.

How to detect it: this is expected, correct behavior, not an anomaly — what's worth monitoring is the rate of 'cancel rejected — order already filled/not found' responses per member, since an unusually high rate can mean a member's algorithm is racing its own cancels too aggressively, or gateway latency is widening the race window. Containment: no halt, no special handling; the one architectural discipline is that a cancel acknowledgment must always reflect the order's actual serialized outcome, never a value computed from when the cancel was *received* at the gateway.

Recovery sequence: nothing to recover — the sequencer's decision is final and durable the moment it's journaled; if a member disputes the outcome, the journal provides an exact, replayable timeline proving which command was sequenced first. How to prevent recurrence: you cannot and should not eliminate this race, since it's inherent to any concurrent market. Minimize gateway-to-sequencer latency variance so members get consistent, low-jitter round trips, and document clearly that cancels are never guaranteed to preempt in-flight matches.

Scenario 05 — Cancel/replace races with other commands

Trigger: a member sends a replace (price/size amendment) at nearly the same time as another cancel, replace, or incoming match targeting the same order. Immediate technical effect: every order carries a version number that increments on each accepted mutation; the engine validates that a replace/cancel targets the version it expects. Impact on orders/trades: a replace arriving after the order was already filled or cancelled by an earlier-sequenced command is rejected as stale, never silently applied to a version that no longer exists; a price-changing replace is treated as cancel-then-new for priority purposes (losing time priority), while a size-decrease-only replace can, per Flow Exchange's policy, keep its original time priority.

How to detect it: track version-mismatch rejection rate per member and symbol — a spike usually means a member's order-management system is issuing replaces without waiting for the prior acknowledgment, racing itself. Containment: no halt; stale replace/cancel attempts are rejected individually and unrelated orders are unaffected. Recovery sequence: none beyond returning the correct rejection and current order state so the member's local view can resynchronize. How to prevent recurrence: encourage members to serialize their own order mutations per order ID client-side, and expose order version numbers in every execution report so member systems can detect and resolve staleness themselves.

Not every failure is inside the matching engine. The path to the engine — the network, the risk gate, the disk the journal lives on, and the integrity of a snapshot file — can each fail independently, and each demands a different containment decision.

Scenario 06 — Gateway-to-engine partition

Trigger: the network link between a session gateway and the matching engine (or sequencer) degrades or drops entirely while the gateway stays reachable to members. Immediate technical effect: the gateway can still accept sessions and receive order messages, but cannot forward them onward for sequencing. Impact on orders/trades: if the gateway accepted and acknowledged orders anyway, it would be lying about durability — those commands would never reach the journal, and members would believe orders are live when they are not.

How to detect it: the gateway monitors round-trip health to its sequencer/engine with a short heartbeat; missed heartbeats beyond a threshold (2–3 consecutive misses) flag the partition unreachable, and queue depth on the gateway's outbound link grows monotonically once the link degrades. Containment: the gateway stops accepting new commands for the affected partition immediately and rejects them with an explicit 'partition unreachable, retry later' response rather than queuing indefinitely — an unbounded backlog just delays discovery and risks a flood of stale orders landing at once if the link returns. Other partitions continue normally.

Recovery sequence: once the heartbeat recovers, the gateway resumes forwarding new commands; no replay is needed since it never accepted anything it couldn't deliver, and rejected commands are the member's responsibility to resubmit. How to prevent recurrence: run redundant network paths between gateway and engine tiers where the topology allows it, and keep the bounded-backlog threshold small so a rejected order is a fast, clear signal rather than a silent multi-second hang.

Scenario 07 — Risk service outage

Trigger: the pre-trade risk engine — validating member limits, notional caps, and price collars before a command reaches the sequencer — becomes unavailable, or its risk state goes stale beyond an acceptable freshness window. Immediate technical effect: new commands can no longer be safely validated against current limits. Impact on orders/trades: resting orders and matching are unaffected — existing liquidity remains matchable — but no *new* order can be safely admitted without a basic validation step.

How to detect it: risk-check latency and error-rate metrics on the gateway's call to the risk service, plus a risk-state staleness metric crossing its configured threshold. Containment: Flow Exchange's chosen policy is fail-safe rejection — while the risk service is unavailable, new orders are rejected with an explicit reason code rather than silently admitted, since an unchecked fat-finger or limit breach is worse than temporarily rejecting legitimate orders. Cancels still proceed, since reducing exposure is always safe.

Recovery sequence: the risk service restarts and reloads current limits from its durable store; once healthy and its state freshness is within tolerance, the gateway resumes routing new orders through it. How to prevent recurrence: run the risk service redundantly with fast failover, and keep risk-limit updates on a durable, replayable store so a restarted instance doesn't start from stale or empty state.

Scenario 08 — Journal disk full / write failure

Trigger: the storage volume backing the durable command/event journal runs out of space, or the write itself fails (I/O error, filesystem remount read-only). Immediate technical effect: the engine cannot durably commit the next command — the acknowledgment contract (ack only after durable commit) means it literally cannot proceed with further matching, because doing so without a durable record would let a crash silently lose a trade. Impact on orders/trades and user-visible behavior: the partition must halt new matching immediately. This is the clearest example in the module of why Flow Exchange never promises zero downtime: matching without a working journal risks an unrecoverable, undetectable trade loss, strictly worse than a visible, bounded halt.

How to detect it: disk-free-space alarms on the journal volume well before zero (e.g., 80% and 95% thresholds), and the engine raises a hard alert the instant a journal write returns an error rather than retrying silently forever. Containment: halt the partition and reject new inbound commands with an explicit 'trading halted — durability unavailable' reason; never accept orders into an in-memory-only queue hoping for a quick fix.

Recovery sequence: free space (or fail over the journal writer to a healthy volume/replica), verify the journal is writable with a test commit, confirm the last successfully committed sequence number, and resume accepting commands from that confirmed point — no replay is needed unless the engine process itself also restarted, in which case follow Scenario 01's sequence. How to prevent recurrence: proactive capacity alarms, automated log compaction/archival once segments are safely snapshotted, and journal storage headroom sized to the assumed peak event rate.

Scenario 09 — Corrupt snapshot or log segment

Trigger: a snapshot file or journal log segment fails a checksum validation during routine health verification or, worse, during a recovery attempt after Scenario 01. Immediate technical effect: the engine cannot trust that snapshot (or segment) to deterministically reconstruct the book. Impact on orders/trades: if found during routine verification, there is no live impact yet — it's a recovery-readiness risk; if found *during* an active recovery, the partition cannot safely resume trading until a trustworthy recovery point is found.

How to detect it: every snapshot and log segment is written with a checksum and schema version, and a scheduled background job periodically re-validates checksums on stored snapshots/segments — not only at recovery time — so corruption is caught early. Containment: if found during routine verification, trading continues uninterrupted while the bad snapshot is quarantined and an alert fires; if found mid-recovery, the partition stays halted rather than resuming from an unverified state.

Recovery sequence: fall back to the next-older snapshot that still passes checksum verification, then replay the full journal from that older point forward — slower, but correctness-preserving; if log segments are also suspect, replay stops at the last segment that verifies cleanly and the gap is surfaced for manual reconciliation, never guessed at. How to prevent recurrence: strong checksums and explicit schema versions on every snapshot/segment, at least two retained snapshot generations, replicated journal storage so one corrupted copy isn't the only copy, and frequent background verification.

A single authoritative writer per partition is the foundation of correctness in this design — these four scenarios are all about what happens when that single-writer guarantee is threatened by a failover, a network split, a region loss, or silent divergence.

Scenario 10 — Primary crash and standby takeover

Trigger: the active (primary) matching engine for a partition fails — the same event as Scenario 01 — but this time a warm standby exists and should take over rather than relying on cold snapshot+replay. Immediate technical effect: the standby has been continuously ingesting the same journal stream and is close to caught up, but it is not yet authoritative. Impact on orders/trades: until explicitly promoted under a new epoch, it must not match any orders, even if it 'looks' ready — otherwise split brain (Scenario 11) becomes possible if the old primary is only partially dead (a long GC pause, not a clean crash).

How to detect it: the standby and an external failure detector monitor the primary's heartbeat/lease; failure is declared once the lease expires without renewal, not on a single missed heartbeat, to avoid promoting on a transient blip. Containment: the partition is halted for new commands from the moment the lease expires until the standby is fully promoted and caught up — there is no window where two engines could both believe they're authoritative.

Recovery sequence: 1) the failure detector confirms the lease has expired; 2) the standby replays any remaining journal entries, reaching the last committed sequence; 3) it is promoted under a strictly higher epoch/fencing token; 4) any write attempt by the old primary under its now-stale epoch is rejected by the journal/storage layer, which enforces monotonically increasing epochs; 5) gateway routing metadata updates to the new primary and trading resumes. How to prevent recurrence: keep standby replication lag low, tune the lease-based failure detector to avoid both false positives and slow detection, and run failover drills regularly rather than discovering gaps during a real incident.

Scenario 11 — Split brain

Trigger: a network partition isolates the primary from the failure-detection/consensus layer without killing the primary process — it stays alive believing it's authoritative, while the consensus layer, unable to reach it, promotes a standby. Immediate technical effect: two processes now believe they are the authoritative writer for the same partition, each potentially willing to accept and commit new commands. Impact on orders/trades: if both were allowed to write, two independently valid-looking but divergent books would form — a catastrophic correctness failure.

How to detect it: the fencing mechanism itself is the detector — every write to the shared journal/storage layer must present its writer's epoch token, and the storage layer rejects any write carrying an epoch lower than the highest it has already seen. Operationally, 'fenced write rejected' errors from a host the routing layer still thinks is primary are the signature of a would-be split brain being successfully prevented. Containment: the old primary's writes are rejected by the fencing layer the instant it tries to commit under its stale epoch, so it cannot persist further commands even though it still believes it's active; gateway sessions pointed at it are severed once routing metadata updates.

Recovery sequence: once the partition heals or the old primary is explicitly terminated, it attempts to rejoin, discovers its epoch is stale, and must demote itself (or be forcibly killed) — it may rejoin later as a standby under the current epoch. Any commands it accepted locally but could not commit are not part of the durable history and must be reported to clients as rejected/unknown, never silently merged into the new primary's book. How to prevent recurrence: enforce fencing tokens at the storage layer, not just the application layer — operator discipline alone is insufficient, since the premise of split brain is that the old primary can't reliably know its own status during a partition.

Scenario 25 — Entire data center unavailable

Trigger: the data center (or region) hosting the active matching infrastructure for one or more partitions loses power, network connectivity, or is otherwise destroyed/unreachable. Immediate technical effect: every primary and local standby for the affected partitions becomes unreachable simultaneously. Impact on orders/trades: trading on those partitions cannot continue until a disaster-recovery (DR) site is promoted — explicitly not a zero-downtime scenario; a controlled halt and later reopen is the correct, honest outcome.

How to detect it: loss of heartbeat/connectivity to an entire data center's worth of hosts simultaneously (versus one or two) is the distinguishing signal, confirmed by external monitoring from outside the affected data center since in-region monitoring is itself unreachable. Containment: affected partitions are halted exchange-wide. Whether a geographically separate DR site exists, its replication lag, and promotion time are assumptions stated up front for this exercise, not guaranteed production facts — Flow Exchange never claims a specific RPO/RTO without naming the assumed replication topology.

Recovery sequence: 1) confirm the primary data center is genuinely unreachable via an independent, out-of-region check; 2) promote the DR site's replicas to primary under a new epoch, following the Scenario 10 fencing discipline; 3) reconcile the DR site's last committed sequence against what was last known from the primary region, identifying any gap; 4) reopen trading only once book state is verified consistent and routing metadata points to the DR site; 5) communicate the reopen time and any affected reconciliation to members explicitly. How to prevent recurrence: replicate journal data to a geographically separate DR site with known, monitored lag, rehearse DR promotion as a drill, and keep the runbook mechanical and low-judgment so it executes quickly under pressure.

Scenario 26 — Silent book corruption / state divergence

Trigger: a bug, a hardware memory error, or an undetected replay inconsistency causes the in-memory book on one engine to diverge from what the durable command history actually implies — with no crash, no error, just quietly wrong state. Immediate technical effect: the book keeps matching orders, but against incorrect internal state — for example, a price level that should have been removed still has resting quantity, or remaining quantity doesn't match the committed fills. Impact on orders/trades: this is the most dangerous class of incident precisely because nothing visibly fails; a subsequent match could violate price-time priority, produce a negative remaining quantity, or execute more quantity than the original order ever had.

How to detect it: continuous invariant checks against the live book — no negative remaining quantity, total executed quantity never exceeds original quantity, buy side strictly descending / sell side strictly ascending by price — plus, where a shadow replica exists, periodic deterministic comparison against independently replayed state. Any violation is a hard alert, never a warning. Containment: the partition is halted immediately on invariant violation; continuing to match against known-wrong state is strictly worse than a visible halt.

Recovery sequence: discard the live (suspect) in-memory state entirely, reload the last verified snapshot, and deterministically replay the journal from there — exactly as in Scenario 01 — since this reconstructs the book purely from the durable command history, unaffected by whatever caused the divergence. Resume trading only once the rebuilt state passes all invariant checks. How to prevent recurrence: run invariant checks continuously, not just at startup; run a shadow replica that independently replays the journal and periodically diffs its state against the live engine as an early-warning cross-check; treat any memory-safety or concurrency bug in the matching core as top priority.

Market data and execution-report delivery sit deliberately off the matching critical path — their failures never corrupt the book, but they can mislead clients and subscribers if not handled with the same discipline as matching itself. The rule governing this whole chapter: a market-data feed is never the authoritative source of execution truth.

Scenario 12 — Lost execution report

Trigger: the outbound execution-report path (engine → gateway → member) drops a message — a network blip, a gateway restart, a mistimed session disconnect. Immediate technical effect: the trade or order-state change was already durably committed in the journal; only its *delivery* to the member session failed. Impact on orders/trades: a lost client response, not a lost order or lost execution — the trade is real and durable; the member just doesn't know yet.

How to detect it: execution reports carry a per-session sequence number, and a member's session-level gap detector notices a skipped number in the stream. Containment: no halt — matching is unaffected, and the member's session simply needs a resend mechanism, which any reliable session protocol already requires. Recovery sequence: the member requests a resend/replay of the missing range (or the gateway proactively resends on reconnect); the gateway re-derives the execution report from the durable journal/trade-capture record and redelivers it. How to prevent recurrence: make execution-report delivery itself replayable from durable storage — never generated only in-memory at delivery time — and keep resend windows generous relative to typical reconnect latency.

Scenario 13 — Duplicate trade publication

Trigger: a retry or resend mechanism causes the same trade event to be published twice to a member or the public feed. Immediate technical effect: the same trade ID appears twice in the outbound stream. Impact on orders/trades: the underlying trade happened exactly once and is recorded exactly once in the durable trade-capture store — a duplicate message, not a duplicate execution, provided downstream consumers handle it correctly.

How to detect it: downstream consumers (billing, position-keeping, analytics) track trade IDs already processed and flag a repeat at ingestion. Containment: no halt — the responsibility is on every downstream consumer to be idempotent against trade ID, exactly as with command deduplication in Scenario 02. Recovery sequence: consumers discard (or no-op on) the duplicate trade ID rather than double-counting it in positions, billing, or published depth. How to prevent recurrence: every trade carries a globally unique trade ID generated once at execution time, never regenerated on redelivery; publishers resend the exact same payload (same trade ID) on retry.

Scenario 14 — Market-data packet loss

Trigger: a subscriber on a UDP multicast (or similarly best-effort) market-data feed misses one or more packets — real network loss, not a slow consumer (that's Scenario 23). Immediate technical effect: the subscriber's local sequence counter jumps — it receives N and then N+2, with N+1 missing. Impact on orders/trades: the subscriber's local book view (snapshot + incremental deltas) is now potentially stale or wrong for whatever changed in the missing delta — but this never touches the actual exchange-side book, which has no dependency on market-data delivery.

How to detect it: per-feed sequence numbers make gap detection mechanical — the subscriber checks that each received sequence is exactly one greater than the last. Containment: the subscriber must stop trusting its incremental view the moment a gap is detected, rather than compounding the error by applying further deltas on top of known-incomplete state — a client-side decision, not an exchange-side halt. Recovery sequence: the subscriber requests (or waits for) a full snapshot, applies it as a clean base state, and resumes applying deltas from that sequence forward; some retransmission services can replay just the missing range instead. How to prevent recurrence: publish periodic full snapshots at a bounded interval so worst-case resync time is capped, and offer a retransmission/replay channel for small gaps.

Scenario 15 — Market-data feed delayed or out of order

Trigger: network jitter, a congested fan-out tier, or a slow link causes market-data packets to arrive late or out of publish order, without being dropped outright. Immediate technical effect: a subscriber may briefly see book or trade-print information reflecting an older moment than what has already happened on the actual exchange book. Impact on orders/trades: none on matching/execution — the exchange's own order book and trade history are governed by the sequencer's committed order, never by when a market-data packet happens to arrive downstream.

How to detect it: per-feed sequence numbers double as an ordering check (receiving N+1 before N means buffer-and-reorder), and feed-latency monitoring (publish-to-receive timestamps) flags abnormal delay. Containment: no exchange-side action — purely a feed-health condition; nothing in the exchange's own order-acceptance or risk path ever reads from a market-data feed as ground truth. Recovery sequence: the subscriber reorders buffered packets by sequence number, or waits briefly for in-order delivery; past a delay threshold, resync from a fresh snapshot is the safer fallback. How to prevent recurrence: provision fan-out capacity with headroom above assumed peak subscriber load, and keep the market-data tier isolated from the matching-engine critical path so congestion on one never backpressures the other.

Scenario 23 — Market-data subscriber too slow

Trigger: a particular subscriber's consumption rate falls behind the publisher's production rate — an underpowered client, a slow link, or a processing bottleneck — distinct from Scenario 14's packet loss because the publisher itself may need to act. Immediate technical effect: if the publisher buffers per-subscriber state for a slow reader, that buffer grows unboundedly unless bounded, risking memory pressure that could affect *other*, healthy subscribers on the same fan-out tier. Impact on orders/trades: none on matching; the risk is entirely to fan-out isolation and the publisher's own stability.

How to detect it: per-subscriber buffer/backlog depth on the publisher side; a subscriber whose outstanding backlog crosses a configured threshold is flagged 'slow.' Containment: the publisher disconnects the slow subscriber once its backlog exceeds the bound, rather than letting one slow reader degrade delivery to everyone else — fan-out isolation means one subscriber's problem must never become every subscriber's problem. Recovery sequence: the disconnected subscriber reconnects, requests a fresh full snapshot (its incremental state is now stale by definition), and resumes consuming from there. How to prevent recurrence: tiered market-data distribution (a low-latency direct tier and a slower public/aggregated tier) so subscribers with different capabilities aren't forced onto the same path, plus bounded per-subscriber buffers as a hard architectural rule.

These incidents are about volume and timing rather than a single component breaking outright: a hot symbol saturating its single-writer engine, an exchange-wide volatility burst, a storm of cancels, clock drift confusing an observer (never the matching order itself), a routing mistake, and memory pressure inside the engine process.

Scenario 16 — Hot-symbol overload

Trigger: a single instrument (say ACME after unexpected news) receives order traffic far beyond its single-writer engine's sustainable rate, while other instruments stay at normal load. Immediate technical effect: the queue of commands waiting to be sequenced and applied for that one partition grows; because matching is single-writer per book, no amount of additional hardware on *other* partitions helps this one instrument's queue drain faster. Impact on orders/trades: p99 latency for ACME commands climbs while every other instrument's latency is unaffected, since partitions are independent failure and performance domains.

How to detect it: per-partition queue depth and p99/p99.9 latency broken out by instrument, never averaged exchange-wide (an average hides a single hot symbol behind otherwise-normal aggregate numbers). Containment: admission throttling kicks in for that specific partition once queue depth crosses a threshold — new ACME commands are rejected with a retry/backoff signal rather than queued indefinitely. A full halt is reserved for volatility-driven scenarios (17/19), not pure volume.

Recovery sequence: once order flow subsides below the sustainable rate, the queue drains and throttling relaxes automatically — nothing was lost, only delayed or rejected with a clear signal. How to prevent recurrence: this is fundamentally the single-writer-per-symbol bottleneck covered in Architecture Deep Dive's Partitioning and Hot Instruments — mitigations are efficiency improvements to the single engine (never parallelizing one book, which would break deterministic ordering) plus admission policies that protect overall exchange health rather than promising unlimited volume on one symbol.

Scenario 17 — Exchange-wide volatility / 100× burst

Trigger: a market-wide shock (as an illustrative assumption: a sudden macro event) causes inbound command volume across *most* instruments to spike roughly 100× above baseline simultaneously, not just one hot symbol. Immediate technical effect: gateway, risk, and sequencing tiers all see proportionally higher load, testing admission and fairness policies at their actual limits. Impact on orders/trades: without admission control, queue depths across many partitions grow together, latency degrades broadly, and extreme rapid price moves may breach configured price-band/circuit-breaker thresholds for individual instruments.

How to detect it: exchange-wide aggregate inbound rate crossing a high-water mark, combined with a rising count of instruments individually approaching their price-band/circuit-breaker thresholds — the combination, not either signal alone, distinguishes this from an isolated hot-symbol event. Containment: admission policies apply exchange-wide fairness controls (proportional throttling across members, not pure first-come-first-served, to avoid structurally favoring the lowest-latency participant); individual instruments that breach their price-band policy enter a halt per that instrument's defined volatility-auction policy — deliberate and policy-governed, not an emergent side effect of overload.

Recovery sequence: halted instruments follow their defined reopen procedure (e.g., a volatility auction producing an indicative uncrossing price before resuming continuous trading) rather than reopening into continuous matching instantly; exchange-wide throttling relaxes gradually as volume returns toward baseline, to avoid an immediate second burst. How to prevent recurrence: capacity headroom provisioned above assumed baseline to absorb illustrative burst multiples (see Scale Walkthroughs), and price bands/circuit breakers configured and tested before they're needed, not improvised during the event.

Scenario 18 — Sudden cancel storm

Trigger: a large volume of cancel requests arrives in a short window — a member's algorithm reacting to news, a runaway cancel/replace loop bug, or a legitimate market-wide flight-to-safety. Immediate technical effect: the cancel-to-new-order ratio spikes; the book churns heavily as resting orders are pulled, even though relatively few new orders or trades are happening. Impact on orders/trades: matching still processes each cancel correctly in sequence, but book depth can thin rapidly, and the command volume consumes sequencer/engine capacity that would otherwise go toward new orders and fills.

How to detect it: cancel-to-new-order ratio per member and exchange-wide, tracked as a first-class metric — a sudden spike in this ratio (not just raw volume) distinguishes a cancel storm from a general volume burst. Containment: per-member order-to-trade/cancel-to-order ratio controls can throttle or flag an abnormal member without halting the market for everyone; resource limits (queue bounds, per-partition throttling) protect overall engine health exactly as in Scenario 16.

Recovery sequence: once the burst subsides, queues drain and normal ratios resume; a throttled member is unthrottled once their behavior returns within policy limits. How to prevent recurrence: publish order-to-trade ratio controls as an explicit, documented policy members can anticipate, and treat cancels as full-weight commands in capacity planning, never as 'cheap' operations.

Scenario 19 — Trading halt / volatility interruption

Trigger: an administrative halt (pending news, a regulatory requirement) or an automatic volatility-triggered halt (price band breach) is invoked for a specific instrument. Immediate technical effect: the partition transitions into an explicit halted market state — a deliberate state transition, not an engine failure. Impact on orders/trades: depending on Flow Exchange's stated halt policy, new orders may be rejected outright or accepted and queued for the reopen auction, but no continuous-trading execution occurs during the halt — real venues differ here, so the chosen policy must be stated explicitly.

How to detect it: the halt is an explicit, published market-state change, not something to be 'detected' as an anomaly — gateways, risk, and market data all consume the same halt-state signal from one source so every component agrees at the same sequence point. Containment: continuous matching stops for the halted instrument; incoming orders are either rejected with an explicit 'instrument halted' reason or queued for the reopen auction, never matched during the halt — a book must never show a crossed continuous-trading state, and auctions are a distinct matching mode, not forced into continuous-matching logic.

Recovery sequence: at the scheduled or administratively triggered reopen, queued auction-eligible orders compute an indicative uncrossing price via the instrument's defined reopen/auction procedure; once uncrossed, the book transitions back to continuous trading at the new reference state. How to prevent recurrence: this incident is itself a designed prevention mechanism for a worse outcome — the main operational improvement is ensuring the halt/reopen state machine is deterministic and consistently applied so members can predict behavior.

Scenario 20 — Clock drift / timestamp mismatch

Trigger: the wall-clock on a gateway, risk, or engine host drifts out of sync with the fleet — NTP failure, VM clock skew, or a misconfigured host. Immediate technical effect: timestamps recorded for audit, latency measurement, and reporting on the drifted host become unreliable relative to other hosts. Impact on orders/trades: critically, matching order is unaffected — the sequencer, not wall-clock arrival time, is the ordering authority for a partition, so clock drift on any one host cannot reorder or mis-prioritize commands in the book.

How to detect it: continuous clock-synchronization monitoring (NTP/PTP offset per host) with alerting past a configured tolerance, plus audit tooling that cross-checks reported timestamps against the sequencer's monotonic sequence numbers for impossible orderings. Containment: no trading impact or halt is required purely for clock drift, since matching correctness never depended on wall-clock time; the affected host's timestamps are flagged degraded in audit/monitoring views until corrected.

Recovery sequence: resynchronize the host's clock via NTP/PTP; once within tolerance, remove the degraded-timestamp flag — no book or trade data needs correction, since none of it depended on the drifted clock for ordering. How to prevent recurrence: redundant time sources and continuous drift monitoring across the fleet, and architecturally keep wall-clock timestamps purely descriptive/audit metadata, never load-bearing for determining matching order.

Scenario 21 — Instrument routing misconfiguration

Trigger: the routing metadata mapping an instrument to its owning engine/partition is updated incorrectly — a deploy error, a manual configuration mistake, or a race during an intentional instrument migration. Immediate technical effect: gateways may forward commands for an instrument to the wrong engine, or two engines may briefly both believe they own the same instrument if the metadata update isn't applied atomically everywhere. Impact on orders/trades: if two engines process commands for the same instrument concurrently, this is a correctness emergency identical in kind to split brain (Scenario 11) — two independently matching 'authorities' for one book, which must never happen.

How to detect it: every engine validates, on each command, that the instrument it's processing matches its own currently fenced ownership/epoch — a mismatch is rejected and alerted immediately; disagreeing routing-metadata version numbers across gateways are also detectable via periodic consistency checks. Containment: the instrument is halted the moment an ownership mismatch is detected, exactly as in split brain — trading never continues on an instrument whose single-writer ownership is ambiguous, even briefly.

Recovery sequence: roll back or correct the routing metadata to a single, atomically applied owner; verify via fencing that only that one engine can write for the instrument; reconcile any commands possibly processed by the wrong engine against the durable journal before resuming trading, following the same replay-and-verify discipline as Scenario 01/26. How to prevent recurrence: instrument migration must use an explicit, fenced, barrier-based protocol — old owner stops and is fenced before the new owner starts, with no window where both are live — never a simple metadata flip that components might observe at different times.

Scenario 22 — Memory pressure / GC pauses

Trigger: the matching engine experiences memory pressure (unbounded order-object allocation, a leak, or — in a managed-runtime implementation — a long garbage-collection pause) under sustained high order volume. Immediate technical effect: a long GC pause (or memory-pressure slowdown) stalls the engine's event loop; commands queue up during the stall, similar in symptom to hot-symbol overload but caused by internal resource pressure rather than external volume alone. Impact on orders/trades: p99/p99.9 latency spikes sharply during the pause; sustained pressure can lead to an out-of-memory crash, folding this into Scenario 01's recovery path.

How to detect it: memory utilization and (for managed runtimes) GC pause-time metrics per engine process, alerting on pause durations that threaten the latency SLO, plus the same queue-depth/p99 signals used for hot-symbol detection. Containment: admission throttling for the affected partition, same as Scenario 16, buys time without unbounded queue growth; if pressure is severe enough to risk a crash, a controlled restart from a verified snapshot is safer than an uncontrolled OOM kill mid-command.

Recovery sequence: if contained via throttling, the engine recovers once pressure subsides and the queue drains; if a restart was required, follow the standard snapshot+replay sequence from Scenario 01. How to prevent recurrence: bounded, pre-allocated object pools for order/book node structures rather than unbounded per-order heap allocation; for managed runtimes, GC tuning or low-pause collectors chosen for latency-sensitive workloads; capacity planning that treats peak memory footprint, not just CPU, as a first-class sizing input.

The last group covers the exchange's front door — rejecting what should never reach the book in the first place — and its back door, where committed trades hand off to systems the exchange does not control.

Scenario 27 — Price band or fat-finger validation failure

Trigger: a member submits an order with a price far outside the instrument's configured price band (collar), or a quantity far outside reasonable size for that instrument/member — a classic fat-finger input error, whether from a human typo or an algorithm bug. Immediate technical effect: the pre-trade validation step (price-band/collar check, size-limit check) evaluates the order before it ever reaches the sequencer. Impact on orders/trades: the order is rejected outright with an explicit reason code; it never enters the book, is never sequenced, and has zero chance of matching — the system working exactly as intended, not a failure to contain.

How to detect it: rejection-reason metrics broken out by cause (price-band breach vs. size-limit breach vs. other) per member and instrument — a sudden spike from one member is itself a useful early warning of a client-side bug, even though the exchange-side outcome was already correct. Containment: no halt; this is inline, per-order rejection like any other validation failure. Recovery sequence: none needed beyond the member correcting and resubmitting if intended. How to prevent recurrence: keep price bands/collars and size limits tuned to the instrument's actual trading range — too tight rejects legitimate orders during real volatility, too loose fails to catch genuine mistakes — and expose clear, specific rejection reasons so member systems can self-correct quickly.

Scenario 28 — Security abuse / message flood

Trigger: a compromised or malicious session (or a buggy client) floods the gateway with an abnormally high message rate — far beyond any legitimate trading strategy's cadence — potentially a deliberate attempt to degrade performance for other members, or a runaway retry loop. Immediate technical effect: the gateway's per-session rate limiting is the first line of defense; without it, this traffic competes for the same admission/sequencing capacity as legitimate order flow from every other member. Impact on orders/trades: if uncontained, broadly elevated latency across many instruments — similar in symptom to Scenario 17's volatility burst, but caused by one abusive source rather than genuine market-wide demand.

How to detect it: per-session/per-member message-rate metrics with an alert threshold well above normal peak cadence; authentication-failure and malformed-message rate from a session as a signal of a compromised or misbehaving client; correlation IDs in audit logs let operations trace exactly which session/message drives an anomaly. Containment: the gateway throttles, and if necessary disconnects, the offending session once its rate exceeds the configured limit — isolated to that session/member, no exchange-wide halt required; authentication/authorization ensures only legitimately credentialed members can submit at all, limiting the attack surface to already-identifiable sessions.

Recovery sequence: once the offending session is disconnected/throttled, normal latency for everyone else recovers immediately, since the abusive traffic no longer competes for shared capacity; operations reviews the audit trail to decide on further restriction or compliance follow-up. How to prevent recurrence: per-member and per-session rate limits enforced as a hard architectural control (not a monitoring dashboard afterthought), strong session authentication (certificates/session keys), and tamper-evident audit logging so every abusive pattern traces back to its originating session.

Scenario 24 — Clearing/post-trade system unavailable

Trigger: the downstream clearinghouse/CCP integration (or the broader post-trade pipeline) becomes unreachable or slow to accept trade-capture deliveries. Immediate technical effect: trades continue to be matched, executed, and durably recorded on the exchange exactly as before — matching has no synchronous dependency on clearing by design; clearing, netting, settlement and custody are never presented as a single synchronous write alongside matching. Impact on orders/trades: execution reports to members are unaffected and timely; only the *delivery* of trade-capture records to the downstream clearing system is delayed.

How to detect it: delivery-lag/backlog metrics on the trade-capture-to-clearing pipeline — the gap between trades committed on the exchange and trades successfully acknowledged by clearing. Containment: no trading halt — matching and execution are unaffected by a downstream clearing outage, since durable trade capture on the exchange side happened independently and immutably at execution time; trade-capture records queue durably, never in a fragile in-memory buffer, awaiting delivery.

Recovery sequence: once clearing is reachable again, queued trade-capture records are delivered, in order, from the durable queue; a reconciliation step confirms every trade executed during the outage was eventually received and acknowledged, with any discrepancy surfaced for manual review rather than assumed resolved. How to prevent recurrence: treat trade-capture delivery to clearing as an asynchronous, durably queued, retryable pipeline from day one — never a synchronous call that could block or fail a trade's execution — and build reconciliation as a routine, automated job rather than an incident-only procedure.

The scenarios above describe what breaks; this closing chapter puts illustrative numbers on how the design behaves as load grows. Every number below is a stated assumption for this educational exercise, not a verified production benchmark for any real exchange.

1,000 → 10,000 → 100,000+ inbound commands/sec

Assumption: a single-writer engine on illustrative modern hardware sustains on the order of 50,000–100,000 simple book operations/sec per partition before queueing latency grows materially — a range, not a single number, itself dependent on hardware, command mix, and implementation. At 1,000 commands/sec exchange-wide, a handful of partitions comfortably absorb normal traffic with large headroom. At 10,000/sec, headroom per partition shrinks, and the hot-symbol partitions from Scenario 16 start to matter — most partitions are fine, but a skewed few approach their ceiling even as the exchange-wide average looks healthy. At 100,000+/sec, sustaining the workload requires genuine partition parallelism across many independently owned instruments (never parallelizing a single book, which would break deterministic ordering) plus the admission/throttling controls from Scenarios 16–18 to keep any individual partition's queue bounded.

Single hot instrument vs. 1,000 balanced instruments

A single hot instrument absorbing, say, 50,000 commands/sec is bounded by that one partition's single-writer ceiling, regardless of total exchange-wide hardware — the core lesson of Scenario 16. The same 50,000 commands/sec spread evenly across 1,000 balanced instruments (50 commands/sec/instrument on average) is comfortably within any one partition's headroom, and the exchange can scale this balanced case roughly linearly by adding more independent partitions/engines. The architectural takeaway: horizontal scale works across books, never within one book.

Market-data subscribers scaling 10x independent of matching workload

Assumption: matching workload and market-data fan-out are decoupled, so subscriber count can grow independently of order/trade volume. If a feed publishes, illustratively, 2,000 updates/sec at 200 bytes/update to 500 subscribers, that's roughly 2,000 × 200 × 500 ≈ 200 MB/sec of aggregate egress bandwidth (an illustrative calculation, not a measured figure). Growing subscribers 10× to 5,000 grows aggregate egress bandwidth 10× to roughly 2 GB/sec — entirely a fan-out/network-capacity problem, addressed by more publisher replicas and distribution-tier capacity, with zero additional load placed on the matching engine itself, since publishing one update once and fanning it out to many subscribers is a market-data-tier concern, not a matching-tier one.

Journal replay duration vs. snapshot interval

Assumption: replaying one journaled command takes on the order of microseconds to low single-digit milliseconds depending on complexity (a plain new-order apply is cheaper than one triggering several fills). The longer the interval between snapshots, the more commands must be replayed after a crash (Scenario 01) before trading can resume — directly increasing recovery time. A short snapshot interval bounds worst-case replay time tightly but adds steady-state overhead (CPU and I/O spent serializing snapshots) and storage cost for keeping multiple snapshot generations — a genuine trade-off, not a free win in either direction.

Capacity headroom, admission rate and recovery time together

Across every scale walkthrough above, the same three levers recur: how much headroom is provisioned above assumed baseline (determines how large a burst can be absorbed before admission throttling kicks in), how aggressively admission control throttles once headroom is exhausted (determines whether degraded latency or outright rejection is the user-visible symptom), and how long recovery takes when something does fail (determined mainly by snapshot interval and replication topology). None of these numbers are universal constants — they are assumptions a real exchange's operators would tune against their own measured workload, member expectations, and tolerance for visible halts.