Designing a Stock Exchange: Complete System Design

We are designing the matching venue at the center of the market — the system that takes orders from many participants and decides, instant by instant, which buy meets which sell. We are not designing a broker's trading app, and we are not designing the back-office systems that settle the resulting trades.

Start with the simplest possible story. Alice owns 100 shares of ACME and wants to sell them. Bob wants to buy 100 shares of ACME. In the physical-world imagination of a stock market, Alice and Bob would need to find each other, agree on a price, and swap shares for cash directly. That doesn't scale past a handful of participants, so instead both route an order through a broker (the firm that holds their account and is a member of the exchange) to a central exchange: Alice's broker submits a sell order, Bob's broker submits a buy order, and the exchange's matching logic decides whether and how they cross.

Broker vs. exchange vs. clearinghouse vs. depository

These four roles are easy to blur together, and the blur is where a lot of system-design confusion about "stock exchanges" comes from. A broker is the member firm a participant has an account with; it handles onboarding, custody of the participant's own cash/shares from the participant's point of view, order routing, and (in real markets) its own pre-trade risk checks before it ever reaches the exchange. The exchange (Flow Exchange in this design) owns order acceptance, sequencing, matching, trade execution, and market-data publication for the instruments it lists — this is the system we are designing in this module. A clearinghouse / central counterparty (CCP) steps in after a trade executes to novate the trade (becoming buyer to every seller and seller to every buyer), netting obligations across many trades. A depository / central securities depository (CSD) holds the official record of who owns which securities and finalizes the transfer of ownership at settlement. We describe clearing and settlement only at the boundary — the exchange hands off a trade event and does not itself perform the final transfer of money and shares.

Four distinct moments, often conflated

Order submission is Bob's broker sending a buy order to the exchange. Order matching is the exchange's engine deciding that Bob's buy order crosses with Alice's resting sell order, at a specific quantity and price. Trade confirmation is the pair of execution reports (one to each side) that says a trade happened. Settlement is the eventual, separate process — typically T+1 or T+2 in many real markets, though this varies by venue and instrument class — where cash and shares actually change hands via the clearinghouse and depository. An accepted order is not an executed trade; an executed trade is not a settled trade. Conflating any two of these four moments is the single most common conceptual error when reasoning about exchange systems, and we will come back to this distinction constantly.

The continuous market and the central limit order book

For most of this design we assume continuous trading: orders arrive at any time during the trading session and are matched immediately if a cross is possible, otherwise they rest in the central limit order book (CLOB) — a live, ordered list of unmatched buy and sell orders for one instrument, organized by price and, within a price, by arrival order. Some venues also run periodic auctions (for example at the open and close) where orders accumulate and are matched all at once at a single clearing price; we describe auctions as a distinct matching mode later rather than bending continuous-matching logic to fit them.

What makes this problem genuinely hard

The difficulty is not CRUD-level complexity in any single operation — placing, cancelling, and matching an order are each individually simple. The difficulty is in the combination of six properties the system must hold simultaneously: fairness (price-time priority must never be violated, not even under load); deterministic ordering (two orders that arrive within microseconds of each other must still resolve to one unambiguous outcome, reproducibly); tail latency (a slow p99.9 directly disadvantages participants, unlike almost any other kind of backend system); durability (a crash must never lose an accepted order or silently reproduce a trade); market-data fan-out (potentially thousands of subscribers must see book and trade updates quickly, without becoming a second matching-correctness channel); and failure handling under volatility (the moments when the system is under the most load — news events, market opens, volatility spikes — are exactly the moments it must not degrade silently).

Scope and assumptions for this design

In scope: a continuous limit order book for equities-like instruments, market/limit/IOC/FOK orders, new/cancel/replace, price-time priority matching, pre-trade risk checks, a durable event journal with recovery, real-time market data, per-instrument partitioning, and the handoff to post-trade systems. Explained but not treated as universal rules: stop orders, auctions, iceberg orders, self-trade prevention, and fee/rebate structures — these are real, important, and venue-policy-dependent, so we describe them as extensions rather than baking them into the core matching logic as if every venue implements them identically. Out of scope: brokerage account onboarding/KYC, portfolio valuation and retail UI, options/derivatives pricing and margin engines, dark pools and smart order routing across multiple venues, and HFT strategy development. We use a fictional venue, Flow Exchange, and fictional instruments ACME and NOVA throughout — nothing here is a claim about the proprietary internals of any real exchange such as Nasdaq, NYSE, or NSE.

Before any architecture makes sense, a small, precise vocabulary is required. These terms are used constantly in the rest of this module, so we define each one before relying on it.

Core vocabulary

Worked example: reading the ACME book

Here is the order book we reuse for every example in this module. Tick size $0.01, lot size 1 share, continuous matching.

Notice that S1 and S2 sit at the same price, $101.00. Within a price level, orders are served FIFO by arrival sequence — S1 (sequence 10) is ahead of S2 (sequence 11), so any incoming buy that matches at $101.00 fills S1 first, then S2. This is the entire essence of price-time priority: price first, arrival time second, nothing else.

Order states and partial fills

An order moves through a small set of states: new / accepted (validated and resting or immediately processed), partially filled (some but not all of its quantity has executed; it keeps its place in the queue for the remaining quantity at its original arrival sequence), filled (fully executed), cancelled (removed before full execution, by explicit request), rejected (never accepted — failed validation or a risk check), and expired (a Day order that reached session close unfilled). A partially filled order does not get a new arrival sequence for its remaining quantity — it keeps the priority it already earned.

Order types used in this design

A brief note on stop and stop-limit orders, covered in depth in the Architecture Deep Dive: a stop order is not resting in the visible limit book waiting to match. It is held separately, watching for a trigger condition (commonly the last trade price crossing a threshold), and only becomes a live limit or market order once triggered. Treating a stop order as if it were already a passive limit order is a frequent source of confusion about book visibility and priority.

Functional requirements describe what Flow Exchange must do. We split the essential MVP from advanced extensions so the core design stays focused on what actually makes an exchange an exchange.

Member session and order management

A member (broker or authorized participant) must authenticate and establish a session with the exchange before submitting any order. Within that session, the member can submit a new order (market or limit, with a time-in-force), cancel an existing resting order, replace/amend an existing order's price and/or quantity (subject to queue-priority rules described in the Architecture Deep Dive), and query the status of an order it previously submitted.

Validation and acceptance

Every inbound command must be validated for message syntax (well-formed fields, valid instrument, valid tick-aligned price, valid lot-aligned quantity), session/market-state rules (is this member authenticated, is this instrument currently open for trading or halted), and pre-trade risk checks (does this order violate the member's size/notional limits or an exchange-wide price collar). Only orders that pass all three categories are sequenced for matching; everything else is rejected with a reason.

Matching and reporting

Accepted orders are matched using strict price-time priority, producing zero or more trades and allowing partial fills. Every state transition — accepted, partially filled, filled, cancelled, rejected, expired — must generate an execution report sent back to the originating member (and, for a trade, to both sides of the trade). The live book must emit order-book updates (new resting order, quantity change, order removed) and trade prints (price, quantity, and a unique trade identifier) for market-data consumption.

Administrative and audit functions

The exchange must support trading halts and resumes for an instrument (operator-initiated or triggered by a volatility/circuit-breaker policy), and administrative controls to manage sessions and investigate member behavior. Every accepted command and every emitted event must be written to a durable audit log sufficient for reconciliation and post-trade export to downstream clearing systems — this audit trail is a functional requirement, not an optional afterthought, because exchanges operate under regulatory and member-dispute obligations that require being able to reconstruct exactly what happened and in what order.

For most systems, availability is the top-line non-functional concern. For an exchange, correctness comes first — an exchange that is always up but occasionally matches orders out of priority order, or occasionally loses a committed trade, has failed at its actual job.

Putting a rough priority ordering on these: correctness/determinism > durability > fail-closed safety > tail latency > raw throughput. This ordering drives essentially every trade-off discussed on the companion Trade-offs & Decisions page — for example, why this design does not default to an unquestioned "just add a message broker in the hot path" architecture, and why a single-writer-per-book model is preferred over a shared multithreaded book even though the latter sounds like it should be faster.

Every number in this chapter is a stated design assumption for this walkthrough — not a measured or verified benchmark of any real exchange. The goal is to show the estimation process and the architectural conclusions it motivates, not to produce authoritative figures.

Assumed workload

Assume Flow Exchange lists 5,000 instruments, with 2,000 active trading members. At peak, the exchange receives roughly 200,000 inbound commands per second exchange-wide (new orders, cancels, and replaces combined), with significant hot-symbol skew: a small number of heavily traded instruments (think of ACME during a volatile session) can individually see a disproportionate share of that traffic, while most of the 5,000 instruments see only a trickle.

Why averages mislead: per-symbol peak

200,000 commands/sec ÷ 5,000 instruments = an average of 40 commands/sec per instrument. That average is almost meaningless for capacity planning. If, as a design assumption, the hottest 1% of instruments (50 symbols) absorb 40% of total traffic, those 50 symbols collectively see 80,000 commands/sec — 1,600 commands/sec per hot symbol on average, and a single exceptionally hot symbol during a volatility spike could see several times that briefly. Since matching for one instrument is handled by a single-writer engine (Chapter 09 explains why), the per-symbol peak, not the exchange-wide average, is what the matching engine for that instrument must sustain.

Cancel-to-new ratio and resulting trade rate

Assume a cancel-to-new ratio of roughly 3:1 (a common pattern in markets with resting limit orders that get frequently repriced) — of the 200,000 commands/sec, roughly 50,000/sec are genuinely new orders and 150,000/sec are cancels/replaces. Assume further that, as a design assumption, roughly 10% of new orders result in an immediate full or partial fill against resting liquidity. That gives an illustrative 5,000 trades/sec exchange-wide at peak — again dominated by whichever instruments are hottest at that moment.

Journal storage per day

Assume each journal record (an accepted command or an emitted execution/trade event) averages 200 bytes including headers, sequence number, and a schema version tag. At 200,000 commands/sec plus a comparable volume of emitted events (roughly 1.5× the command rate once acks, fills, and book deltas are counted, so ~300,000 records/sec), that is: 300,000 records/sec × 200 bytes = 60 MB/sec, or 60 MB/sec × 86,400 sec/day ≈ 5.2 TB/day of raw journal data exchange-wide. This is a substantial but entirely manageable write volume for modern disks and a sensible retention/compaction policy — it is not, by itself, a reason to avoid durable journaling.

Market-data bandwidth

Assume an average of 50,000 book/trade update messages per second exchange-wide (book deltas plus trade prints — lower than the raw command rate because many cancels/replaces at the back of the book don't move the top levels that most feeds care about), each averaging 80 bytes, fanned out to an assumed 20,000 market-data subscribers who, on average, are each interested in a subset of instruments equivalent to about 10% of total message volume. Raw publish-side bandwidth: 50,000 msgs/sec × 80 bytes = 4 MB/sec of unique content. Fan-out bandwidth (subscriber-side, before any multicast efficiency): 20,000 subscribers × 10% × 4 MB/sec ≈ 8 GB/sec of total egress if every subscriber required an independent unicast stream of their subset — which is precisely why fan-out architecture (tiered distribution, multicast where applicable) matters far more for market data than for the matching path itself, discussed further in Chapter 12.

Snapshot size and recovery-time implications

Assume an average resting order record of 120 bytes (instrument, side, price, remaining quantity, order ID, sequence) and an assumed average of 2,000 resting orders per active instrument at peak for the busiest symbols. A single hot-instrument snapshot: 2,000 × 120 bytes ≈ 240 KB — trivial to write and trivial to load. Recovery time after a crash is dominated not by snapshot load but by journal replay from the last snapshot to the current sequence; if snapshots are taken every 30 seconds of journal volume, worst-case replay is bounded to roughly 30 seconds of that instrument's command rate, which at 1,600 commands/sec for a hot symbol is about 48,000 records — replayable in a small fraction of a second for an in-memory engine with no external I/O per record. This is an illustrative calculation, not a guaranteed recovery-time SLA.

Latency budget across the synchronous path

The order-to-acknowledgment path has a small number of synchronous hops: gateway receipt → pre-trade risk check → sequencer assignment → matching engine apply → journal commit → acknowledgment. Everything after journal commit — market-data publication, post-trade event delivery — is asynchronous and must not block the acknowledgment. As an illustrative budget (not a benchmark): gateway parsing ~50 µs, risk check ~100 µs, sequencer hop ~50 µs, matching apply ~50 µs, journal commit ~500 µs–2 ms depending on the durability policy chosen (Chapter 13), acknowledgment send ~50 µs. The journal commit step dominates the budget whenever synchronous durability is required — which is exactly the trade-off explored in the Trade-offs & Decisions page.

The data model has to represent three very different kinds of state: the live in-memory book that changes on every match, the immutable journal that is the durable source of truth, and the analytical/reporting stores that downstream systems query. Confusing these three is a common design mistake.

Why prices are integer ticks, never floating point

Every price in this data model is stored as an integer number of ticks, not a floating-point decimal. $101.00 at a tick size of $0.01 is stored as the integer 10100. Floating-point arithmetic can introduce rounding error that is unacceptable when comparing prices for exact equality (price-time priority depends on exact price matching to decide which level an order belongs to) and when summing fill values for trade reporting. This is a standard, well-understood practice for any monetary or price field, not specific to exchanges — but it matters especially here because matching logic directly compares prices for ordering and equality.

Sample order record

{
  "exchange_order_id": "XO-00098213",
  "client_order_id": "brk7-ord-44120",
  "instrument_id": "ACME",
  "side": "BUY",
  "order_type": "LIMIT",
  "time_in_force": "DAY",
  "limit_price_ticks": 10100,
  "original_quantity": 60,
  "remaining_quantity": 0,
  "member_id": "MEMBER-0042",
  "state": "FILLED",
  "version": 2,
  "arrival_sequence": 9981204
}

State transitions and constraints

In-memory state vs. immutable journal vs. analytical stores

The in-memory operational state (the live order book inside the matching engine process) is optimized for matching speed and is, by design, disposable — it can always be reconstructed from the journal. The immutable journal (durable, append-only, sequence-ordered) is the actual source of truth; every committed command and emitted event lives here, and it is never edited, only appended to. Analytical/reporting stores (used for post-trade reporting, compliance queries, historical analysis) are derived, asynchronously populated projections of the journal — they are convenient for querying but are never the authority for "did this trade happen." If the analytical store and the journal ever disagree, the journal wins, by construction.

Members interact with Flow Exchange through a small set of message types: new order, cancel, replace, status query, and market-data subscription. The illustrative examples below use FIX-style tags and REST-style pseudocode purely as teaching devices — they are not a claim that production exchanges necessarily expose REST matching APIs, and real venues vary in their actual wire protocols.

Illustrative FIX-style new order

# Illustrative FIX-style New Order Single (tags abbreviated for teaching purposes)
35=D (MsgType=NewOrderSingle)
11=brk7-ord-44120        (ClOrdID — member's client order ID, used for idempotency)
55=ACME                  (Symbol)
54=1                     (Side: 1=Buy, 2=Sell)
38=60                    (OrderQty)
40=2                     (OrdType: 2=Limit)
44=10100                 (Price, in integer ticks of $0.01 for this illustration)
59=0                     (TimeInForce: 0=Day)
34=55102                 (MsgSeqNum — session-level sequence number)

# Illustrative FIX-style Execution Report in response
35=8 (MsgType=ExecutionReport)
11=brk7-ord-44120        (ClOrdID, echoed back)
37=XO-00098213           (OrderID — exchange-assigned)
17=EX-7781120            (ExecID — unique per execution report)
150=0                    (ExecType: 0=New/Accepted)
39=0                     (OrdStatus: 0=New)
151=60                   (LeavesQty — remaining quantity)

Illustrative REST-style pseudocode (educational representation only)

POST /v1/orders
Authorization: Bearer {member_session_token}
Idempotency-Key: brk7-ord-44120

{
  "instrument": "ACME",
  "side": "BUY",
  "order_type": "LIMIT",
  "limit_price_ticks": 10100,
  "quantity": 60,
  "time_in_force": "DAY",
  "client_order_id": "brk7-ord-44120"
}

Response 202 Accepted:
{
  "exchange_order_id": "XO-00098213",
  "status": "ACCEPTED",
  "sequence_number": 9981204
}

POST /v1/orders/{exchange_order_id}/cancel
Authorization: Bearer {member_session_token}
Idempotency-Key: brk7-cancel-99231

Response 200 OK:
{
  "exchange_order_id": "XO-00098213",
  "status": "CANCELLED",
  "remaining_quantity_at_cancel": 20
}

GET /v1/orders/{exchange_order_id}
Authorization: Bearer {member_session_token}

Response 200 OK:
{
  "exchange_order_id": "XO-00098213",
  "client_order_id": "brk7-ord-44120",
  "status": "PARTIALLY_FILLED",
  "original_quantity": 60,
  "remaining_quantity": 20,
  "version": 1
}

Idempotency and session sequence numbers

Every new-order request carries a member-assigned client_order_id (FIX ClOrdID / REST Idempotency-Key). If a network retry causes the exact same client_order_id to arrive twice, the exchange must recognize the duplicate and return the same outcome without creating a second order or a second match — it must never create duplicate matching effects. This is the mechanism behind canonical Test G (duplicate client command ID): no duplicate order or execution is created; the exchange returns a consistent status for the already-processed command. Separately, each member session carries its own monotonically increasing MsgSeqNum-style sequence, used to detect gaps or out-of-order delivery at the session layer and trigger a resend/replay request — this is distinct from the exchange-internal matching sequence number assigned by the sequencer (Chapter 10).

Validation performed before sequencing

Before a command is handed to the sequencer, the gateway and risk layer check: price tick alignment (a limit price must be an exact multiple of the instrument's tick size), quantity lot alignment, member entitlement for this instrument and order type, and trading-halt state for the instrument. Any failure here produces a reject execution report and the command is never sequenced — it never gets an arrival_sequence and never touches the book.

Correlation IDs and replay/gap recovery

Every execution report carries both the member's client_order_id and the exchange's exchange_order_id, so a member can correlate outbound reports with its own outbound requests regardless of delivery order. If a member detects a gap in its inbound execution-report stream (missing sequence numbers), it requests a replay of the missing range rather than assuming what happened — execution reports are retrievable, not just a best-effort push.

The smallest correct version of this system is one gateway process and one matching process, with the order book held entirely in memory. This is not a toy to be embarrassed about — it is the right starting point, and it is correct within its limits.

Picture a single process: it accepts member connections, validates incoming orders inline, and holds the ACME order book as a simple in-memory structure — two sorted collections of price levels, each a FIFO queue of resting orders. A new order arrives, gets validated, and is handed directly to the matching function, which compares it against the opposite side of the book and either matches it immediately or rests it. There is no network hop between "accept" and "match" — it's a single function call.

Walking through placing and matching an order

Starting from the canonical ACME book (S1 $101.00×50 seq 10, S2 $101.00×30 seq 11, S3 $101.50×70 seq 12, B1 $100.50×40 seq 13, B2 $100.00×80 seq 14), an incoming limit BUY 60 @ $101.00 arrives (Test A). The matching function compares the incoming order's price against the best ask ($101.00) — they cross, since the buy's limit is at or above the best ask. It fills against the head of the $101.00 queue: S1 first (50 shares, exhausting S1), then S2 (10 of its 30 shares, since 10 remain of the incoming order). Two trades are produced: 50 @ $101.00 and 10 @ $101.00. S2 now rests with 20 remaining, still at $101.00, still at its original arrival sequence 11. The incoming buy has 0 remaining — fully filled, nothing rests.

What works, and where it breaks

This single-process design is genuinely correct for a single instrument under light load: price-time priority is honored exactly (it's just sorted-collection logic), and there is no concurrency bug to worry about because there is exactly one thread touching the book. What it does not yet have is durability — if the process crashes, the entire in-memory book for ACME is gone, along with any orders resting in it, with no way to recover what was there. It also has no network resilience (a single gateway connection drop loses in-flight state), no scaling beyond one instrument's worth of traffic on one machine, and no market-data publication to anyone outside the process.

Introducing the journal before claiming crash-safety

The very first evolution — before anything resembling microservices — is to append every accepted command to a durable, append-only journal before applying it to the in-memory book, and only acknowledge the member after that journal write is durably committed. This single change is what allows the design to honestly claim "a crash doesn't lose an accepted order": on restart, replaying the journal from the beginning (or from the last snapshot) reconstructs the exact same book, because the matching function is deterministic given the same input sequence. We defer the full mechanics — snapshotting, checksums, replication — to Chapter 13 and the Architecture Deep Dive; here the point is simply that durability changes what you're allowed to claim about the acknowledgment, and you should not claim crash-safe acknowledgment without it.

No premature microservices: there is no reason, at this stage, to split validation, risk-checking, sequencing, and matching into separate network-hopping services. That split becomes valuable only once specific pressures (described in Chapter 10) justify the added latency and operational complexity of a network hop between them.

This chapter walks through price-time priority conceptually using the canonical test cases. The full algorithm pseudocode, data structures, and complexity analysis live in the Architecture Deep Dive — this chapter is about understanding the behavior, not the implementation.

How does a stock exchange match orders?

An incoming order is compared against the best price on the opposite side of the book. A buy order crosses if its limit price is at or above the best ask; a sell order crosses if its limit price is at or below the best bid. If it crosses, it fills against the resting order(s) at the best price, in arrival order (FIFO) within that price level, for min(incoming remaining, resting remaining) shares at a time, continuing to sweep successively worse price levels if its remaining quantity and limit still allow it, until either its quantity is exhausted or it can no longer cross. Any quantity left over rests in the book at the incoming order's own limit price and gets a new arrival sequence at the moment it starts resting.

What is price-time priority?

Price-time priority means: better price always wins, and among orders at the same price, earlier arrival always wins. It is a strictly per-instrument rule — there is no global cross-instrument ordering guarantee, and nothing in this design implies that an ACME order and a NOVA order have any defined relative priority to each other, because they live in entirely separate books processed independently.

Test A — same-price FIFO fill

Incoming limit BUY 60 @ $101.00 against the canonical book. Both S1 and S2 sit at $101.00. S1 (sequence 10) is ahead of S2 (sequence 11), so the incoming order fills 50 against S1 (exhausting it) and then 10 against S2, leaving S2 with 20 remaining and the incoming buy fully filled (0 remaining). Two separate trade records are produced — the quantity split is a direct consequence of FIFO within a price level, not an arbitrary choice.

Test B — multi-level sweep, limit price as a ceiling

Resetting to the original book, incoming limit BUY 100 @ $101.50. It can cross both the $101.00 level and the $101.50 level, since both are at or below its limit. It fills S1 (50 @ $101.00), then S2 (30 @ $101.00), exhausting the $101.00 level, then continues to S3 (20 of its 70 @ $101.50). S3 rests with 50 remaining at $101.50. The key lesson: the incoming order's limit price of $101.50 is a ceiling on how much it will pay, not a guarantee that it executes at $101.50 — most of its fill happened at the strictly better price of $101.00.

Test C — no execution, new best bid

Resetting again, incoming limit BUY 20 @ $100.75. The best ask is still $101.00, so this buy's limit does not cross anything — $100.75 is below $101.00. It rests, and because $100.75 is better (higher) than the previous best bid of $100.50, it becomes the new best bid, ahead of B1 in price priority even though B1 arrived earlier — price always beats time when the prices differ.

How do market orders behave?

A market order has no limit price — it accepts whatever price is available, sweeping through resting levels until its quantity is filled or the book on that side is exhausted. It never guarantees a price and, if the book is thin or empty on that side, may be left partially filled or entirely unfilled, depending on venue policy for what happens to any unfilled residual (commonly cancelled immediately, since a market order resting indefinitely at an undefined price doesn't make sense in a continuous book).

How do IOC and FOK orders differ?

Test E — IOC BUY 100 @ $101.00 from the original book: it fills whatever is immediately executable at or better than $101.00 — that's S1 (50) and S2 (30), for 80 shares total — and the remaining 20 shares are cancelled immediately, never resting. Test F — FOK BUY 100 @ $101.00 from the original book: the engine first checks whether the *entire* 100 shares are executable at $101.00 or better. Only 80 shares are available at that price (S3 is at $101.50, outside this order's limit), so the feasibility check fails and the entire order is killed with zero execution — no partial side effects. The distinguishing idea: IOC takes what it can get; FOK takes all of it or none of it. Both are evaluated against the book as it is serialized to appear to this one command — not reassessed partway through against a book that could be changing concurrently, because the book is only ever mutated by one serialized command at a time (Chapter 10).

Day expiry and an optional stop trigger

A Day order that never executes simply expires, unfilled, when the session's close is reached — this is a scheduled, deterministic transition, not a failure. A stop order (mentioned here, detailed in the Architecture Deep Dive) is not part of the visible resting book at all until its trigger condition fires; once triggered, it is submitted as a new order (market or limit) and only then takes a fresh arrival sequence.

Cancel/replace and queue-priority consequences

Cancelling a resting order removes it from its price level's FIFO queue entirely. Replacing (amending) an order's price always loses queue priority — a changed price is logically a different price level, so it must re-enter the back of that level's queue. Whether a quantity-only decrease preserves the original arrival sequence or also loses priority is a venue-specific policy choice, not a universal rule — some venues preserve priority for a pure reduction and only reset it for a price change or a quantity *increase*. We state our own policy explicitly in the Architecture Deep Dive rather than assume one convention applies everywhere.

Market-state checks

Before any command is matched, the engine checks the instrument's current trading phase. During a halt, no new matching occurs (specific handling of in-flight commands during the halt transition is a policy choice, covered in Chapter 15 and on Failure & Scale Scenarios); during an auction phase, matching logic is entirely different (a single clearing-price calculation rather than continuous crossing) and is not forced into the continuous-matching pseudocode.

The exact algorithm — pseudocode for incoming limit buy/sell, the data structures that make price-level lookup and FIFO queueing efficient, and the precise invariant checks — is covered in full in the Architecture Deep Dive: Price-Time Priority Algorithm and Order Book Data Structures.

Each production component below is introduced because the single-process design from Chapter 08 cannot satisfy a specific requirement from Chapters 03-05 — not because it is a standard component every system design is expected to include.

This chapter traces one concrete order — Test A's BUY 60 @ $101.00 — through every production component introduced in Chapter 10, from arrival to post-trade handoff, making the acknowledgment/durability contract explicit at each step.

The numbered sequence

1. Member submits a new limit BUY 60 @ $101.00 for ACME, with client_order_id = brk7-ord-44120, over an authenticated session. 2. Gateway parses and frames the message, checks the session is valid, and forwards it. 3. Risk engine checks tick alignment ($101.00 is a valid multiple of $0.01), lot alignment (60 is a whole number of shares), the member's size/notional limits, and that ACME is not halted. It passes. 4. Instrument router directs the command to the partition that owns ACME. 5. Sequencer assigns the next sequence number for the ACME partition — say, sequence 9981204 — establishing this command's exact position relative to every other command on this instrument. 6. Matching engine applies sequence 9981204: it compares the incoming buy against the book, finds it crosses, and executes two fills (50 @ $101.00 vs. S1, 10 @ $101.00 vs. S2), leaving S2 with 20 remaining and the incoming order fully filled. 7. Journal durably commits the accepted command and both resulting fill events, synchronously, before anything is acknowledged. 8. Acknowledgment is sent to the submitting member: an accepted execution report, immediately followed by two fill execution reports (or combined, depending on protocol) — this is the durability point: once acknowledged, these fills are guaranteed to have survived a crash. 9. Both counterparties receive execution reports: the incoming buyer sees two partial fills completing their order, and separately, S1's and S2's original sellers (from earlier in the day) each receive their own execution reports reflecting that their resting orders were filled or partially filled. 10. Book delta and trade prints are asynchronously published to the market-data pipeline (two trade prints at $101.00, and a book delta reflecting S1 fully removed and S2's remaining quantity reduced to 20). 11. Trade capture forwards the two immutable trade records toward the post-trade/clearing integration, asynchronously, for eventual netting and settlement — this step does not block or affect the already-committed execution.

Acknowledgment semantics: precisely defining the durability point

The acknowledgment the member receives means exactly this: "this command has been durably committed to the journal at this sequence number, and any resulting fills are final." It does not mean the trade has settled, and it does not mean the member's own broker-side books have reconciled — those are separate, later, asynchronous facts. If the matching engine process crashed one instruction after committing the journal entry but before sending the acknowledgment, replay on restart reconstructs the identical committed state (same fills, same trade IDs), and the delayed acknowledgment is simply sent once recovery completes — the member sees a late but correct response, not a lost or duplicated one. This exact scenario is explored further in Failure & Scale Scenarios.

Event ordering vs. command acceptance

It's worth being explicit about what is and isn't guaranteed to happen "at the same time." Command acceptance (steps 1-5) establishes that the order is now part of the deterministic sequence — it does not yet say anything about whether it will execute. Execution (step 6) is a separate, subsequent fact determined by matching against the book as it exists at that exact sequence position. A command can be accepted and sequenced and still result in zero fills (it simply rests), and this is a completely normal, successful outcome — not a partial failure.

Data table: the full event trail for Test A

Market data tells the world what the book and the tape look like. It must never be mistaken for the authoritative record of what actually executed — that authority lives only in the matching engine's committed journal.

What market data contains

At minimum: the best bid and offer (BBO) — top-of-book price and quantity on each side; depth / level 2 — quantity at multiple price levels beyond the top; last traded price and size; and trade prints — a public record of each executed trade (price, quantity, timestamp), without necessarily revealing the identities of the two counterparties.

Snapshot + incremental deltas

Rather than resending the full book on every change, the feed publishes an initial snapshot (full book state at a point in time) and then a stream of incremental deltas (this price level's quantity changed, this level was added/removed, a trade occurred). Each message on the feed carries its own feed sequence number, independent of the internal matching sequence — a subscriber applies deltas in feed-sequence order on top of its last known snapshot to maintain an accurate local view.

Gap detection and resynchronization

If a subscriber observes a jump in feed sequence numbers (for example, receiving sequence 5008 right after 5005, having missed 5006-5007), it must not guess what it missed. It requests (or simply waits for) a fresh snapshot and resumes applying deltas from that snapshot's sequence forward. This is exactly canonical invariant: market data can be reconstructed/resynchronized from a snapshot plus contiguous deltas — it is never patched over with a guess.

Direct feeds vs. public data APIs

Real venues commonly distinguish direct/raw feeds (lower latency, typically entitlement-gated and offered to paying subscribers or co-located participants) from consolidated/public data APIs (higher latency, broader distribution, sometimes delayed by policy). This design acknowledges that distinction as a venue-policy and entitlement question rather than asserting one universal distribution model — Flow Exchange's architecture supports tiered distribution without mandating a specific commercial entitlement scheme.

Why market-data fan-out stays off the matching critical path

From Chapter 05's capacity estimate, market-data fan-out bandwidth (potentially several GB/sec at the subscriber edge) can exceed the matching engine's own command-processing volume by a wide margin. If market-data publication were synchronous with matching — for example, if the engine had to wait for an acknowledgment from a slow subscriber before processing the next command — a single slow or congested subscriber could stall the entire book. Publication is therefore strictly asynchronous and downstream of a committed match: the matching engine emits an event once, and the market-data pipeline's own fan-out performance is its own concern, isolated from matching latency.

Backpressure and coalescing

A subscriber that cannot keep up with the feed's rate needs a defined policy, not silent data loss passed off as "fine." Common approaches include coalescing (collapsing several deltas to the same price level into one combined update when a consumer is behind, since only the latest state at that level matters for book reconstruction) and bounded backlog with explicit disconnect-and-resync (rather than unbounded buffering that eventually exhausts memory). Either way, the correctness boundary is firm: a slow subscriber may see stale data, but it must never be allowed to silently diverge into wrong data without a detectable gap.

The full snapshot/delta protocol design, sequence-gap mechanics, and fan-out tier architecture are detailed in the Architecture Deep Dive: Market Data Architecture.

What happens if the matching engine crashes? The short, precise answer: the committed journal plus the most recent snapshot deterministically reconstruct the exact book and trade history that existed at the last committed sequence number — nothing is re-decided, only re-derived.

Write-ahead journal and commit semantics

Every sequenced command and every resulting event is appended to a durable, ordered, append-only journal before the corresponding acknowledgment is sent. This ordering is what makes the acknowledgment contract from Chapter 11 honest: "acknowledged" genuinely means "durably recorded, will survive a crash from this point on." The exact durability mechanism — synchronous disk write, group commit, or synchronous replication to a standby — involves a real latency/durability trade-off explored on the Trade-offs & Decisions page; this chapter assumes some such mechanism exists and focuses on what recovery does with it.

Snapshots, offsets, checksums, and epochs

Periodically (Chapter 05's example: roughly every 30 seconds of journal volume for a hot instrument), the engine serializes a full snapshot of its order book, tagged with the exact sequence number it corresponds to, a checksum (to detect corruption), and the current epoch (the leader term under which it was produced, explained below). Recovery after a crash loads the most recent valid snapshot and replays the journal from snapshot-sequence + 1 onward, deterministically reapplying each command exactly as it was originally applied, reconstructing an identical book and identical trade IDs.

What happens if the matching engine crashes?

Concretely: the process (or its host) fails. A standby for that partition — already continuously replaying the same journal stream, or catching up from the last point it had reached — takes over. Before it is allowed to begin accepting new commands as the active engine, it must be granted a new epoch (a strictly increasing term number) and the old engine must be fenced: any attempt by the old instance to write to the journal or publish as if it were still active must be rejected, because epoch numbers only move forward and writers must present a current epoch to be accepted. This is what prevents two primaries from ever matching simultaneously for the same instrument — a correctness invariant, not merely an operational nicety.

Reconciling outbound reports without duplicate effects

Recovery also has to handle outbound delivery, not just internal state: if a fill was committed to the journal but the execution report confirming it had not yet reached the member before the crash, the recovered engine must be able to redeliver that report (since members can detect and request replay of gaps, Chapter 07) without this redelivery somehow causing a second, duplicate fill. Because the fill itself is already a committed fact in the journal — replaying the journal reconstructs the same trade ID, not a new one — redelivering the resulting report is a safe, idempotent operation from the member's point of view.

Temporary halt vs. unsafe continued trading

If a partition's standby is not yet caught up, or fencing cannot be confirmed with confidence, the correct response is a temporary halt of that instrument rather than allowing uncertain or potentially duplicated processing to continue. This connects directly back to Chapter 04's NFR ranking: fail-closed safety outranks availability whenever the alternative is matching that cannot be guaranteed correct.

Detailed incident walkthroughs — crash after commit but before acknowledgment, split-brain prevention specifics, corrupted snapshot recovery, and more — are covered with full sequence timelines on Failure & Scale Scenarios. The replication protocol itself (quorum assumptions, standby catch-up mechanics, when promotion is safe) is detailed in the Architecture Deep Dive: Replication and Leader Fencing.

Why is a matching engine often single-writer? Because determinism and strict price-time priority are far easier to guarantee with one serialized writer per book than with any scheme that lets multiple threads mutate the same book concurrently — so scaling has to come from more independent books, not from parallelizing one book.

Symbol-based partitioning

Each instrument is assigned to exactly one partition, owned by exactly one active matching-engine instance at a time (per Chapter 13's epoch/fencing rules). ACME and NOVA, for instance, can be routed to entirely different partitions, processed by different engine instances, on different machines, with no coordination required between them — because price-time priority is a per-instrument guarantee, there is no correctness requirement linking the two books together.

Hot symbols and the single-writer ceiling

From Chapter 05's estimate, a hot instrument might see on the order of 1,600 commands/sec at peak, with occasional bursts well above that during a volatility event. Because each instrument's matching is single-writer, a single hot instrument's throughput ceiling is the throughput ceiling of one engine instance for that instrument — adding more machines elsewhere does nothing to help the one engine handling ACME if ACME itself becomes extraordinarily hot. This is a genuine, architecturally important limitation, not an edge case to hand-wave past.

How do you handle a hot stock?

Several complementary responses, none of which involve splitting one instrument's single-writer book: vertical headroom for the engine process itself (fast single-threaded or carefully pinned execution, minimizing per-command overhead, discussed further in the Architecture Deep Dive's latency chapter); admission control and backpressure upstream of the sequencer, so that an overload produces bounded queueing and explicit throttling signals rather than unbounded memory growth or silent latency collapse; and, at the extreme, venue-level circuit breakers / volatility halts that temporarily pause trading in that instrument entirely, trading continuity for safety during genuinely extreme conditions (Chapter 15). What does not help, and would break correctness, is attempting naive parallel writes to the same book from multiple threads — see the single-writer-vs-multithreaded comparison in Trade-offs & Decisions.

Horizontal scale across independent books

The actual scaling lever for exchange-wide throughput is more partitions, each independently single-writer, spread across more machines — not a faster single book. Five thousand instruments distributed across, say, dozens of engine instances (grouped so that no single instance is overloaded by the sum of its assigned symbols) scales roughly linearly with the number of partitions, as long as hot symbols are distributed rather than accidentally clustered onto the same instance.

Gateway scaling and session affinity

Session gateways scale horizontally and independently of matching partitions — a member's session can be handled by any available gateway instance, with the gateway responsible for routing each command to the correct instrument partition. Gateway failover requires its own care: a dropped session must not silently lose in-flight commands, and reconnection must reconcile the member's own sequence numbers (Chapter 07) to detect and recover any gap.

Market-data fan-out and network capacity

As established in Chapter 12, market-data fan-out bandwidth scales with subscriber count × subscribed-instrument breadth, largely independent of matching-engine partition count. Scaling the publisher tier (more distribution nodes, tiered fan-out) is a separate scaling axis from scaling matching partitions, and the two should be capacity-planned separately.

Separating reporting/analytics from critical matching

Historical queries, compliance reporting, and analytics workloads must run against derived, asynchronously populated stores (Chapter 06) — never against the live matching engine's in-memory state or its synchronous journal-write path. This keeps a heavy analytical query from ever being able to add latency to the matching critical path.

Regional deployments and venue authority

A venue may operate gateways in multiple geographic regions for member connectivity latency reasons, but this design does not claim or require an active-active same-book guarantee across regions — there remains exactly one authoritative matching authority (one active engine, one epoch) for a given instrument's book at any moment, regardless of how many regions members connect from. Claiming otherwise would violate the single-writer-per-book correctness model from Chapter 13.

An exchange is a regulated, adversarial-environment system: it must defend against member error, market stress, and deliberate abuse simultaneously, while remaining fully auditable after the fact.

Authentication, permissions, and per-member limits

Every session is authenticated (Chapter 07), and every member carries explicit entitlements (which instruments and order types it may use) and risk limits (max order size, max notional) enforced at the pre-trade risk layer before a command is ever sequenced. These are exchange-side controls, layered on top of — and independent from — whatever risk checks a member's own broker systems perform upstream before an order ever reaches the exchange.

Fat-finger checks, price collars, throttling, kill switches

A fat-finger check rejects an order whose size or price is wildly outside reasonable bounds (for example, a quantity with an extra zero accidentally typed). A price collar rejects or flags an order priced too far from the current reference price (for example, a limit far outside a configured percentage band around the last trade or a reference price), protecting against a single erroneous order sweeping an unreasonable distance through the book. Throttling limits how many commands a single member session may submit per unit time, bounding the blast radius of a malfunctioning client application. A kill switch lets an operator (or, in some designs, the member itself) immediately halt a specific member's ability to submit new orders and optionally cancel all of that member's resting orders, as an emergency control during a detected malfunction.

Self-trade prevention as a policy-dependent extension

Some venues offer self-trade prevention (STP): an optional mechanism that stops a member's own buy and sell orders from matching against each other (which can otherwise create the appearance of artificial trading volume with no actual change in beneficial ownership). We describe this as a policy-dependent extension, configurable per member or per order, rather than a universal matching rule every venue necessarily applies by default.

Why might you halt rather than keep serving?

Trading halts and volatility interruptions exist because, at certain moments, continuing to match orders would itself be the unsafe choice — for example, a sudden detected imbalance, a suspected data/connectivity problem, or a price move so extreme it suggests the book no longer reflects a fair, orderly market. A halt pauses new matching for the affected instrument (with a clearly stated policy for what happens to in-flight and newly arriving orders during the halt — reject, queue, or cancel, depending on the chosen policy), and a controlled reopen (commonly via an auction mechanism that establishes a single clearing price from accumulated orders, rather than resuming continuous matching cold) restores orderly trading. This again reflects Chapter 04's ranking: safety can outrank availability.

Audit trail, compliance interfaces, and time synchronization

Every committed command and event is retained in the immutable journal (Chapter 06) with its sequence number and exchange-receipt timestamp, forming a tamper-evident audit trail sufficient to reconstruct exactly what happened, in what order, for any member dispute or compliance inquiry. Because matching order is sequencer-determined rather than wall-clock-determined (Chapter 09), accurate time synchronization across components still matters — not to decide matching order, but so that timestamps attached for audit purposes are trustworthy and comparable across the system.

Monitoring: what actually gets watched

DDoS and protocol-abuse boundaries

The session gateway layer is the natural boundary for connection-level abuse defenses (rate limiting, connection caps, malformed-message handling) before any such traffic can reach the risk engine or sequencer. This keeps protocol-level abuse handling architecturally separate from trading-rule enforcement, even though both ultimately protect the same system.

Pulling every chapter together: one synchronous critical path that guarantees correctness and durability, surrounded by asynchronous consumers that extend the system's reach without ever being allowed to threaten that guarantee.

Tracing the full lifecycle one more time

A member submits an order. The gateway authenticates and frames it. The pre-trade risk engine validates it against tick/lot rules, entitlements, size/notional limits, and halt state. The instrument router sends it to the correct partition. The sequencer assigns it a deterministic position in that partition's command stream. The single-writer matching engine applies it — comparing against the opposite side of the book under strict price-time priority, producing zero or more fills. The journal durably commits the command and every resulting event, synchronously, before anything is acknowledged. The member (and any counterparties) receive execution reports. Asynchronously and independently, market data publishes book deltas and trade prints to subscribers, and the post-trade pipeline forwards immutable trade records toward clearing and eventual settlement — a separate, later, explicitly asynchronous process that this exchange design hands off to but does not itself perform.

Annotating ownership

Design evolution, summarized

Explicit trade-offs and limitations of this design

This design is representative, not exhaustive, and it makes real trade-offs rather than claiming to be universally optimal. It assumes single-region authoritative matching per instrument, not a true active-active multi-region matching model. It assumes a synchronous pre-trade risk check, which adds latency compared to a deferred-check design that some venues might choose under different assumptions. It treats auctions, stop orders, icebergs, and self-trade prevention as explained extensions rather than core matching-path features. And every numeric figure in this module — capacity estimates, latency budgets, recovery times — is an explicitly stated design assumption for teaching purposes, never a verified benchmark of a real production exchange.