Stock Exchange System Design — Trade-offs & Architectural Decisions

Every order book mutation — insert, match, cancel — must appear to happen atomically and in a single agreed order, or price-time priority stops being a meaningful guarantee. The question is whether to enforce that with one thread per book or with locks across many threads.

The underlying reason single-writer wins for a single book is explained at the algorithm level in Architecture Deep Dive → Matching Engine Execution Model: threads are still free to handle networking, risk checks, journaling and market-data fan-out concurrently — only the book mutation itself is serialized.

The live order book is read and mutated on every single incoming command. Whether that structure lives in process memory or behind a database round-trip determines whether the matching engine can meet a microsecond-to-low-millisecond latency budget at all.

This is not an argument against databases generally — post-trade capture, reporting, and compliance stores are database-backed by design (see Architecture Deep Dive → Post-Trade Integration). It is specifically about the structure that absorbs every match on the hot path.

When multiple resting orders are eligible to fill against an incoming order, the allocation rule decides who gets filled first — and different venues legitimately choose different rules for different instrument types.

Price levels need to be kept sorted so the engine can always find the best bid and best ask instantly. The data structure choice trades generality against raw lookup speed.

Both structures are explored with pseudocode and complexity detail in Architecture Deep Dive → Order Book Data Structures; this decision is about which family to pick for a given instrument's behavior, not the implementation itself.

Because matching is single-writer per book (Decision 01), scaling across many instruments means scaling across independent engine instances rather than scaling one engine's internal concurrency.

Determinism requires every accepted command for a given book to be assigned one unambiguous position in a single serial order. The question is whether that order is assigned by one global sequencer across the whole exchange or by a sequencer scoped to each partition.

Every accepted command must be durable before it is acknowledged as committed — but exactly when 'durable' happens relative to 'acknowledged' is a direct trade between the durability guarantee and acknowledgment latency.

The exact commit-point definition and a concrete crash-between-commit-and-ack walkthrough are covered in Architecture Deep Dive → Journal, Commit & Replay and Failure & Scale Scenarios' crash incidents.

A book needs a standby in case its current owner fails — but letting two nodes both believe they own the same book at the same time is a correctness violation, not a performance optimization.

After a crash or planned restart, the engine must rebuild an exact in-memory book from durable state. How far back it has to replay from determines recovery time.

Order entry needs a wire format that both sides agree on, that handles retransmission and gaps, and that is fast enough for the latency budget. FIX and custom binary protocols sit at different points on that trade-off.

Message framing, versioning, checksums, heartbeats and session reset mechanics for both families are covered in Architecture Deep Dive → Network Protocols & Session Management; this is not every venue standardizing on the same stack — real exchanges genuinely differ here.

TCP and UDP multicast solve different problems and are not universal substitutes for each other — the right choice depends on whether the data is a reliable one-to-one session or a lossy-tolerant one-to-many broadcast.

Subscribers need to see the book and trade prints change in near real time without putting read load anywhere near the matching hot path.

Within the matching hot path — gateway to risk to sequencer to engine to journal — every extra hop costs latency and, more importantly, every extra layer of indirection risks reordering the very commands whose order must be deterministic. This is the one decision where the industry default ('just put a message broker in the middle') deserves explicit scrutiny rather than being adopted unquestioned.

This directly reflects Section 2's explicit constraint: no 'Kafka in the matching hot path' as an unquestioned default. The question to ask before adopting a general broker on this specific path is whether its ordering and latency model was built for exactly this job, not whether it is popular or already running elsewhere in the stack.

Pre-trade risk checks (order size, notional, price collar, symbol permission) add latency on the path before a command is even sequenced. The question is whether that check must block sequencing or can be relaxed.

Amending a resting order's price or size can either preserve its existing place in the FIFO queue or push it to the back, as if it were a fresh order — and that choice changes the fairness guarantee for every other resting order at that price.

The precise state machine, versioning and idempotency handling for replace are covered in Architecture Deep Dive → Cancel/Replace Semantics; this decision is about the priority policy, not the mechanics of applying it.

Market and limit orders with Day/IOC/FOK time-in-force cover the large majority of educational and practical matching scenarios. Iceberg, stop, and auction order types each add meaningfully different logic, not just extra fields.

Once commands reach the sequencer, the policy for how it admits and orders them can be tuned for raw speed or for treating similarly-timed arrivals fairly — and pushing too far toward either extreme has a real cost.

Whatever policy is chosen, the sequencer remains the sole ordering authority — see Decision 21 and Architecture Deep Dive → Command Sequencing & Fairness for how retries, reordering and gateway failover are handled without ever falling back to wall-clock comparisons.

When a dependency the engine needs for correctness becomes unavailable or uncertain — risk state is stale, the journal can't confirm a write, a partition's ownership is ambiguous — the system must choose between stopping and continuing on uncertain information.

When overall exchange throughput needs to grow, there are two distinct levers: make each single-writer engine process commands faster, or add more independently-owned partitions. They solve different bottlenecks.

Matching for a given book must have exactly one authoritative location at any moment, but the gateways that accept orders from participants can be geographically distributed to reduce access latency.

It is tempting to order commands by the wall-clock time they were received, but clocks across different gateway machines are never perfectly synchronized, and network jitter means receipt time does not reliably reflect true arrival order anyway.

Wall-clock timestamps remain essential for audit records, latency measurement and compliance traceability (Architecture Deep Dive → Command Sequencing & Fairness and Security, Audit & Observability) — they are simply never the mechanism that decides match order.

Once a match commits, execution reports must reach both counterparties and trade prints must reach the market-data pipeline and post-trade integration. Whether that happens synchronously inline with the match or via a committed event handed off for independent fan-out changes both latency and failure isolation.

Every latency-engineering technique discussed in Architecture Deep Dive — kernel bypass, CPU pinning, NUMA-aware allocation, busy-polling — trades operational simplicity for tail-latency improvement. Most of the exchange does not need to pay that cost.

No row in this table is a universal truth — each baseline choice is correct for the assumptions stated across this module (correctness and deterministic replay prioritized over raw single-book throughput; per-instrument isolation preferred over a single shared engine; fail-closed over uncertain availability). Change the assumptions, and some of these choices change with them.