Event Sourcing

Event sourcing stores the authoritative history of a business entity as an ordered sequence of events instead of storing only its latest state. To get the current state, the application starts from an initial state and applies the events one after another. A bank account is AccountOpened, FundsDeposited, FundsWithdrawn; an order is OrderPlaced, PaymentAuthorized, OrderCancelled. The stream explains how the current state came to exist, not just what it is now.

The key word is authoritative. A conventional application updates a balance column and may write audit messages beside it; if the two ever disagree, the column is the truth. In an event-sourced system the committed events are the truth, and the balance is a value computed from them. Snapshots and read tables are caches of that computation, and they can be thrown away and rebuilt as long as the history and the deterministic rules that interpret it still exist.

The mental model is a left fold: state = fold(apply, initialState, events). Everything else in the pattern follows from taking that sentence seriously. Because state is a function of history, you can compute it as of any past version, derive several different views from the same events, and fix a wrong view by recomputing it. Because history is the record, you must also get the events right the first time, keep them readable for years, decide how concurrent writers are ordered, and never confuse re-reading history with re-doing what it describes.

This lesson follows one account through those consequences: what an event is and how it differs from a command, how replay and snapshots work, how concurrent writers are kept consistent, how events change shape over time, how projections are built and rebuilt, and where the guarantees stop. Event sourcing is independent of CQRS, and the two are often confused; that boundary is drawn in Event Sourcing and CQRS.

A command asks the system to do something: WithdrawFunds(500). It can be rejected because the account is frozen or the balance is too low. An event records a fact that has already been accepted: FundsWithdrawn(500). Events are named in the past tense, in business language, and are never refused after the fact. A command handler loads the aggregate's current state, checks the invariants, and appends new events only if the command is valid.

The handler has two distinct pure functions, and keeping them apart is what makes replay safe. The decide function looks at current state and a command and returns either events or a rejection; it is where business rules live. The apply (or evolve) function takes state and an event that has already happened and returns the next state; it contains no rules, because it cannot refuse history. If a new invariant were added to apply, old streams that predate it could no longer be loaded.

function decide(state, command) {
  if (command.type === 'WithdrawFunds') {
    if (!state.opened) return { ok: false, reason: 'ACCOUNT_NOT_OPEN' };
    if (state.balanceMinor < command.amountMinor) return { ok: false, reason: 'INSUFFICIENT_FUNDS' };
    return { ok: true, events: [{ type: 'FundsWithdrawn', amountMinor: command.amountMinor }] };
  }
  return { ok: false, reason: 'UNKNOWN_COMMAND' };
}

What an event should contain

An event carries enough to rebuild state deterministically without asking anyone else. That includes the results of decisions, not only the inputs. If a withdrawal incurred a fee computed from a rate table, the event records the fee; if an order was priced with a currency conversion, the event records the rate that was used. Otherwise replay would recompute them with today's data and arrive at a different history. Prefer events that express intent and meaning, such as OrderCancelled(reason), over generic RowUpdated or AccountChanged events that record that something changed but not why. A UI log line like "user clicked Save" is not a domain event.

Rejected commands are normally not stored. If the business needs to remember a refusal, for example a declined withdrawal that feeds a fraud model, model it as its own event, WithdrawalDeclined, which is a fact in its own right.

Do not replay actions

An event is not an instruction to perform a side effect again. Replaying PaymentAuthorized rebuilds internal state; it must not charge the card a second time. Keep the pure projection of events into state strictly separate from integration handlers that call payment gateways or send emails. The first can run as many times as you like, the second must be tied to the live path and guarded against duplicates. The boundary is covered in Side Effects and Where the Guarantees Stop.

Take an account that starts empty, with this stream:

v1 AccountOpened(accountId=42)
v2 FundsDeposited(amountMinor=10000)
v3 FundsWithdrawn(amountMinor=2500)
v4 FundsDeposited(amountMinor=1200)

Applying them in order gives 0 + 10,000 − 2,500 + 1,200 = 8,700 minor currency units. The example is deliberately small. A real ledger also has currencies, pending versus settled funds, reversals, fees and double-entry invariants. What carries over is that state is derived from a versioned sequence and is not the only copy of the truth. Drag the slider to see the same history yield any earlier state; stopping at v3 is a point-in-time query that a table with a single balance column cannot answer.

const initial = { opened: false, balanceMinor: 0 };

function apply(state, event) {
  switch (event.type) {
    case 'AccountOpened':  return { ...state, opened: true };
    case 'FundsDeposited': return { ...state, balanceMinor: state.balanceMinor + event.amountMinor };
    case 'FundsWithdrawn': return { ...state, balanceMinor: state.balanceMinor - event.amountMinor };
    default: throw new Error('Unsupported event type: ' + event.type);
  }
}

const stream = [
  { type: 'AccountOpened', accountId: 42 },
  { type: 'FundsDeposited', amountMinor: 10000 },
  { type: 'FundsWithdrawn', amountMinor: 2500 },
  { type: 'FundsDeposited', amountMinor: 1200 },
];
console.log(stream.reduce(apply, initial)); // { opened: true, balanceMinor: 8700 }

This is a teaching reducer, not a banking ledger. It assumes an already validated stream and ignores schema versions, which a later section adds. Throwing on an unknown type is a choice: when loading an aggregate it surfaces a missing handler immediately, whereas silently skipping unknown events would produce wrong state without any signal. A projection that must keep running usually quarantines the event and raises an alarm instead.

Determinism is the contract

The reducer must produce the same state for the same history every time and on every machine. That rules out calling a live pricing service, reading the wall clock, generating random IDs, or consulting mutable configuration while applying an event. Timestamps that matter to the logic belong in the event, written when the decision was made. A reducer that reads "now" will give different answers when replayed next year, and the difference will be invisible until someone compares.

Events are grouped into streams, usually one per aggregate instance such as account-42 or order-42. The aggregate is the consistency boundary: the unit within which a command sees a coherent state and within which events have a strict order. Each event in a stream has a version (1, 2, 3, …), and the stream's current version is its latest event's number.

A minimal relational event store shows the moving parts. This is PostgreSQL syntax and is runnable as a starting point, though it leaves out partitioning, retention and indexes you would add for scale.

CREATE TABLE events (
  global_position bigint      GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  stream_id       text        NOT NULL,
  version         integer     NOT NULL CHECK (version >= 1),
  event_id        uuid        NOT NULL UNIQUE,
  type            text        NOT NULL,
  schema_version  integer     NOT NULL DEFAULT 1,
  data            jsonb       NOT NULL,
  metadata        jsonb       NOT NULL DEFAULT '{}',   -- commandId, correlationId, actor
  recorded_at     timestamptz NOT NULL DEFAULT now(),
  UNIQUE (stream_id, version)
);

The UNIQUE (stream_id, version) constraint is not decoration; it is the mechanism behind optimistic concurrency, described next. global_position gives a store-wide sequence that subscribers can use as a checkpoint.

Order exists per stream, not across them

Within one stream the version gives a total order. Across streams you get, at best, a store-wide position, and that is weaker than it looks. In the table above the identity value is assigned when a row is inserted, not when its transaction commits. Transaction A can take position 1057, transaction B position 1058, and B can commit first. A projector that has read up to 1058 and stored that as its checkpoint will never see 1057 when it finally commits. Systems handle this in several ways: serialising appends so positions commit in order, having readers stop below the oldest in-flight transaction, waiting for gaps to fill within a bounded time, or relying on a store that publishes a gapless commit-ordered log. Whichever you use, projections that span many aggregates must state their cutoff, for example "as of position P", instead of assuming every aggregate is updated atomically.

Two processes can read the same aggregate at version 7 and each decide to append version 8. Take an account at 8,700 and two withdrawals, 6,000 and 5,000. Each is valid against the balance it read, and together they would overdraw it by 2,300. Without concurrency control both commit. With an expected-version check, each append says "add this event only if the stream is still at version 7". One writer wins. The other receives a conflict, reloads, sees a balance of 2,700, and the 5,000 withdrawal is now correctly rejected.

The figure uses an order rather than an account because the second writer's command is not merely reordered but invalidated: adding an item to a cancelled order. That is why the loser must reload and revalidate, not just retry the append with a bumped number. Blindly appending at version 9 would defeat the purpose of the check.

handle(command):
  repeat up to MAX_ATTEMPTS:
    { state, version } = load(command.streamId)         // snapshot + later events
    result = decide(state, command)
    if not result.ok: return rejected(result.reason)
    try:
      append(command.streamId, result.events,
             expectedVersion = version,
             metadata = { commandId: command.id })
      return accepted
    on VersionConflict:
      if stream now contains an event with metadata.commandId == command.id:
        return accepted                                  // an earlier attempt already committed
      continue                                           // reload, revalidate, maybe reject
  return failed("too much contention")

The check must be atomic in the store

"Read the version, compare, then write" as three client-side steps is a race: both writers can pass the comparison before either writes. The comparison and the append have to be one atomic operation in the store. In the relational table above, the unique constraint provides it, and an insert that guards on the current version also gives a clean signal in the non-racing case:

INSERT INTO events (stream_id, version, event_id, type, data, metadata)
SELECT 'order-42', 8, gen_random_uuid(), 'OrderCancelled',
       '{"reason":"customer_request"}', '{"commandId":"c-9917"}'
WHERE (SELECT coalesce(max(version), 0) FROM events WHERE stream_id = 'order-42') = 7;

Stores that expose this as an API typically accept an expected version that is an exact number, "the stream must not exist yet" for the first event, or "any". The last option switches the check off and is only appropriate for events that cannot conflict with each other.

Retries and duplicates

When an append times out the caller cannot tell whether it committed. A retry that carries the same expected version is self-protecting: if the first attempt did commit, the stream is now ahead and the retry conflicts instead of writing a second event. The risk appears when the retry reloads and re-decides, because it will then see the earlier result as ordinary history. Tagging events with the command ID lets the handler recognise its own earlier success, as in the loop above. This is the same principle as request IDs in Idempotency.

Contention and aggregate boundaries

Frequent conflicts on one stream mean many writers want the same aggregate at once. That is a modelling signal: perhaps the aggregate is too large, or a counter-like value should be split. The expected-version check is also scoped to a single stream. An invariant spanning two streams, such as moving money between two accounts, cannot be protected by it, which is where process managers and the Saga Pattern come in.

Loading an aggregate with a million events for every command is wasteful. A snapshot stores derived state at a known event version. To load, read the latest snapshot and replay only the events after it: a snapshot at version 1000 plus events 1001 to 1012 reads far fewer events than replaying all 1,012.

The figure counts events, because that is something we can state exactly. Actual load time depends on payload size, storage latency and deserialisation, and the in-memory fold over simple events is often the cheaper part. How long an unsnapshotted aggregate takes is a number to measure on your own store, not to assume; many aggregates never grow long enough to need snapshots at all.

What a snapshot must be

• Consistent with its stream. It records the stream version it was computed at, so that replay resumes at exactly the next event.
• Versioned. It records the schema of the state and, ideally, the version of the reducer that produced it. When reducer code changes meaning, snapshots from the old code are invalid.
• Disposable. If a snapshot is missing, stale or incompatible, load without it or rebuild it from events. Never put information in a snapshot that exists nowhere else.
• Written off the critical path where possible. A background job or an every-N-events trigger works; the command should not fail because a snapshot write failed.

There is no universal snapshot interval. Fixed event counts are easy to reason about; time-based or size-based triggers fit uneven aggregates better. A different technique for naturally long-lived entities is to bound the stream itself: close an account's monthly stream with a carry-forward event holding the opening balance of the next period, so no stream grows without limit and each one stays small and complete.

A snapshot does not let you delete the events it covers unless you are prepared to lose the ability to reconstruct, audit or rebuild projections from them. Retention is a separate decision, covered next.

Retaining events indefinitely has storage, privacy and legal consequences. If events contain personal data and a person requests erasure, you cannot simply DELETE from a log whose value rests on being complete. There is no single answer; the options below trade completeness, complexity and risk differently, and the legal question of what suffices belongs to counsel, not to the pattern.

• Keep personal data out of events. Store a stable identifier in the event and the personal data in an ordinary mutable store that can be edited or deleted. This is the simplest option and should be the default design when it fits.
• Crypto-shredding. Encrypt the personal fields of a subject's events with a per-subject key held outside the store; destroying the key makes those fields unreadable while the events and their ordering remain. Reducers and projections must tolerate redacted payloads. It does not reach plaintext copies in projections, caches, exports or backups, and whether it satisfies a given regulation is a legal judgement.
• Copy-and-transform migration. Write a new stream with the sensitive data removed or redacted and retire the old one. This is an explicit exception to immutability and needs the same care as a schema migration, including downstream consumers that cached positions.
• Archive after snapshotting. Move old events to cheaper storage once a snapshot or period close exists, accepting that rebuilds from the beginning now need the archive.

"Immutable" describes the logical contract of the history. Storage maintenance, compaction, legally required deletion and a deliberate migration may still touch the physical bytes, and each one should be an explicit, audited procedure and never an ad-hoc UPDATE.

An event written in 2024 must still be readable by code deployed in 2030. Suppose CustomerRegistered version 1 has a single name and version 2 splits it into givenName and familyName. Projection and reducer code must either understand both shapes or transform old events into the current shape while reading them. That transformation is called upcasting. It should be deterministic, version-aware, and tested against real historical samples.

// Splitting a free-text name is a documented heuristic, not a fact: the last token
// becomes familyName. Record whichever rule you choose and keep it stable.
function splitName(name) {
  const parts = name.trim().split(/\s+/);
  const familyName = parts.pop() ?? '';
  return { givenName: parts.join(' '), familyName };
}

const upcasters = {
  'CustomerRegistered:1': (e) => ({
    ...e,
    schemaVersion: 2,
    data: { ...splitName(e.data.name) },
  }),
};

function upcast(event) {
  let e = event;
  while (upcasters[e.type + ':' + e.schemaVersion]) {
    e = upcasters[e.type + ':' + e.schemaVersion](e);
  }
  return e;
}

The example shows a real hazard: version 1 never stored the information needed for a correct split, so any upcaster invents it. That is acceptable only if the invented values are treated as a stated approximation. Where the new shape needs information the old events never captured, do not pretend otherwise; either allow the field to be absent or fill it from a documented enrichment source and say so.

Habits that keep history readable

• Prefer additive changes: new optional fields with defaults. Never reuse an event type name for a different meaning.
• Map stored type names to code explicitly through a registry. Renaming a class from MoneyWithdrawn to FundsDeducted must not make every historical event undeserialisable; the stored name stays, and the registry or an upcaster maps it.
• Store a schema_version with each event, as in the table above, so upcasters can be chained (v1→v2→v3) instead of each one knowing every older shape.
• Keep old versions of sample events as test fixtures and replay them through the current code on every build.

Avoid rewriting stored events in place. It destroys the audit property that justified the pattern, and it can invalidate event hashes, positions and the checkpoints of downstream consumers. When a real correction is needed, see Correcting Mistakes Without Rewriting History. When a bulk transformation is unavoidable, such as splitting a stream, do it as a deliberate copy-and-transform migration into a new stream with its own versioned cutover.

Two different mistakes look alike and need different repairs. A wrong projection is a bug in how events are interpreted; the events are right, so the fix is code and a rebuild, described in the incident section below. Wrong events are a bug in how decisions were made: a pricing error that wrote 5,000 incorrect FundsDeposited events over two hours. Those events are history, and the history is that the system did record them.

The repair is to append new events that correct the outcome, such as DepositCorrected or FundsWithdrawn with a reason that links back to the original, and then to rebuild every projection that was affected. The record then shows both the error and its correction, which is what an auditor wants. If the correction triggers real-world effects, a refund or an email, those go through the normal live handlers with their own idempotency keys. They are new decisions and they are not a replay.

Plan for this before the first incident. Define the corrective event types for money and inventory up front, and test event-producing code harder than anything else, because mistakes there are the most expensive to undo: they are permanent, they propagate to every projection, and they may already have driven external actions.

An event stream can feed many read models at once: a current-status table for each order, a search index, a finance report, an operations dashboard. Each is a projection, a function from the event history to some queryable shape, and each can be rebuilt by replay as long as the full history it needs exists. This pairs naturally with CQRS, but it is a separate decision; see Event Sourcing and CQRS. The read-side design itself, including lag and read-your-writes techniques, is the subject of the CQRS lesson. This section covers what the event stream demands of a projector.

Projectors see duplicates, gaps, and old shapes

A projector tails the store from a checkpoint. Delivery is normally at-least-once: after a crash it resumes from its last saved position and may reapply a few events. So a projection must be idempotent, and it must be able to detect a gap, an event whose predecessor in the same stream it has not seen. Keeping the last-applied stream version in the read row, and writing it in the same transaction as the change, solves both:

-- Applying v4 FundsDeposited(1200) for account 42
UPDATE account_overview
SET    balance_minor = balance_minor + 1200,
       last_version  = 4
WHERE  account_id = 42 AND last_version = 3;

-- 1 row  → applied.
-- 0 rows → either last_version >= 4 (duplicate: skip) or < 3 (gap: stop and investigate).

Projectors must also upcast old events, tolerate event types they do not care about, and decide what to do with a poison event, one that cannot be applied. Halting preserves order but stalls the projection; quarantining it keeps the projection moving but makes it knowingly incomplete. Either is acceptable if it is chosen deliberately and visible on a dashboard.

Cross-aggregate reports need a stated cutoff

Order is guaranteed within a stream and may not exist across them. A report that sums every account must say how it defines "now": as of a global position, as of a timestamp with allowances for late commits, or as of the slowest projection's checkpoint. Different projections update at different speeds, so two screens built from the same events can legitimately disagree for a short time. See Streams, Versions, and Ordering for why commit order and position order can differ.

Feeding other systems

Publishing events to a broker for other services raises the dual-write problem: committing to the store and publishing are two steps that can fail between each other. Publishing by tailing the committed log sidesteps it, because nothing is published that was not committed, at the price of at-least-once delivery and downstream deduplication. The Transactional Outbox lesson explains the failure and the relay mechanics, and Pub/Sub covers consumer fan-out. Treat the events published to other teams as a deliberate contract, separate from the internal stream shape, so that internal refactors and upcasting do not become breaking changes for them.

The figure below combines the pieces. The stream is the only authoritative element. The reducer turns a snapshot plus a tail of events into current state for the command handler. Projections consume the same stream independently, each with its own checkpoint. The side-effect handler is drawn crossed out in the replay path, because it belongs to the live path only.

Two paths read the stream and should not be confused. Loading an aggregate starts from a snapshot (here at v100) and replays only v101 to v103 to get the state the next command needs. A projection rebuild ignores aggregate snapshots, reads the history from the beginning into a fresh table, then switches to tailing. In both cases the events are only read; nothing in the stream is changed.

Event sourcing is about how writes are stored: as events, with state derived. CQRS is about separating the command model from the query model. You can use either without the other. A system can keep ordinary tables for state and split read and write models, and an event-sourced aggregate can serve its reads by replaying its own stream with no separate query side.

They combine well because an event stream is a convenient source for projections, and the projection mechanics above are exactly what a CQRS read side needs. Choosing both adds the costs of both: eventual consistency between write and read sides, plus event schema permanence and replay tooling. The CQRS lesson owns the argument for and against separating read and write models and the techniques for read-your-writes; this lesson owns the event history.

An audit log records who changed something, when, and perhaps what. It sits beside a conventional state store, may be incomplete for reconstruction, and is not what the application reads to make decisions. An event-sourced system needs events sufficient to reproduce state and enforce domain behaviour, because the application depends on them to work at all.

It is also not change data capture. CDC reads a database's internal change log and describes row-level changes as a by-product of a state-based design; the table remains the truth and the changes rarely carry business intent. And it is not just event-driven architecture, in which services react to events passing through without necessarily persisting them as the source of truth. Any of these may be a better fit than event sourcing, and they can coexist with it.

The relationship does not run the other way either: event sourcing does not by itself give compliance-grade auditing. Who issued the command, tamper evidence (for example hash chaining or write-once storage), retention controls, and the logging of privileged and read access are separate requirements, and some of them conflict with in-place maintenance such as migrations. Say what the audit requirement is first, then check whether the event design meets it.

It helps to list the guarantees precisely. A well-built event store gives atomic append of the events in one call to one stream, a strict order within that stream, and a detectable conflict when concurrent writers collide. A well-built projector adds idempotent application and a rebuild path. That is the extent of it.

• No exactly-once side effects. An integration handler that reacts to PaymentAuthorized by calling a gateway can run twice after a crash. Make the external call idempotent with the event ID as the key, record that it was done, and use the outbox for reliable hand-off. The mechanism is in Idempotency and the Outbox Pattern.
• No atomicity across streams. Two aggregates are two consistency boundaries. Coordinate them with a process manager or a saga, which react to events and issue follow-up commands, and accept the intermediate states.
• No immediate consistency for readers. Projections lag the stream. Whether that is tolerable, and how to hide it, is a read-side question.
• No automatic CQRS, and no automatic audit compliance. See the previous two sections.

The rule that protects you in practice is simple: pure state derivation may run any number of times, anywhere, including during replay; anything that touches the outside world runs on the live path, once per event ID, and is switched off by default in every replay and rebuild tool.

A deployment changes how the reporting projector interprets DiscountApplied, and daily totals start coming out wrong. Because the stream is intact, this is recoverable without touching any event.

1. Stop the faulty projector so wrong data stops accumulating, or leave it running only if readers are already marked as stale.
2. Fix the versioned event handling and add a test with real DiscountApplied events of each schema version.
3. Build a fresh projection into a separate table by replaying from the start of history, then let it catch up to the live tail. New events keep arriving during the rebuild, so the shadow table is only ready once its checkpoint is within a small, known distance of the head.
4. Compare totals and sampled aggregates between old and new, and explain every difference. Some are expected, since the old table was wrong.
5. Switch reads atomically, for example by renaming tables in one transaction (PostgreSQL DDL is transactional, though a rename takes a brief exclusive lock) or by repointing a view, and keep the old table for a while in case the comparison missed something.

Throughout, external side effects must not run. The billing and email handlers are not attached to the rebuild; if they were, replaying a year of discounts would re-bill a year of customers. If a rebuild reveals that an old event lacked a field the corrected calculation needs, do not invent the data during replay. Append a corrective business event, or define a documented enrichment rule, such as reading the value from a reference table as of the event's recorded date, and apply it consistently.

The pattern shifts effort from schema migrations to operations on history. Watch these signals:

• Append conflicts per stream and per command type. Rising conflicts point to hot aggregates or retry storms.
• Stream load time and the events read per load, which tell you when snapshots or period closes are needed.
• Snapshot age: the number of events between the newest snapshot and the stream head.
• Projection lag: head position minus the projector checkpoint, in events and in time.
• Replay throughput, which sets how long a rebuild takes.
• Poison events and quarantined items, with an owner assigned to each.
• Divergence checks between authoritative streams and derived views, for example a nightly recomputation of sampled aggregates compared with the projection.

Replay throughput is worth turning into arithmetic before you need it. As a hypothetical, a store with 500 million events replayed at 20,000 events per second takes 25,000 seconds, a little under seven hours, and the projection stays behind for all of it. If that is unacceptable, the answers are partitioned parallel rebuilds (streams are independent, so they can be split by stream ID), more compact events, or retention designs that keep the replay window bounded. Rebuild procedures should be rehearsed on production-sized data before an incident, including the step where the rebuild catches up to the live tail and the cutover.

Event sourcing earns its cost when the domain needs historical reconstruction, temporal queries, reversible decisions, or several different views derived from a rich history. Ledgers and accounting are the textbook case, since a ledger is already an append-only sequence of facts. Order and payment lifecycles with disputes, trading and booking workflows where "what did we know when we decided?" matters, and domains with frequent business-level corrections all benefit.

The costs are concrete and should be weighed up front:

• Event schemas become permanent, and every change goes through versioning and upcasting.
• Concurrency rules, conflict handling and aggregate boundaries must be designed deliberately.
• The log is not a query engine. Every question that is not "the history of one aggregate" needs a projection, and new questions need new projections and replays.
• Replay, rebuild and snapshot tooling must be built, tested and operated.
• Privacy and deletion need an explicit design, as described above.

For straightforward CRUD, a normal relational table with a carefully designed audit trail is usually better. Sometimes the right scope is partial: event-source the one aggregate where history matters, such as payments, and keep the rest conventional. If the real requirement is only "show me what changed", temporal tables, change data capture or an append-only ledger table beside current state may meet it at a fraction of the complexity.

The read side of this pattern is explained in CQRS. Getting events out of the store reliably is the job of the Outbox Pattern, with fan-out to consumers in Pub/Sub. Work that spans several aggregates is coordinated by the Saga Pattern, and the safe handling of duplicate commands and events rests on Idempotency.