Payment System Architecture Deep Dive

A payment is not one record. Collapsing 'the customer wants to pay' and 'we tried to charge a card' into a single row is the root cause of most correctness bugs in naive payment designs.

Five distinct concepts need to stay separate throughout the architecture: the logical payment (what the customer is trying to pay for), the payment attempt (one interaction with a provider/rail), the provider transaction (the external provider's own record of that attempt), the financial ledger entry (durable accounting evidence of money movement), and the downstream business effects (fulfillment, receipt, notification, entitlement).

Logical Payment

Created once per customer-facing payment intent, identified by a durable idempotency key scoped to the merchant/account. It carries the order/business reference, amount, currency, and an aggregate state that summarizes the payment's overall progress. It does not carry provider-specific mechanics.

Payment Attempt

One attempt is one interaction with one provider for one logical payment. A logical payment can have more than one attempt -- for example, attempt 1 to Provider A fails definitively, attempt 2 to Provider B succeeds. An attempt is never overwritten by a later attempt; each is preserved as history.

See Complete Design -> Logical Payment vs Payment Attempt for the introductory version of this model, and Multi-Provider Routing below for why the split becomes essential once more than one provider is involved.

A payment's state is not a boolean, and it is not even a fixed universal enum -- it is a guarded transition graph whose exact states depend on which optional flows (authorization/capture, refunds) are enabled.

A commonly useful simplified core is CREATED/INITIATED -> PROCESSING/PENDING -> SUCCEEDED/FAILED, with UNKNOWN representing an unresolved ambiguous outcome. This is a good teaching model, not a universal specification every provider must match.

Extending the core model

Depending on scope, additional states are commonly needed: REQUIRES_ACTION (the customer must complete a challenge, e.g. 3-D Secure), AUTHORIZED (funds reserved but not yet captured, only if authorization/capture is modeled), CANCELLED/VOIDED, PARTIALLY_REFUNDED, and REFUNDED. Do not force every provider's actual status vocabulary into exactly these names -- normalize at the provider adapter boundary instead (see Provider Adapter below).

Terminality is not universal

Whether a given state is truly terminal is business- and provider-specific. A SUCCEEDED payment is not always immutable -- it can transition to PARTIALLY_REFUNDED or REFUNDED later. What must never happen is an uncontrolled regression, such as a late-arriving webhook silently moving a payment from SUCCEEDED back to PROCESSING.

Guarded transitions

Every write to payment state should go through an explicit transition table that defines which (current_state -> new_state) pairs are valid, combined with a conditional/versioned update (e.g. UPDATE ... WHERE state = :expected AND version = :expected_version) so that two writers racing on the same payment -- a synchronous response, a webhook, and a reconciliation repair, for instance -- cannot silently overwrite each other with last-write-wins semantics.

Idempotency is the mechanism that lets the same logical request arrive any number of times without producing more than one charge -- but it has to be anchored in durable state, not a cache.

Why a cache alone is insufficient

A common but insufficient pattern is: check a fast key-value store for the idempotency key; if absent, call the provider; then store the result. The failure window: the key is absent, the provider successfully charges, the process crashes before the result is stored, the cache entry (or its lock) expires, the client retries, and the service -- believing the request is new -- may call the provider again. A cache can accelerate lookups, but it cannot be the authoritative correctness boundary.

Durable idempotency record

Use a durable uniqueness constraint, conceptually UNIQUE(merchant/account scope, idempotency_key). The first request atomically creates the logical payment/idempotency record inside a single database transaction. A concurrent or later request presenting the same key finds the existing record instead of creating a new one.

Request fingerprint

Store a fingerprint of the request alongside the key -- amount, currency, order/business reference, operation type, and any other immutable request fields. If the same key is reused with a different fingerprint, reject it as an idempotency-key conflict rather than silently returning the old result for what may be a semantically different request (see Failure & Scale -> Same Idempotency Key, Different Payload).

In-progress requests

If a duplicate request B arrives while the original request A for the same key is still processing, B does not need to block on a distributed lock until A finishes. Valid strategies include: return the current (in-progress) payment state, return 202/PENDING, wait a bounded amount of time and then return pending, or let the client poll/subscribe for completion. The one non-negotiable property is that B must resolve to the same logical payment and never create a second, independent charge.

Retention

Retention is a product and provider contract, not an arbitrary cache TTL. It should be sized to the API contract, the maximum client retry window, payment dispute/recovery requirements, and the provider's own idempotency window. A durable idempotency mapping is commonly retained far longer than a short-lived cache entry -- 'the key expired so this is a new payment' is a decision to make deliberately, not an accepted side effect of a cache eviction policy.

There are two separate idempotency boundaries in this system, and conflating them is a common design mistake.

When the provider supports idempotency, derive or generate a stable provider idempotency key for one payment attempt, and reuse that exact key across any safe retry of that specific attempt. Do not generate a new provider key merely because the HTTP call to the provider timed out -- a new key on retry is what turns a safe retry into a second independent charge.

If the provider does not support idempotency at all, the ambiguous-outcome resolution order in Ambiguous Outcome Resolution (query status, await webhook, reconcile) becomes the primary safety mechanism instead of a safe retry.

The provider adapter is the seam that keeps every other component -- state machine, retry engine, webhook handler -- from needing to know provider-specific details.

Responsibilities: translate the canonical internal request into the specific provider's API shape; generate or reuse the provider idempotency identity for a given attempt; normalize provider-specific statuses into canonical internal states; classify provider errors into the definitive/transient/ambiguous taxonomy (see Error Classification and Retry Engine); query a provider transaction's current status by its stable reference; normalize incoming webhook events into the internal event model; and expose refund/capture/void capabilities where the provider supports them.

Do not assume identical semantics

Different providers differ in: which operations are synchronous vs asynchronous, whether idempotency keys are supported and how long they are honored, what status vocabulary they use, how webhooks are signed and delivered, and what capabilities (partial refund, void, capture) exist at all. The adapter's job is to absorb these differences behind one canonical interface -- not to assume they are interchangeable.

A request to the provider can fail in a way that leaves the actual outcome unknown -- this is the case the architecture has to be built around, not an edge case bolted on afterward.

A timeout or dropped connection can occur at several distinct points, each with a different true outcome: the request never reached the provider; the provider received it but had not yet processed it; the provider processed it and it failed; the provider processed it and it succeeded; the provider's response succeeded but was lost in transit; or the provider accepted the request for asynchronous processing and has not yet resolved it. From the caller's side, several of these look identical.

Safe decision tree

On an ambiguous outcome, do not automatically issue a brand-new payment attempt. Instead, in order: (1) if the provider documents that the same operation can be safely retried using the same provider idempotency key, do so; (2) otherwise, query the provider by the stable provider reference/idempotency identity to ask what actually happened; (3) if neither is immediately possible, await the provider's webhook; (4) if the ambiguity persists past a bounded window, hand it to reconciliation.

The local system marks the attempt as unresolved (a PROCESSING/UNKNOWN-style state, without necessarily needing a state literally named UNKNOWN) rather than FAILED, because marking it FAILED and later discovering the provider had actually succeeded would require reversing a decision that downstream consumers may already have acted on.

Not every provider error is the same kind of error, and treating them identically is what turns a normal decline into a retry storm, or an ambiguous outcome into a duplicate charge.

Retry policy properties

Beyond classification, a production retry policy needs: a retry budget and maximum attempt count; a maximum elapsed time before giving up and escalating to reconciliation; respect for a provider's Retry-After header where present; exponential backoff with jitter to avoid synchronized retry storms across service instances; and integration with the error classification above so definitive business declines are never retried.

Circuit breaker

A circuit breaker is a useful resilience pattern for protecting the service and the provider from a pile of repeated technical failures -- it is not itself a payment-correctness mechanism, and it does not resolve ambiguous outcomes that already occurred. Opening a circuit after a run of timeouts tells you nothing about whether payments already sent during that window actually succeeded at the provider. Circuit breaker state is commonly scoped per provider, per operation, and sometimes per region/endpoint or payment method/rail, since a technical failure on one provider or operation should not necessarily halt traffic to another.

The database enforces the invariants that keep the payment and attempt model correct under concurrent writers.

Money representation

Never represent monetary amounts using binary floating point. Use integer minor units (e.g. cents) where the currency model permits, or a fixed-precision decimal type. Store the currency explicitly alongside the amount, and do not assume every currency uses two decimal places (some use zero, some use three).

Constraints as invariants

Push as many invariants as possible into database constraints rather than only application logic: a unique idempotency key within its merchant/account scope; a unique provider transaction/reference where the provider guarantees one; non-negative amounts and refund-total bounds; currency immutability once a provider attempt exists for that payment; and a guarded state/version column.

Optimistic concurrency

Webhooks, synchronous provider responses, and reconciliation repairs can all attempt to update the same payment concurrently. Use a row version and conditional updates -- UPDATE payments SET state = :new, version = :version + 1 WHERE id = :id AND version = :expected_version -- wrapped in a transaction around any related financial state change. Do not allow a blind last-write-wins update to decide the outcome of a race between these three writers.

Transaction boundaries

Keep database transactions narrowly scoped around the durable state change itself -- do not hold a long-running transaction open while an arbitrary external provider call or business handler executes, since that both blocks other writers and holds locks across an operation whose latency you do not control.

Publishing a payment.success event and updating the payment's database state are two separate operations against two separate systems -- and that gap is a real source of lost or duplicated events if left unaddressed.

Consider the naive sequence: write SUCCESS to the database, then publish payment.success to the event queue. Two failure cases: (Case A) the database commits SUCCESS but the service crashes before the publish call -- the event is silently never sent, and order/notification/ledger consumers never see it; (Case B) the publish call succeeds but the process retries the publish anyway (e.g. due to an ambiguous response from the broker), producing a duplicate event.

The pattern

In the same database transaction that updates payment state, also insert a durable outbox row representing the event to be published. A separate relay process reads unpublished outbox rows and publishes them to the broker, marking them published once the broker acknowledges receipt. Because the relay can crash between publishing and marking a row published, it may republish the same event on restart -- so this pattern only closes the durability gap; it does not eliminate the need for idempotent consumers, since a republished event can reach them more than once.

See Trade-offs & Decisions -> Direct Event Publish vs Transactional Outbox for when the added relay/outbox infrastructure is and isn't justified.

The webhook handler is where an external, untrusted, at-least-once, possibly-out-of-order event stream meets durable financial state -- every one of those properties needs explicit handling.

Verify before processing

Before any state mutation: verify the request signature/MAC using the provider's signing secret; verify the timestamp against a replay window if the provider supports one; verify the endpoint secret/version matches what was configured for this integration; and verify payload integrity. Where the provider's signature scheme requires it, verify against the exact raw request body -- not a re-serialized version of the parsed payload, which can differ byte-for-byte from what was signed.

Persist before ACK

Preferred sequence: authenticate the webhook; durably persist the webhook event and its provider delivery ID; ACK the provider quickly; then process the event asynchronously and idempotently. This prevents slow downstream business processing from causing the provider to time out and redeliver the same webhook, which would otherwise compound the deduplication problem.

Deduplication

Use the provider's event/delivery ID for deduplication where available -- do not assume duplicate webhook payloads for the same event will always be byte-identical or arrive in immediate succession. Store, per received webhook: the provider event ID, the provider transaction ID, the event type, an event timestamp/version if the provider's ordering signal is reliable, the local received_at time, and a processing status.

Provider webhooks are not guaranteed to arrive in the order the underlying events occurred.

Concrete example: a SUCCESS webhook is processed and the payment reaches its terminal state. An older PROCESSING webhook for the same payment -- generated before the SUCCESS event but delayed in delivery -- arrives afterward. Applying it naively would regress the payment from SUCCESS back to PROCESSING.

How this is prevented

The guarded state machine from Payment State Machine is what makes this safe: every incoming event is checked against the valid-transition table before being applied, so an event proposing an invalid transition (SUCCESS -> PROCESSING) is rejected rather than applied. Where the provider supplies a reliable sequence number or version, use it to detect and discard stale events even when the proposed transition would otherwise look superficially valid. Where ordering is genuinely ambiguous and the stakes are high enough, query the provider's status endpoint directly rather than trusting event order alone.

The mutable payments.state row and the financial ledger answer two different questions, and treating one as a substitute for the other is a common gap in simpler designs.

Payment state answers: where is this workflow right now? Ledger answers: what money movement has actually been recorded? The payments table is useful, current, mutable operational state -- it is not the entire financial record, and it is not append-only.

What the ledger records

An append-only sequence of monetary movement entries: charge/capture, refund, fee (if fees are in scope), and reversal/adjustment. Each entry references the payment and attempt it originated from, so the ledger can always be traced back to the workflow event that produced it.

Double-entry, briefly

For deeper financial correctness, a double-entry model represents every monetary movement as balanced entries between accounts -- money debited from one account is credited to another, and the books always sum to zero across a transaction. This module introduces the idea because it explains why a ledger is structured the way it is; it does not attempt to teach a complete accounting system.

Duplicate-safe posting

Because the outbox and downstream event delivery are at-least-once (see Transactional Outbox), the ledger-posting consumer must be idempotent -- posting the same charge/capture/refund event twice must not create two ledger entries. A stable idempotency key derived from the event (e.g. payment_id + attempt_id + operation type) is what makes a duplicate post a safe no-op instead of a duplicated financial entry.

Reconciliation is the safety net that catches divergence that webhooks, polling, and synchronous responses did not resolve -- and it covers considerably more ground than sweeping payments stuck in an unresolved state.

Scope

Beyond unresolved (PROCESSING/UNKNOWN-style) payments, reconciliation should also be able to detect: successful provider transactions with no matching internal record; internal SUCCESS states with no corresponding provider evidence; amount or currency mismatches between internal and provider records; refund mismatches; duplicate provider transactions; and gaps in webhook delivery inferred from a provider settlement export.

Operational vs financial reconciliation

Reconciliation must be idempotent

Running the same reconciliation batch twice must not double-create ledger entries, emit duplicate business effects, or issue duplicate refunds. Every repair action needs a stable reconciliation/action ID so a re-run of the same batch recognizes an already-applied repair as a no-op rather than repeating it.

A refund is a new financial operation with its own identity and its own failure modes -- it is not simply flipping a payment's state to FAILED or REFUNDED.

Model

A refund needs its own idempotency key/request ID, a reference to the original payment, an amount, a provider refund reference, a state, and its own attempt history -- structurally similar to how a payment attempt relates to a logical payment.

Supported shapes

Full refund; partial refund, if the product allows it; multiple partial refunds against one payment, enforced against a total constraint; refund timeout/ambiguity, handled with the exact same resolution order as a charge timeout (safe retry with the same provider refund identity, status query, webhook, reconciliation); and refund webhook/reconciliation coverage matching what charges already have.

Invariant

sum(successful refunds) <= captured amount, enforced transactionally at the database level (not just checked in application code before the call), unless the specific domain explicitly supports additional adjustments beyond that bound.

Separating authorize (reserve funds) from capture (actually move the money) is a bounded, optional extension -- not a requirement of every payment system.

Useful when there's a meaningful gap between confirming a customer can pay and knowing the final chargeable amount or fulfillment outcome: inventory confirmation before capture, hotel/rental holds, or order fulfillment delay where the exact shipped amount isn't known at authorization time.

States

Where this flow is modeled, typical additional states are AUTHORIZED (funds reserved, not yet captured), CAPTURED (money actually moved), VOIDED (authorization released without capture), and EXPIRED (the authorization hold itself has a provider-defined time limit and lapsed before capture).

Adding a second provider is a real availability lever, but the most important rule is about what you must not do when the first attempt is ambiguous.

The failover trap

Sequence: request sent to Provider A; the call times out; Provider A may or may not have actually charged the customer; the router, seeing a failure, immediately routes the same logical payment to Provider B. If A did in fact charge successfully, the customer is now charged twice -- once by each provider, with each provider having no visibility into the other.

The rule

Only fail over to a different provider for a given logical payment when the first attempt is known not to have created a charge (a definitive failure, not an ambiguous one), or when the business/provider architecture has an explicit, safe orchestration or reconciliation mechanism specifically designed to detect and resolve a double-charge across providers before it reaches the customer.

Maintain a separate payment attempt per provider (see Payment Intent and Attempt Model) -- this is what makes it possible to later reconcile 'attempt 1 failed at A, attempt 2 succeeded at B' from 'both attempts independently succeeded.'

Provider capability differences

Providers differ in supported operations, idempotency semantics, status vocabulary, and refund/capture capability. Routing decisions should account for these differences rather than assuming any provider can be substituted transparently for any other mid-flow.

Only the architecturally necessary security concerns -- not a full compliance tutorial.

What to measure, separated deliberately so that a business decline never gets counted as an infrastructure failure.

Failure mode to mechanism