Designing a Payment System

We are designing a payment system where a customer initiates a payment, and for one logical payment we must prevent accidental duplicate charges and converge to the correct financial outcome despite retries, timeouts and duplicate messages -- even when the network is unreliable and providers time out.

The deceptive simplicity of 'charge a customer' conceals one of the hardest problems in distributed systems engineering. A payment involves at least three parties: the customer, your system, and an external payment provider. Any of the links between them can fail at any point. Your system must make progress without double-charging under any failure combination.

Functional and non-functional requirements -- with emphasis on what makes payment systems different from typical web services, and an explicit note on what's in scope.

Scope

Baseline scope: one-time payment initiation, provider interaction, asynchronous confirmation, payment status, safe retry, webhook handling, reconciliation, refund, a financial ledger/audit trail, and downstream payment events. Optional/advanced extensions covered later: authorization + capture, multiple providers, multi-region, partial refunds, and payment-method tokenization. Out of scope beyond a brief mention: card-network internals, full acquiring/issuing bank architecture, a PCI compliance tutorial, a fraud/ML system, subscription billing, invoicing, a tax engine, an FX engine, and a complete double-entry accounting course.

Functional

Create/initiate a payment. Retrieve payment status. Retry safely without creating a duplicate charge. Receive asynchronous provider updates. Reconcile ambiguous payments. Refund a successful payment. View payment history. Optionally cancel/void before final capture, if the chosen model supports authorization/capture.

Non-functional

Capacity estimate (illustrative)

These are illustrative planning numbers, not measurements of any real system. Assumptions: a mid-sized marketplace processes 10 million payments/day; peak-to-average factor 5; one provider call per payment; two provider webhooks per payment (for example processing, then final status); about 8 database row writes per payment (payment, attempt, idempotency record, two ledger entries, two outbox rows, one transition-history row); about 1 KB of durable data per payment across those rows; idempotency keys retained 7 days at about 200 bytes each; a provider account limit of 1,500 requests/s; and a primary database sustaining about 20,000 row writes/s. Day = 86,400 s.

Assumed targets (illustrative SLOs): create-payment p99 latency of 3 s including one provider call with a 2 s timeout; ambiguous (timed-out) payments resolved by reconciliation within 15 minutes; RPO of zero for committed payment and ledger state; RTO of 5 minutes for the payment API on regional failover. What the numbers imply for this design: a single relational primary is a reasonable baseline at this volume, so sharding is a deliberate later choice rather than a default; the provider limit (about 1,500/s), not the database (about 2,500/s), is the first throughput ceiling in a spike; and webhook ingestion at about 1,160/s needs the persist-then-process-asynchronously split described in Scaling. The Failure & Scale Scenarios page traces a spike against these same numbers.

Start with the simplest possible architecture: Customer → API → Payment Service → Provider → Database.

The precise order matters: (1) validate the request; (2) durably create or reuse the logical payment for this idempotency key; (3) create a payment attempt; (4) call the provider using a stable provider idempotency/reference identity; (5) persist whatever result is actually known; (6) return a confirmed, pending, or unresolved response to the caller -- not a blind guess; (7) if the outcome was ambiguous, resolve it asynchronously via retry/status-query/webhook/reconciliation.

Payment state must be persisted to the database before calling the provider and after receiving a response. If the service crashes between these steps, the persisted state lets us reason about what happened.

A payment is not a boolean. It has explicit states that represent what actually happened -- including intermediate and unresolved states.

Logical payment vs payment attempt

One logical payment can have more than one payment attempt. Example: payment_123 has attempt_1 to Provider A, which fails definitively, and attempt_2 to Provider B, which succeeds. This is different from retrying the same ambiguous attempt -- a genuinely new attempt gets its own identity and its own provider reference, and the data model never overwrites earlier attempt history. This distinction becomes essential once multiple providers or reconciliation are involved -- see Architecture Deep Dive -> Payment Intent and Attempt Model.

A customer clicks Pay twice. Or a client automatically retries a timed-out request. Without protection, both requests produce separate charges.

This is not a theoretical edge case. In production systems, duplicate payments are one of the most common causes of customer support escalations and refund costs. Network timeouts, user double-clicks, client retries, and message delivery-at-least-once semantics all produce duplicate requests.

An idempotency key is a client-generated unique identifier attached to every payment request. The same key submitted multiple times must resolve to the same logical payment -- not necessarily by blocking on a lock, but by never creating a second, independent charge.

A commonly seen but insufficient pattern checks a fast cache for the key, calls the provider if absent, and stores the result -- guarded by a distributed lock. The failure window: the key is absent, the provider successfully charges, the process crashes before the cache write, the lock expires, the client retries, and the service -- believing the request is new -- may call the provider again. A cache can accelerate lookups; it should not be the authoritative correctness mechanism.

The provider processes the payment. The response is lost on the network. The service times out. Did the payment succeed?

A timeout means 'no response was received' -- not 'the operation failed.' The ambiguity can arise at several distinct points: the request never reached the provider; the provider received it but hadn't processed it yet; the provider processed it and it failed; the provider processed it and it succeeded; the success response was lost in transit; or the provider accepted it for asynchronous processing. The local system frequently cannot tell which of these actually happened.

The correct immediate response to a timeout: record the attempt as unresolved, keep the same provider idempotency identity for any safe retry, and let the resolution mechanisms above determine the actual outcome. See Architecture Deep Dive -> Ambiguous Outcome Resolution for the full decision tree.

After a timeout or dropped response, the internal system and the provider can temporarily disagree about the payment's state. The architecture must converge them.

Two mechanisms converge divergent state: webhook callbacks (provider pushes the result when it becomes available) and active status polling (Payment Service periodically asks the provider for the current status of pending payments).

Payment success triggers downstream operations -- order fulfillment, notifications, ledger entries. These must not block payment confirmation, and the hand-off between the payment database and the event queue has its own failure window worth naming explicitly.

When a payment succeeds, the service publishes a payment.success event to a queue, and workers consume it to trigger downstream operations independently. If the email service is down, the payment still confirms -- the email is retried later.

The DB-to-queue dual-write problem

Writing SUCCESS to the database and then publishing to the queue are two separate operations. If the database commits and the service crashes before publishing, downstream consumers never see the event. If the publish call is retried after an ambiguous broker response, consumers may see duplicates.

The payment provider becomes unavailable, or starts returning errors. The system must respond with conservatism, and must first tell the difference between a definitive failure and a transient one.

A reconciliation process compares internal payment records against provider records, finds discrepancies, and repairs them -- covering considerably more ground than payments stuck in an unresolved state.

Reconciliation handles: unresolved payments that never received a webhook, SUCCESS/FAILED mismatches, delayed callbacks, successful provider transactions missing internally, internal SUCCESS states missing provider evidence, refund mismatches, amount/currency mismatches, duplicate provider transactions, and webhook gaps.

Every discrepancy found by reconciliation is logged -- not just the repair, but the fact a repair was needed, when it was made, and what the original and corrected states were.

Every state transition is permanently recorded in an immutable audit log -- and that audit log is a distinct concept from the financial ledger, even though both are typically append-only.

The audit log records: payment ID, amount, currency, provider reference ID, every state transition with timestamps, who or what triggered the transition, and the final outcome. It is append-only -- no modification, no deletion.

Payment state vs financial ledger

The mutable payments.current_state row is useful operational state -- it answers 'where is this workflow right now?' It is not the entire financial record. A financial ledger is a separate, append-only account of monetary movements -- charge/capture, refund, fee, reversal/adjustment -- that answers 'what money movement has actually been recorded?' An audit log tracking who changed what state is not automatically a financial ledger tracking money movements; the two serve different purposes even where both are append-only. See Architecture Deep Dive -> Financial Ledger for the full model, including the idea of balanced double-entry postings.

Scale after correctness is established -- never sacrifice correctness guarantees for throughput.

The complete system — every component earns its place by solving a specific demonstrated failure mode.