Payment System Trade-offs & Architecture Decisions

Decision 01

Synchronous vs Asynchronous Confirmation

The precise requirement is not that 'the charge itself must be synchronous' -- some payment methods and providers are inherently asynchronous, and the architecture has to model both shapes rather than assume one.

	Synchronous provider outcome	Asynchronous provider outcome
What happens	Provider returns a definitive success/failure in the request/response path	API returns pending; the final outcome arrives later via webhook or polling
Examples	Many card charges	Bank redirects, some local payment methods, 3-D Secure challenges
What the internal API must do	Model a fast terminal response	Model a PENDING/REQUIRES_ACTION response and a separate resolution path

The internal API needs to represent both shapes, because which one applies is a property of the provider and payment method, not a choice the architecture gets to make. What is a genuine design decision is the layer downstream of payment confirmation: downstream business effects (order fulfillment, notification, ledger posting) should generally be event-driven, triggered off durable payment state rather than the request/response path of the original call -- regardless of whether that call was itself synchronous or asynchronous.

Decision 02

Retry vs Reconcile

When a provider outcome is ambiguous, 'retry with backoff' is not the only option, and often is not the first one to reach for.

Approach	When it's the right call
Retry the same provider operation, same provider idempotency key	The provider explicitly documents this operation as safely retryable with that key
Query provider status by stable reference	A status-query endpoint exists and is cheaper/safer than a retry
Wait for webhook	The provider's webhook delivery is reliable enough to trust within the SLO window
Reconcile later	The ambiguity has persisted past the bounded window for the above three, or none of them is available

These are not mutually exclusive -- a real implementation typically layers them: attempt a safe retry or status query immediately, fall back to waiting for the webhook, and use reconciliation as the backstop that catches whatever the first three didn't resolve in time. See Architecture Deep Dive -> Ambiguous Outcome Resolution for the full decision tree.

Decision 03

Durable DB Idempotency vs Redis Cache

These solve different problems and are not substitutes for each other.

Swipe to pan

Redis accelerates the common idempotency-key lookup, but only the Payment DB's unique constraint is authoritative -- a Redis-only design loses the dedup signal entirely on a cache miss, risking a duplicate charge rather than just a slower lookup.

	Durable DB idempotency record	Redis/cache-based lookup
Correctness role	Authoritative -- a unique constraint guarantees only one logical payment is ever created per key	Accelerator only -- speeds up the common case, but a cache miss or eviction must never be interpreted as 'this key is new'
Survives a crash between provider success and result storage?	Yes -- the durable record and the payment's state live in the same transactional store	No -- a crash before the cache write leaves no durable trace, exactly the failure window described in Architecture Deep Dive -> Durable Idempotency
Failure mode if unavailable	Payment correctness is unaffected -- the DB path is unaffected by cache downtime	Must fall back to the durable DB check, never fall through to allowing a duplicate provider call -- see Failure & Scale -> Idempotency Store/Redis Failure

Decision 04

Strong Local Transaction vs Distributed Consistency

Local database invariants and external provider state need fundamentally different consistency treatment -- and blurring that line is a common source of overclaimed guarantees.

What can be made strongly consistent with an ordinary local database transaction: creation of one logical payment per idempotency key; guarded state transitions; financial ledger writes; durable outbox/event creation; and refund amount constraints. All of these live entirely inside the local database and can be wrapped in one ACID transaction.

What cannot: the provider's own charge status. It is external state, owned and mutated by a separate system, and a local database transaction has no way to reach into it. The local system does not make provider state 'strongly consistent' -- it converges local and provider state over time, through the provider's synchronous response, webhooks, status queries, and reconciliation.

Decision 05

Single vs Multiple Providers

A second provider is a real lever for availability -- but only if the failover logic is built to avoid the ambiguous-attempt double-charge trap.

	Single provider	Multiple providers
Simplicity	One integration, one set of API semantics, one reconciliation format	Routing logic, per-provider semantics, multiple reconciliation formats, more complex attempt tracking
Availability	A provider outage is a payment outage	Can route around an outage -- but only if failover safely distinguishes definitive failure from ambiguous outcome
Risk if handled naively	N/A	Failing over an ambiguous attempt can double-charge the customer -- see Failure & Scale -> Multi-Provider Failover Double Charge

Decision 06

Direct Event Publish vs Transactional Outbox

Publishing directly from the request path is simpler to build, but leaves a real dual-write gap that the outbox pattern exists to close.

	Direct publish	Transactional outbox
Implementation cost	Lower -- one call to the broker after the DB commit	Higher -- an outbox table, a relay process, and monitoring for relay lag
Failure window	A crash between the DB commit and the publish call silently drops the event; a retried publish can duplicate it	The event's existence is guaranteed once the DB transaction commits, because it's written in the same transaction
Consumer requirement either way	Consumers must still be idempotent	Consumers must still be idempotent -- the outbox guarantees at-least-once delivery, not exactly-once

For anything a downstream consumer treats as financially or operationally significant (order fulfillment, ledger posting, notifications that matter), the transactional outbox is worth its added operational cost. See Architecture Deep Dive -> Transactional Outbox for the mechanics.

Decision 07

Payment State Table vs Ledger

These are complementary, not substitutes for one another -- a common design gap is building only the state table and treating it as sufficient financial record-keeping.

	Payment state table	Financial ledger
Answers	Where is this workflow right now?	What money movement has actually been recorded?
Mutability	Mutable -- current_state changes as the workflow progresses	Append-only -- entries are never modified after being written
Used for	Operational decisions: is this payment done, pending, or needs attention?	Accounting, reconciliation against provider settlement, financial reporting

Decision 08

Provider-Hosted Tokenization vs Handling Card Data

Where raw payment instrument data touches your infrastructure is a scope decision with real compliance and risk consequences.

	Provider-hosted / tokenized	Handling raw card data directly
What you store	A provider token/reference standing in for the instrument	Raw PAN (stored unreadable, only in a system built and assessed for PCI DSS scope); the CVV generally must not be retained after authorization
Compliance scope	Substantially narrower, though not zero -- tokenization reduces PCI scope, it does not eliminate every obligation	Full PCI DSS scope for whatever systems touch the raw data
Default recommendation	Default choice for the vast majority of payment system designs	Only when the product and compliance posture explicitly require it

Decision 09

Authorization/Capture vs Immediate Charge

Whether to separate authorize from capture is a product-fulfillment decision, not a default best practice.

	Immediate charge	Authorize then capture
Complexity	Lower -- one state transition per successful attempt	Higher -- AUTHORIZED/CAPTURED/VOIDED/EXPIRED states, capture timing logic, auth-expiry handling
Fits well when	Fulfillment happens immediately or the exact amount is known upfront	There's a meaningful gap between confirming payment and knowing final fulfillment/amount (inventory checks, hotel/rental holds, delayed shipping)
Risk if skipped when needed	Capturing more than was actually fulfillable, or charging before confirming availability	N/A

Decision 10

Fail Closed vs Queue for Later

During a provider outage, the system has to choose what to tell the customer -- and one of the two options is dangerous if applied by default.

	Fail closed	Accept intent for later
Behavior	Tell the caller the payment cannot currently be completed	Accept the request and queue it for execution once the provider recovers
When it's safe	Always a safe default	Only if product semantics explicitly allow asynchronous future execution, and the customer is clearly told no payment has succeeded yet
The trap	N/A	Silently queuing a charge to happen later while implying it already succeeded -- this misrepresents the true state of the payment to the customer

Decision 11

Single Database vs Sharded Payment Store

Sharding a payment store is not a casual 'shard by customer ID or payment ID' decision -- the chosen key ripples through nearly every other component.

A payment sharding key affects: idempotency uniqueness scope, merchant-facing queries, reconciliation, order lookup, ledger/accounting aggregation, and refund routing. Common partition keys are merchant/account, payment ID, or region -- each with different implications for which queries stay single-shard and which become cross-shard.

	Single (or vertically scaled) database	Sharded payment store
Complexity	Lower -- no cross-shard transaction concerns	Higher -- cross-shard queries, potentially cross-shard reconciliation
When appropriate	Financial systems often prefer scaling a strongly consistent database vertically or with careful partitioning before introducing cross-shard transaction complexity	Once a specific, measured bottleneck (not a fixed volume threshold) justifies the added complexity

Decision 12

Webhook Processing Inline vs Async

Whether business processing happens inside the webhook request handler or is handed off to be processed asynchronously is a real latency-vs-simplicity trade-off.

	Inline processing	Persist-then-async processing
Provider-facing latency	Depends on how long business processing takes -- risks provider timeout and redelivery on slow paths	Fast -- ACK happens right after durable persistence, independent of downstream processing time
Simplicity	Simpler -- no separate queue/worker	Requires a queue/worker and monitoring for processing lag
Recommended default	Only for genuinely trivial, fast processing	Preferred pattern -- verify, persist durably, ACK quickly, process asynchronously and idempotently (see Architecture Deep Dive -> Webhook Ingestion)