Failure Scenarios & Scaling

The second request carries the same idempotency key as the first, which already completed. Resolve it against the durable idempotency record and return the same result -- no second provider call.

The second request finds an in-progress record for the same key. It resolves to the same logical payment and returns/waits for the current state -- it does not need to block on a distributed lock, and it must not proceed to an independent provider call.

Reject as an idempotency-key conflict. Do not silently execute the new amount and do not silently return the old result -- both would misrepresent what the caller actually asked for.

If the same attempt genuinely needs to be retried at the provider, reuse that attempt's provider idempotency key rather than generating a new one.

A request to the provider times out or the connection drops after the request was sent.

A blind retry with a new provider identity can double-charge if the original request actually succeeded; blindly marking it FAILED can cause a missed charge to go undetected if the provider actually failed and the customer needed to be informed.

Do not mark it FAILED. Do not generate a fresh, independently-identified attempt. Resolve it via the ambiguous-outcome decision tree: same-identity retry if the provider documents it as safe, status query by stable reference, await webhook, or hand off to reconciliation.

Resolution typically completes once one of the above channels reports a definitive outcome -- how long that takes is provider- and operation-specific, not a fixed universal window.

Use a provider whose API supports a status-query-by-reference endpoint wherever possible, so ambiguity resolution doesn't depend on webhooks alone.

PROCESSING is a valid, non-error intermediate state while a webhook is in flight. The payment is not resubmitted. If the delay exceeds the reconciliation threshold, reconciliation queries the provider directly rather than waiting indefinitely.

The same webhook event arrives more than once. Deduplicate on the provider's event/delivery ID -- reprocessing an already-applied event is a safe no-op.

An older event (e.g. PROCESSING) arrives after a newer one (e.g. SUCCESS) was already applied. The guarded transition table rejects the invalid SUCCESS -> PROCESSING regression.

The transition table and idempotent processing apply equally here -- an event proposing a transition that's no longer valid given the payment's current state is rejected, not blindly applied.

No ambiguity -- the operation definitively did not reach the provider. Safe to fail immediately or retry the whole operation.

The provider explicitly rejected or failed to accept the operation -- treat as a transient technical failure eligible for retry per the retry policy, not as an ambiguous outcome.

This is the ambiguous case -- see Provider Timeout above. A circuit breaker prevents new traffic from piling onto a failing provider, but it cannot resolve whether operations already in flight when it opened actually succeeded.

Reject cleanly -- no provider call, a clear error to the client. This is the one absolute rule: never call the provider before a durable local identity exists for the attempt.

Highly undesirable, but not automatically unrecoverable -- if a stable merchant reference or provider idempotency identity was sent to the provider before the crash, the provider's own records, a subsequent webhook, or financial settlement reconciliation may allow recovery. See Failure & Scale -> Crash After Provider Success Before Local Commit for the full walkthrough.

The durable idempotency record already reflects the correct outcome -- a client retry with the same key resolves to it rather than creating a new charge.

A read from a lagging replica must never be used to decide whether to charge -- only the authoritative, sufficiently fresh store can make that call. See Trade-offs & Decisions -> Single Database vs Sharded Payment Store.

The provider charges the customer and returns success, but the service process crashes before writing SUCCESS (or even before writing PROCESSING) to the local database.

The provider believes the charge succeeded. If the local system loses track of the attempt entirely, a client retry with a fresh identity could produce a second charge -- or the successful charge could go completely unrecorded.

A stable provider idempotency/reference identity, generated before the provider call, is what makes this detectable -- either via the provider's webhook, a status query against that reference, or reconciliation against provider records.

The retry path resolves via the same provider idempotency identity, not a new one. The local state remains PROCESSING/unresolved until one of the resolution channels reports the true outcome -- it is never blindly marked FAILED or blindly assumed SUCCESS.

Webhook, status query, or reconciliation eventually reports the true outcome and the local record is updated through a guarded transition.

Always create the durable local attempt record with its provider idempotency identity before calling the provider -- so recovery has something stable to query against, even if the write of the result is what got lost.

Local state is durably SUCCESS. The client, having seen no response, experiences this exactly like any other timeout.

If the client's retry is treated as a brand-new request, it risks a second charge -- even though the original charge already succeeded and is durably recorded.

The client retries using the same idempotency key.

The durable idempotency record resolves the retry to the existing, already-SUCCEEDED payment and returns that result -- no new provider call is made.

No recovery action needed beyond the idempotency resolution itself -- this is exactly the case durable idempotency is designed to make a non-event.

Ensure the client always retries with the same idempotency key it used originally, and that the durable record (not a cache) is what the retry path checks first.

Redis (or an equivalent acceleration cache) used in front of the durable idempotency record becomes unavailable.

None, if the design is correct -- the risk only appears if the cache was mistakenly treated as the authoritative source of truth.

Cache health checks / connection errors from the idempotency lookup path.

Fall back to the durable database uniqueness check, and never fall through to allowing a duplicate provider call. The database check is slower without the cache, but it is still correct.

Once the cache recovers, it resumes accelerating lookups; no data reconciliation is needed because the cache was never the source of truth.

Never design the idempotency path such that a cache outage removes the durable database check -- the cache should only ever be a read-through accelerator in front of it.

A request arrives with an idempotency key that already maps to an existing payment, but the amount, currency, or business reference differs from the original request's stored fingerprint.

Silently executing the new request would create an unexpected charge under a key the client believes maps to the original request. Silently returning the old result would silently ignore a request the client may have genuinely intended to be different.

Comparing the incoming request's fingerprint against the stored fingerprint for that key.

Reject with an explicit idempotency-key conflict error -- do not execute the new parameters and do not silently return the stale result.

The client is expected to generate a new idempotency key for what is, in fact, a new logical payment.

Track idempotency-key-conflict rate as an observability metric -- a sustained rate often indicates a client-side bug generating non-unique keys.

The same provider event (same event/delivery ID) is delivered two or more times.

None, if deduplicated correctly. If the handler blindly reapplies every delivery, a duplicate SUCCESS event could double-post to the ledger or re-trigger downstream effects.

Deduplicate on the provider's event/delivery ID, recorded at persistence time.

The second and subsequent deliveries are recognized as already-processed and are acknowledged without reapplying the business effect.

None needed -- this is the expected steady-state behavior of a correctly built handler.

Persist the event ID durably before ACKing, and always check it before applying any state change or side effect.

An older event (e.g. PROCESSING, generated before a SUCCESS event) is delivered after the newer SUCCESS event has already been applied.

Applying the stale event naively would regress a completed payment back to a non-terminal state, potentially re-triggering downstream effects that assumed the payment was done.

The guarded transition table recognizes SUCCESS -> PROCESSING as an invalid transition for the current state; a reliable provider sequence/version number, where available, gives an additional signal.

The stale event is rejected/ignored rather than applied -- the payment's current state is left untouched.

None needed beyond logging the rejected transition for observability.

Never apply a webhook event without checking it against the current state's valid-transition table first; use provider sequence/version fields when the provider offers them.

An inbound webhook request fails signature/MAC verification, timestamp/replay-window verification, or endpoint secret verification.

If processed anyway, a forged webhook could mark a payment as successfully charged when no money actually moved, or falsely mark a legitimate payment as failed/refunded.

Signature/MAC check against the raw request body, replay-window check, and endpoint secret verification performed before any parsing of business fields.

Reject before any financial state mutation. Never rely on the parsed payload's own claims (e.g. a 'signature' field inside the JSON body) as a substitute for verifying the actual transport-level signature.

Log and alert on verification failures; investigate whether it reflects a misconfigured secret (operational) or an actual forgery attempt (security incident).

Rotate webhook signing secrets on a defined schedule, and verify against the exact raw bytes the provider signed -- not a re-serialized version of the parsed payload.

A direct (non-outbox) publish call after the DB commit fails, times out, or is never reached because the process crashed between the commit and the publish call.

Downstream consumers (order fulfillment, notifications, ledger posting) never receive the event -- the payment succeeded but nothing downstream happens.

Outbox lag monitoring (rows written but not yet marked published) surfaces this directly if using the outbox pattern; without an outbox, this may only surface when a customer notices a missing order.

Use a transactional outbox so the event's existence is guaranteed by the same transaction that committed the payment state -- the publish gap becomes a relay-lag problem to monitor, rather than a silent event loss.

The outbox relay retries publishing the unpublished row until the broker acknowledges it.

Do not rely on a direct publish-after-commit call for anything downstream treats as consequential -- use the transactional outbox pattern. See Architecture Deep Dive -> Transactional Outbox.

A consumer receives the same payment.success (or similar) event twice.

If the consumer isn't idempotent, order fulfillment could ship twice, a notification could send twice, or a ledger entry could post twice.

The event carries a stable event_id/payment_id/business-operation identity the consumer can check against its own already-processed record.

Each consumer deduplicates or makes its operation idempotent using that stable identity before applying its effect.

No system-level recovery needed -- this is the expected behavior of a correctly built consumer under at-least-once delivery.

Design every downstream consumer's operation to be idempotent from the start, keyed on event_id or the equivalent business operation identity -- not just the payment consumer, every consumer.

The reconciliation job crashes partway through a batch, is restarted and re-runs the same batch, or hits provider rate limits mid-sweep.

A naively re-run batch could double-create ledger entries or issue duplicate refund repairs if repairs aren't themselves idempotent.

Reconciliation job monitoring: run duration, completion status, repair counts, and a stalled/failed-run alert.

Repairs use stable, idempotent reconciliation/action identities, so a re-run of an already-completed (or partially-completed) batch recognizes prior repairs and skips them rather than reapplying them.

Checkpoints/watermarks let a restarted job resume from where it left off rather than restarting the entire sweep; provider rate limits are respected via backoff within the job itself.

Build reconciliation with the same idempotency discipline as the rest of the system from the start -- it is not exempt from the rules it enforces on everything else.

Examples: internal state is FAILED but the provider shows SUCCESS; internal state is SUCCESS but the provider now shows the transaction reversed; the amount recorded internally doesn't match the provider's amount.

Trusting the wrong side of the conflict can mean treating a real charge as if it never happened (missed revenue tracking, no ledger entry) or treating a reversed/failed charge as if it still succeeded (incorrect fulfillment, incorrect customer-facing status).

Reconciliation's comparison step, or a status query triggered by unexpected divergence.

Do not resolve by blindly picking the state with the latest timestamp. Use the provider's own semantics for what the conflicting state actually means (e.g. a reversal is authoritative once confirmed), combined with financial reconciliation against settlement data where available.

The resolution updates internal state through a guarded transition and logs the conflict and its resolution to the audit log; if money movement is involved, the ledger reflects the correction as its own entry, not a silent edit to the original.

Track provider/internal mismatch rate as an observability metric -- a sustained rate is a leading indicator of a systemic integration issue, not just isolated incidents.

A request to Provider A times out without a definitive response. The router treats this as a failure and immediately routes the same logical payment to Provider B, which succeeds. Provider A had, in fact, also processed the charge successfully before timing out.

The customer is charged twice, once at each provider -- and because they're different providers, neither has visibility into the other's transaction.

Reconciliation comparing payment attempts across providers for the same logical payment; a customer or support report of a double charge.

This is prevented at the routing decision, not detected afterward: only fail over to a different provider when the first attempt is known not to have created a charge (a definitive failure), or when there's an explicit, safe cross-provider orchestration/reconciliation mechanism designed for this specific case.

If it still occurs, resolution requires a refund at one of the two providers, tied together by a reconciliation record linking both attempts to the one logical payment.

Maintain separate payment attempts per provider under one logical payment (see Architecture Deep Dive -> Payment Intent and Attempt Model), and never treat an ambiguous timeout as equivalent to a definitive failure when deciding whether to fail over.

A refund request is sent to the provider; the response is lost or the connection drops before a definitive result is received.

A blind retry with a new refund identity risks a duplicate refund; treating it as failed and never following up risks a refund the customer is owed silently never happening.

The refund's own durable state remains unresolved, discoverable via the same reconciliation sweep used for charges.

Resolve exactly like a charge timeout: reuse the same refund idempotency/provider identity for any safe retry, query refund status by stable reference, await the refund webhook, or reconcile.

One of the above channels eventually confirms the refund's true outcome, and the total-refunded invariant is checked before finalizing.

Give every refund its own idempotency key and provider reference from the start, exactly as a charge attempt has -- do not treat a refund as a simpler, ambiguity-free operation.

Two refund requests for the same payment are submitted concurrently, each individually valid against the captured amount, but not valid together.

Without a transactional check against the running refunded total, both can succeed independently and their combined total can exceed the captured amount.

The database constraint/transactional check itself is the detection mechanism -- it should fail the second request at write time, not after the fact.

Enforce sum(successful refunds) <= captured amount transactionally, using a conditional check against the current running total within the same transaction that records the new refund -- not a check-then-write pattern with a gap between them.

The second, over-limit refund request is rejected outright rather than needing to be reversed after the fact.

Never validate a refund amount against a previously-read total without also transactionally re-checking it at write time -- read-then-write without a transactional guard is exactly the race this scenario describes.

The payment state transitions to SUCCESS, but the ledger-posting consumer fails to write the corresponding entry, or never receives the event.

Financial records under-represent actual money movement -- accounting and settlement reconciliation will eventually disagree with the operational payment state.

Ledger-posting lag/backlog monitoring, and financial reconciliation comparing the ledger against provider settlement data.

Order fulfillment and other downstream effects should not be made to depend on the ledger posting having already happened -- they depend on the payment's operational SUCCESS state, which is already durable. The ledger gap is repaired independently.

Depending on architecture: the transactional outbox retries delivery to the ledger consumer, or reconciliation detects the missing entry and posts the correction with a stable, idempotent action identity.

Treat the ledger-posting consumer with the same idempotent, at-least-once-aware design as every other downstream consumer -- and never make fulfillment logic implicitly depend on an unmonitored, untracked financial posting having already succeeded.

Payment creation rate, webhook volume, and downstream event volume all spike well beyond normal levels at once.

None directly from the spike itself, provided correctness mechanisms hold under load -- the risk is that correctness gets silently traded away under pressure.

API latency/error rate, database write contention, provider rate-limit responses, webhook processing lag, outbox/queue backlog growth, and reconciliation lag, tracked as a set rather than any single metric.

Scale the stateless API tier horizontally; let database write contention and provider rate limits set the real throughput ceiling rather than trying to force more throughput than the true bottleneck allows; let the queue/outbox absorb temporary backlog within its retention window.

Backlogs (webhook processing, outbox publishing, reconciliation) drain once the spike subsides, provided their retention/backlog limits weren't exceeded.

Load-test against realistic burst shapes ahead of known high-traffic events, and ensure idempotency and constraint checks are never bypassed as a load-shedding measure -- correctness guarantees are exactly what must not be sacrificed for throughput.

The provider returns rate-limit responses (e.g. 429s), often with a Retry-After header, once request volume crosses a threshold on their side.

Ignoring the signal and retrying aggressively risks getting further throttled or temporarily blocked, worsening the very problem being retried around.

Rate-limit response rate, tracked per provider, separate from other error categories.

Respect the provider's Retry-After guidance; apply bounded concurrency and a client-side rate limiter tuned to the provider's documented limits; apply backpressure or queuing where the operation's semantics allow a delay.

Throughput to that provider recovers once request volume drops back under the provider's limit or the rate-limit window elapses.

Use a provider-specific circuit/bulkhead so rate limiting on one provider doesn't starve capacity meant for others, and size outbound concurrency against the provider's documented limits up front rather than discovering them under load.

A region hosting the payment API, database, or webhook ingestion becomes unavailable.

Ambiguous in-flight attempts during the failover window carry the same double-charge risk as any other ambiguous outcome, now compounded by the question of which region has write authority for a given payment.

Regional health checks and failover automation; RPO/RTO targets defined ahead of time for the payment database and ledger.

Define, before any failure occurs: which region has payment write authority for a given payment (and how idempotency uniqueness is preserved across regions so the same key can't be double-honored in two regions); which region owns a given in-flight provider attempt; how webhook routing reaches the newly-active region; and how the ledger's replication guarantees bound acceptable data loss.

Failover promotes a standby region using the same fencing discipline as any single-leader failover, so the original region cannot resume writing after recovering unexpectedly; jobs/attempts in flight during the window are resolved via the same ambiguous-outcome and reconciliation mechanisms used for any other unresolved attempt.

Design idempotency uniqueness, provider attempt ownership, and ledger replication with regional failover in mind from the start, rather than treating multi-region as a bolt-on -- and rehearse the failover path before it's needed for real.