Failure & Scale Scenarios

A charge exists with no matching order -- the worst-case customer experience if not resolved: money taken, nothing purchased.

A stable checkout/payment reference created before the payment attempt lets this be found -- via the payment provider's webhook, a status query, or reconciliation, rather than depending on the crashed request thread.

Do not blindly create an arbitrary new order. Retry order creation using the same durable checkout/order identity that was created before the payment attempt (see Complete Design's baseline ordering) -- this is why creating a durable pending order/checkout identity before calling the provider matters.

If retries succeed, the order is created/confirmed against the existing identity. If the database was unavailable for an extended period, reconciliation detects the payment with no associated confirmed order and completes it retroactively.

Always create the durable checkout/order identity before the payment attempt (see Trade-offs & Decisions -> Order-Before-Payment vs. Order-After-Payment), so recovery has a stable anchor to resume against.

None if handled correctly -- the reservation is released and the SKU becomes available again for other shoppers.

The payment provider returns a definitive failure/decline response.

Transition the order to a payment-failed state and trigger the compensating release of the inventory reservation.

The compensating release call runs idempotently; if it fails, reservation expiry provides a backstop so the unit doesn't stay held indefinitely (see Reservation Release Failure below).

Reservation TTL/expiry is a required correctness backstop, not a luxury -- without it, a crashed compensation step leaves inventory permanently held.

If the reservation is released and the payment later turns out to have succeeded, the customer was charged for an item no longer reserved for them -- a much worse outcome than a slightly longer hold.

The order sits in PAYMENT_PENDING/UNKNOWN as the reservation's expires_at approaches.

Do not release on timeout alone. Apply an explicit bounded policy: extend the reservation while payment resolves, or -- if it must expire before resolution -- define a later-success recovery/refund path in advance rather than discovering the gap after the fact.

Payment resolves (via webhook, status query, or reconciliation) to either success or definitive failure. If it resolves to success after the reservation already expired, recovery re-attempts allocation or falls back to the pre-defined refund path if the unit is no longer available.

Design the reservation-extension and later-success-recovery policy before launch, not as an incident response the first time it happens. See the Payment System module for how the payment side classifies and resolves ambiguous outcomes.

A shopper may see an outdated price or an unavailable product in search results. This is a discovery-quality issue, not a checkout-correctness issue.

Indexing-lag metric breaches its SLO; a backlog of unprocessed catalog-change events grows.

Continue serving the stale index for discovery -- the catalog database remains authoritative. Checkout always revalidates current price and availability regardless of what search displayed, so stale search data cannot cause an oversell or a wrong charge.

The indexing pipeline drains the backlog once its dependency (indexer capacity, index cluster health) recovers.

Monitor and alert on indexing lag against its own SLO (see Architecture Deep Dive -> Search Read Model), independent of checkout-path health metrics.

Checkout confirmation is unaffected -- customers get their order confirmation immediately. Fulfillment is delayed, not lost.

Consumer lag and backlog-depth metrics on the fulfillment/notification queues.

Let the backlog absorb the burst within its retention window rather than forcing more throughput than the true downstream bottleneck allows. This is graceful degradation, not automatically harmless -- a severe enough backlog can breach fulfillment SLOs, exceed broker storage/retention limits, or exceed operational capacity, and needs its own alerting and admission/degradation policy rather than being assumed safe by default.

As traffic normalizes, the backlog drains; if fulfillment SLOs are breached, proactively communicate delay to affected customers.

Define backlog alert thresholds and a degradation policy ahead of the next expected traffic spike, not as an ad hoc reaction during one.

Catalog browsing is slower, not incorrect. Inventory, checkout and orders are unaffected -- they never depended on the product cache.

Cache health checks/connection errors; a sudden spike in catalog DB read traffic and latency.

A circuit breaker, request coalescing, and rate limiting in front of the catalog DB prevent the full-miss traffic from overwhelming it. Do not assume the database can automatically absorb a jump from a typically low miss rate to 100% traffic -- that assumption is exactly what a circuit breaker and load-shedding policy exist to protect against.

Once the cache recovers, hit rate returns to normal and DB load drops back down; no data reconciliation is needed since the cache was never authoritative.

Load-test the catalog DB against a simulated full cache-miss burst ahead of time, and size the circuit breaker/rate limiter against that measured capacity rather than an assumed one.

A brief latency spike or elevated error rate for that specific product's page, potentially cascading to the catalog DB if many hot keys expire near-simultaneously.

A spike in DB load correlated with a specific cache key's TTL expiry, or generally with periodic bulk cache warm-up/expiry cycles.

Request coalescing/single-flight (only one request repopulates the cache while others wait for that result) prevents the thundering herd; TTL jitter avoids many hot keys expiring in the same instant; stale-while-revalidate serves the slightly-stale value while a single request refreshes it in the background.

The cache repopulates after the single coalesced request completes; load returns to normal immediately.

Apply TTL jitter and single-flight/coalescing specifically to known hot products, not just uniformly low-traffic ones.

Without protection: a duplicate order, a duplicate reservation, or a duplicate charge.

The retried request carries the same checkout idempotency identity as the original.

Resolve the duplicate against the durable checkout/order identity and return the same logical checkout/order -- no second reservation, no second order, no second payment attempt.

No repair action needed -- this is the expected, correct behavior of a properly built idempotent checkout path.

See Architecture Deep Dive -> Checkout Idempotency for the durable identity and fingerprint mechanism that makes this a non-event.

None if recovery works correctly -- the reservation already exists durably and is not lost.

A resumed orchestrator (or a workflow-age monitor) finds a checkout stuck before the expected next transition.

On resume, the orchestrator discovers the existing ACTIVE reservation (via the durable checkout/reservation identity) rather than attempting a second reservation, and continues the saga from that point.

The saga proceeds to the payment step using the already-reserved inventory; if the checkout never resumes, reservation expiry eventually releases it.

Every workflow step must be durably recorded before triggering the next one, so a crash always leaves a resumable state -- see Architecture Deep Dive -> Checkout Orchestration.

The unit stays reserved and unavailable to other shoppers longer than it should, if the compensating release isn't retried.

The release call returns an error, or the reservation's state doesn't transition as expected after the compensation step ran.

Retry the release idempotently. Do not assume the compensation succeeded merely because it was invoked -- see Architecture Deep Dive -> Checkout Compensation.

If retries also fail, reservation expiry (TTL) is the backstop that eventually releases the unit; reconciliation detects any reservation that stayed ACTIVE despite its order reaching a terminal failed/cancelled state.

Never rely on a single invocation of a compensating action succeeding -- always pair it with a durable retry policy and a TTL-based backstop.

Without protection: the reservation could be simultaneously committed (sold) and expired (released), an internally inconsistent state that risks oversell or a confirmed order with no backing stock.

A conditional/versioned state transition on the reservation record rejects one of the two concurrent attempts.

Use a conditional state transition (WHERE state = 'ACTIVE' AND version = expected) for both the commit path and the expiry path, so only one of them can win -- the loser's caller (confirmation flow or expiry scan) handles the rejection explicitly rather than assuming success.

If expiry wins and payment later confirms as successful, the order falls back to the payment-unknown/reservation-expiry recovery policy (see above) rather than silently confirming against a reservation that no longer exists.

Never implement commit and expiry as unconditional writes -- both must go through the same guarded, versioned transition.

If unhandled: the customer is silently charged a different amount than what they saw, damaging trust.

The checkout quote's price/version differs from the cart's displayed-price snapshot.

Surface the change to the customer and require acceptance according to product policy before completing the charge -- never silently charge the stale or the new value without acknowledgment.

The customer accepts the new terms and checkout proceeds with the current quote, or declines and the checkout is abandoned without a charge.

See Architecture Deep Dive -> Pricing Architecture for the display-price vs. checkout-quote vs. order-line-snapshot model that makes this detectable in the first place.

Without revalidation, a shopper could complete a purchase of an item the business no longer intends to sell.

Checkout's purchasability check against current catalog/SKU status finds the item disabled.

Reject that line item at checkout with a clear message, rather than trusting the cart's earlier snapshot of purchasability.

The shopper removes the item and continues with the rest of the cart, or the checkout is abandoned.

Always revalidate purchasability (not just price and availability) at checkout -- a cart is intent captured at an earlier point in time, never a guarantee.

Without an outbox, fulfillment and notification never receive the event -- the order is confirmed and paid, but nothing downstream happens.

With a transactional outbox: outbox-lag monitoring surfaces unpublished rows directly. Without one: this may only surface when a customer notices a missing shipment.

Use a transactional outbox so the event's existence is guaranteed by the same transaction that committed the order state -- the publish gap becomes a bounded relay-lag problem to monitor, not a silent event loss.

The outbox relay retries publishing the unpublished row until the broker acknowledges it.

Do not rely on a direct publish-after-commit call for anything downstream treats as consequential. See Architecture Deep Dive -> Transactional Outbox and Trade-offs & Decisions -> Direct Publish vs. Transactional Outbox.

If the fulfillment consumer isn't idempotent: a duplicate shipment. If the notification consumer isn't idempotent: a duplicate confirmation email.

The event carries a stable event/order/business-operation identity the consumer checks against its own already-processed record.

Each consumer deduplicates or makes its own operation idempotent using that stable identity before applying its effect.

No system-level recovery needed -- this is expected steady-state behavior of a correctly built consumer under at-least-once delivery.

Design every downstream consumer to be idempotent from the start -- see Architecture Deep Dive -> Event Consumer Idempotency.

Applying events blindly in arrival order could confirm fulfillment for an order that was actually cancelled, or vice versa.

A consumer checks the incoming event's implied transition against the order's current recorded state/version.

Consumers use state/version semantics rather than blindly applying events in arrival order -- an event proposing an invalid transition for the order's current state is rejected or deferred, not applied.

The rejected/deferred event is logged for observability; if it represents a genuine gap, reconciliation surfaces it.

Never assume in-order delivery from a message broker without an explicit ordering key or a guarded consumer-side transition check.

Orders remain confirmed and paid; new orders continue to be accepted. Fulfillment is delayed, not lost -- customers experience a shipping delay, not a checkout failure.

Fulfillment consumer lag and error-rate metrics; a growing backlog of unprocessed order.confirmed events in the outbox/broker.

The outbox/broker retains the work durably; checkout and order confirmation continue independently of fulfillment's health.

Once fulfillment recovers, it drains the backlog; if the outage was long enough to breach fulfillment SLOs, proactively communicate delay to affected customers.

If capacity/backlog limits are exceeded for a sustained period, the business may need to throttle new order acceptance for affected categories rather than accept orders it can't fulfill in a reasonable time -- this is a product/ops decision, not an automatic architectural default.

Without protection: two shipments and two shipping labels for one order line -- a real cost and a confusing customer experience.

The redelivered command carries the same business idempotency key (order_id + fulfillment_operation) as an already-processed one.

The fulfillment workflow recognizes the existing shipment for that key and returns/no-ops rather than creating a second one.

No repair needed if idempotency held; if a duplicate slipped through before this was in place, it needs manual reconciliation with the carrier and a customer-facing correction.

See Architecture Deep Dive -> Event Consumer Idempotency and Fulfillment Workflow -- every command that creates a physical, costly side effect needs this discipline.

Checkout for that SKU cannot proceed safely -- this is the one place where failing fast is correct, because guessing risks an oversell.

Connection errors/timeouts from the inventory reservation call.

Fail or pause the checkout step rather than falling back to a stale cache to grant a reservation decision -- a cached count is never authoritative for whether a reservation succeeds (see Trade-offs & Decisions -> Cache vs. Authoritative Read).

Checkout resumes once the inventory store recovers; any checkout that was paused (not failed outright) can proceed from its durable saga state.

If a separately designed, explicitly authoritative reservation path (e.g. a replica promoted with fencing, or a queued-reservation pattern with its own correctness guarantees) exists, it can be used instead of failing outright -- but only if it preserves the same conditional-reservation invariant, not as a casual fallback.

Elevated latency and reservation-failure rate specifically for that SKU's checkout path, even while the rest of the platform is healthy.

Reservation conflict/failure-rate metric spikes for one specific SKU, disproportionate to overall traffic.

Scaling the stateless Inventory Service does not remove contention on the underlying storage row/partition for that one SKU -- see Architecture Deep Dive -> Flash-Sale Hot-SKU Contention for the serialization/partitioning/admission-control responses.

Contention subsides once the SKU sells out (fail fast on 'sold out' to stop further doomed attempts) or the sale event ends.

For known high-demand launches, apply admission control or serialized processing proactively rather than discovering the hot-key bottleneck during the event.

Without independent protection, one overloaded component (e.g. the payment provider rate-limiting the platform) can cascade into failures elsewhere (checkout timeouts, retry storms on inventory).

A dashboard tracking each layer's load independently: catalog/cache hit rate, checkout admission rate, inventory reservation conflict rate, payment provider latency/rate-limit responses, broker/outbox backlog depth.

Protect each layer independently: CDN/cache absorb browse traffic; bounded concurrency and admission control protect checkout and the payment provider; the broker/outbox absorbs backlog within retention limits. Shed nonessential features (recommendations, reviews, personalization) before shedding checkout correctness.

Each layer recovers independently as load subsides; backlogs drain within their retention windows.

Load-test each layer against a realistic flash-sale traffic shape ahead of a known high-traffic event, not just the platform in aggregate.

Legitimate shoppers can be crowded out of scarce inventory by automated purchasing at a scale humans can't match.

Request-pattern anomalies (velocity, timing precision, header/behavior fingerprints) feeding into rate limiting and admission control.

Rate limiting and admission/fairness policy at a conceptual level -- this module does not attempt to teach a full anti-bot security system.

N/A at the architectural level covered here -- specific mitigation is a product/security team responsibility beyond this module's scope.

Combine rate limiting, admission control, and a waiting room where fairness is a hard product requirement (see Trade-offs & Decisions -> Flash-Sale Admission Control).

None if recovery is correct -- the customer may experience a brief delay, but no duplicate charge, reservation, or order.

A checkout/workflow-age metric flags a saga that hasn't progressed within its expected time window.

Resume from the last durably recorded state; every command in the saga (reserve, charge, confirm) is idempotent, so resuming and retrying the current step is always safe.

The saga continues from where it left off, using the durable checkout identity to avoid re-doing already-completed steps.

See Architecture Deep Dive -> Checkout Orchestration -- durable state before triggering each next step is what makes this recoverable by construction.

Depends on when: before any durable order/checkout identity exists, the safest response is to reject cleanly rather than proceed toward a payment attempt with nothing durable to anchor recovery to.

Database connection errors/timeouts from the order-write path.

No confirmation is issued until the required durable order state actually exists. Never call the payment provider before a durable checkout/order identity has been committed.

Once the database recovers, checkout can proceed normally; any in-flight attempt that failed before creating a durable identity is simply retried as a new attempt.

Treat 'durable identity exists before any external side effect' as an absolute ordering rule, not a best-effort guideline.

Paid-but-unconfirmed risks the order never being fulfilled despite a successful charge; confirmed-without-payment-evidence risks fulfilling an order that was never actually paid for.

Reconciliation's comparison step between payment state and order state for the same checkout identity.

Do not resolve by guessing -- use the payment system's own authoritative status (via status query or webhook) as the source of truth for what actually happened financially, and drive the order transition off that.

Reconciliation completes the missing order confirmation, or flags the confirmed-without-payment case for review/cancellation depending on product policy.

Track payment/order mismatch rate as an observability metric -- a sustained rate is a leading indicator of a systemic integration issue between the checkout saga and the payment system.

A confirmed order without a committed reservation risks fulfilling an order for stock that was never actually allocated to it -- a hidden oversell.

Reconciliation's cross-check between order state and reservation state for the same checkout identity.

Flag the discrepancy rather than silently confirming fulfillment; depending on severity, this may require quarantining the order for manual review rather than auto-repairing.

Reconciliation either recreates the missing committed reservation (if stock still allows it) or escalates for manual resolution (if it doesn't).

Ensure the checkout saga always commits the reservation and confirms the order within the same guarded transition sequence -- see Architecture Deep Dive -> Order Reconciliation for the full discrepancy list this sweep should check.

Without a guard, the system could accept a cancellation for an order that's already physically shipped, creating a confusing customer experience and a costly reverse-logistics problem.

The cancellation request checks the order/fulfillment state at the moment it's processed, not the state the customer saw when they clicked cancel.

A conditional transition determines whether cancellation is still allowed given the fulfillment state at that instant; if fulfillment has already progressed past the cancellable point, route the request to the return/refund-after-shipment path instead.

The customer is informed the item already shipped and offered the return workflow instead of a cancellation.

See Architecture Deep Dive -> Cancellation, Returns & Refunds -- the eligibility check must be conditional on current state, not the state at the time the customer initiated the request.

The customer's order shows CANCELLED while their money hasn't actually been returned -- a real financial impact if not tracked to resolution.

The refund's own state (owned by the Payment System) remains pending/failed/ambiguous past its expected resolution window.

Never report a cancellation as financially complete until the refund state actually resolves -- order=CANCELLED and refund=SUCCEEDED are independent facts. Surface refund status to the customer separately from order status.

The refund resolves through the Payment System's own ambiguous-outcome handling (retry, status query, webhook, reconciliation) -- see the Payment System module for the full mechanics.

Never collapse order cancellation and refund completion into one status field -- track them as separate, independently observable state machines.

Ambiguous in-flight checkouts during the failover window carry the same double-reservation/double-charge risk as any other ambiguous outcome, now compounded by the question of which region has write authority for a given SKU's inventory or a given order.

Regional health checks and failover automation; defined RPO/RTO targets for the inventory and order stores.

Define, ahead of any failure, which region owns write authority for a given SKU's inventory and for a given in-flight checkout, and how idempotency uniqueness is preserved across regions so the same checkout identity can't be double-honored in two regions at once.

Failover promotes a standby region using the same fencing discipline as any single-leader failover, so the original region cannot resume writing unexpectedly after recovering; in-flight checkouts are resolved via the same ambiguous-outcome and reconciliation mechanisms used for any other unresolved attempt.

Treat multi-region active-active inventory as an advanced extension to design deliberately, not a casual default -- and rehearse the failover path before it's needed for real.