Failure & Scale Scenarios

The booking stays PAYMENT_PENDING; the user doesn't yet see a confirmation, but nothing is charged twice and no seat is stolen.

The client retries with the same idempotency key, or reconciliation queries the provider for bookings that have stayed PAYMENT_PENDING past a threshold.

Do not automatically retry as a brand-new payment and do not classify this as a decline -- it's unknown, not failed. Keep the booking pending while resolution is in flight.

On confirmed success, transition to CONFIRMED if the hold still owns the seats (see Payment/Hold Boundary for the case where it doesn't). On confirmed failure, release the hold and return the seats to AVAILABLE.

There is no hold duration that guarantees reconciliation always finishes before expiry -- see Architecture Deep Dive -> Payment/Hold Boundary. Design the late-success recovery path deliberately rather than assuming the TTL will always be long enough.

Without a precise guard, the seat could be confirmed and released at the same time -- an internally inconsistent state that risks either an oversell or wrongly losing a paid seat.

A conditional, versioned transition on the hold record rejects one of the two concurrent attempts.

Confirmation must validate booking/hold identity, payment state, current hold ownership, and the hold's expiry/version condition together -- not merely 'is the seat still marked HELD.' Expiry must equally validate the specific hold's identity/version, not just the seat's current state (see Expiry Generation Protection). Whichever conditional transition commits first wins; the other's conditional check fails cleanly.

If expiry wins, the late-arriving payment success falls through to the Payment Success After Hold Expiry recovery path below rather than being silently applied.

Never implement either side as a bare state check -- both must be scoped to the specific hold's identity and version.

During reservation: the hold request fails cleanly with no hold created. During confirmation: the booking may sit unresolved until the database recovers.

Connection errors/timeouts from the booking-write path.

Reject the hold request cleanly rather than guess -- never create a hold that can't be durably persisted. If the client experiences a lost response after the DB actually committed, its retry (carrying the same idempotency key) must discover and reuse the existing hold rather than blindly allocating a second one.

Once the database recovers, in-flight PAYMENT_PENDING bookings are picked up by reconciliation and resolved against actual payment state.

Never optimistically confirm a booking without database durability -- a delayed confirmation is recoverable; a confirmation that was never durably recorded is not.

A seat shown as AVAILABLE was actually just held or confirmed by someone else. The user's reservation attempt is correctly rejected by the authoritative path, and they're shown a fresh view.

A seat that just became available again (its hold expired) still shows as unavailable in the cached view until the read model catches up -- a missed-opportunity UX cost, not an allocation error.

A reservation attempt against a stale-positive seat fails its conditional update; stale-negative cases surface as availability-freshness lag metrics.

The reservation path never trusts the cache -- it always re-validates against authoritative state. Neither staleness direction can cause a double allocation as long as this holds.

The UI refreshes the seat map after a rejected reservation attempt so the user sees corrected state for their next action.

Choose a cache/invalidation strategy against a stated freshness SLO -- see Trade-offs & Decisions -> Availability Cache Strategy -- rather than assuming a single fixed TTL fits every event.

The vast majority of users don't get a seat -- that's an expected outcome of scarce inventory, not a system failure, as long as the ones who do succeed get a correct, non-duplicated booking.

Read traffic spike on the availability path; reservation-attempt volume spike on the booking path; queue/admission-controller backlog growth.

Read traffic is absorbed by the availability cache/read model. Reservation requests pass through admission control sized to the slowest correctness-critical dependency (see Trade-offs & Decisions -> Reject vs. Admit), not simply application-server count.

As the event sells out or demand normalizes, admission and reservation load return to baseline.

Any specific traffic number used to illustrate this scenario is an example, not a universal capacity fact -- size real capacity against measured load tests for the specific event and infrastructure in question.

Without protection: a second, independent hold on different or overlapping seats.

The retried request carries the same idempotency key as the original.

Resolve the duplicate against the durable idempotency record and return the same logical hold -- no second reservation attempt.

No repair needed -- this is the expected, correct behavior of an idempotent hold path.

See Architecture Deep Dive -> Hold Idempotency for the durable key and request-fingerprint mechanism.

The user sees no confirmation and may retry, believing nothing happened.

The client's retry carries the same idempotency key used in the original request.

The retry must discover and reuse the existing hold rather than allocate a second one -- exactly the idempotency mechanism above.

The retry returns the existing hold's details (seats, expires_at) as if it were the original response.

Always resolve retries against the durable idempotency record before considering a request 'new.'

Without protection, User B's legitimately-acquired new hold (or already-confirmed booking) is incorrectly released, potentially causing a lost sale or -- worse -- opening the seat to a third party while B still believes they hold it.

The delayed task's conditional release check includes the original hold's identity (hold_id) and finds it no longer matches the seat's current holder.

The release condition is scoped to the specific hold identity and version, not just seat state -- WHERE hold_id = :old_hold_id AND state = 'HELD' AND held_until <= now. Because the seat's current hold_id is now B's, this condition matches zero rows and does nothing.

No repair needed if the identity-scoped condition was in place from the start -- this is exactly the failure the mechanism exists to prevent.

See Architecture Deep Dive -> Expiry Generation Protection -- this is the single most important safeguard in the hold-expiry design.

Seats whose holds have business-expired may still appear held/unavailable in the read model, even though no one is actually paying for them anymore.

Expiry lag (now - expires_at for holds still ACTIVE) grows and breaches its alert threshold; count of active holds past expiry rises.

Reservation and confirmation logic must still treat an expired hold as invalid based on authoritative time semantics -- expires_at defines business validity immediately, independent of when a cleanup worker physically gets around to updating the row.

Once the worker recovers, it processes the backlog in bounded batches rather than one giant sweep.

Alert on sustained expiry lag well before it becomes user-visible; run the worker with enough redundancy that a single instance failing doesn't halt expiry entirely.

None, if release is correctly idempotent -- both workers attempt the same conditional release, only one succeeds, the other's update affects zero rows.

Duplicate release attempts visible in logs/metrics as zero-row-affected conditional updates -- expected noise, not an error.

The identity-scoped conditional release (see Expiry Generation Protection) makes a duplicate attempt a safe no-op by construction.

None needed.

Partition expiry work across workers (by event, by hold-ID range, or a claim/skip-locked pattern) to reduce wasted duplicate work, while still relying on the conditional check as the actual correctness guarantee.

Elevated latency or contention on the reservation path caused by the expiry worker's own load, ironically during the highest-demand moments when timely expiry matters most.

Database load/latency correlated with expiry worker scan cycles.

Use an index on (state, expires_at), bounded batch sizes, and throttled/partitioned work rather than one large unbounded query -- see Architecture Deep Dive -> Expiry Worker Design.

Reduce batch size or worker concurrency temporarily if contention is observed; let the backlog drain at a sustainable rate rather than forcing it.

Load-test the expiry worker against a realistic number of concurrent active holds for a blockbuster event before it happens for real.

If the product promises all-or-none and the system silently reserves A1 and A3 while failing A2, the user ends up with a partial, unwanted booking they didn't ask for.

The multi-seat transaction's verification step finds A2 already unavailable before committing.

Enforce the all-or-none contract explicitly: if any requested seat can't be acquired and partial holds aren't an explicit supported feature, roll back the entire attempt and report which seat(s) conflicted.

The user is shown the conflict and can retry with a different seat selection.

See Architecture Deep Dive -> Multi-Seat Reservation -- verify all requested seats inside one transaction before committing any of them.

Both transactions block waiting on each other, eventually timing out or being deadlock-aborted by the database -- a failed request for one or both users.

Database deadlock detection aborts one of the two transactions.

Always acquire seats within a multi-seat transaction in a deterministic order (e.g. sorted by seat_id), regardless of the order the user selected them in -- this prevents the circular-wait condition that causes deadlocks.

The aborted transaction retries the whole multi-seat attempt from the start, now free of the ordering conflict.

See Architecture Deep Dive -> Multi-Seat Reservation -- deterministic lock ordering is the standard fix for this class of deadlock.

The customer paid successfully and reasonably expects their seat -- but the hold that backed the reservation has technically expired.

Reconciliation or the payment webhook resolves success; the confirmation step checks current hold/seat ownership and finds the hold no longer ACTIVE.

This is a product policy decision, not a universal rule: the system may attempt a conditional reacquisition/confirmation (if the seat is still unallocated to anyone else) or fall back to refund/void. Do not silently confirm against an expired hold as if nothing happened.

If reacquisition succeeds, confirm the booking normally. If policy chooses refund/void instead, initiate that and notify the customer clearly about what happened and why.

Define this policy explicitly ahead of time -- see Architecture Deep Dive -> Payment/Hold Boundary -- rather than deciding ad hoc during an incident.

The original customer paid but cannot receive that specific seat -- someone else legitimately holds it now.

Confirmation's ownership check finds the seat now belongs to a different, valid hold or confirmed booking.

Never steal the seat back from the party who now legitimately owns it. Preserve the other booking exactly as-is.

Void or refund the original late-successful payment, mark the original booking as an exception/reconciliation case, and notify the original customer according to product policy (offer an alternate seat, a refund, or both).

This scenario is why the payment/hold boundary policy has to exist deliberately -- see Architecture Deep Dive -> Payment/Hold Boundary -- rather than assuming it will never happen.

None, if deduplicated correctly -- if not, a duplicate confirmation attempt or duplicate side effect could occur.

Deduplicate on the provider's event/delivery ID.

The second and later deliveries are recognized as already-processed and acknowledged without reapplying the business effect.

None needed -- this is expected steady-state behavior.

See the Payment System module for the full webhook-deduplication mechanics; this module's booking side just needs to trust that the payment integration tolerates duplicate events by design.

Applying the stale event naively could regress a successfully confirmed booking's payment status back to pending or failed.

A guarded transition check on payment/booking state rejects the invalid regression.

Never let a terminal payment/booking state be regressed by a late-arriving, out-of-order event -- validate every incoming event against the current state's valid-transition table before applying it.

The rejected event is logged for observability; no state change occurs.

Apply the same guarded-transition discipline used elsewhere in the booking state machine (see Architecture Deep Dive -> Booking State Machine) to payment events specifically.

Without an outbox, ticket issuance and notification never fire -- the customer paid and is confirmed in the database, but receives no ticket.

With a transactional outbox: outbox-lag monitoring surfaces unpublished rows directly. Without one: a customer complaint about a missing ticket.

Use a transactional outbox so the event's existence is guaranteed by the same transaction that committed CONFIRMED -- the gap becomes a bounded relay-lag problem to monitor, not a silent loss.

The outbox relay retries publishing the unpublished row until the broker acknowledges it.

See Architecture Deep Dive -> Transactional Outbox and Trade-offs & Decisions -> Direct Publish vs. Outbox.

If ticket issuance isn't idempotent: a second, independently valid ticket for the same seat. If notification isn't idempotent: a duplicate confirmation email -- a smaller, but still real, UX cost.

The event carries a stable event/booking/business-operation identity the consumer checks against its own already-processed record.

Each consumer deduplicates or makes its own operation idempotent using that stable identity before applying its effect.

No system-level recovery needed -- this is expected steady-state behavior of a correctly built consumer.

See Architecture Deep Dive -> Ticket Issuance -- every downstream consumer needs this discipline, not only the ticket generator.

The customer's booking is confirmed and paid, but no ticket/entitlement is immediately available.

Ticket issuance lag metric, or a confirmed booking with no corresponding ticket record past an expected window.

The booking remains CONFIRMED regardless -- ticket generation failure does not roll back a valid, paid booking.

Retry ticket issuance asynchronously; the customer can retrieve their ticket once issuance succeeds, without needing to redo checkout.

Monitor ticket issuance lag as its own metric, separate from booking confirmation latency -- see Architecture Deep Dive -> Observability.

Without protection: two independently usable tickets exist for one paid seat -- a real risk if the product involves admission scanning (two people could both show up with a 'valid' ticket).

A stable business idempotency key (e.g. UNIQUE(booking_id, seat_id)) on the ticket record detects the duplicate attempt.

The second issuance attempt recognizes the existing ticket and returns/no-ops rather than minting a new one.

No repair needed if the constraint held from the start; if a duplicate somehow already exists, invalidate the extra one and audit how it was created.

See Architecture Deep Dive -> Ticket Issuance -- enforce the uniqueness constraint at the database level, not only in application logic.

The seat map shows outdated availability until the entry's TTL expires or the read model otherwise refreshes -- a display-freshness issue only.

Availability-freshness-lag metric, or a user report of a seat map that doesn't match reality.

TTL expiry (or a periodic read-model rebuild) is the backstop that bounds how long the staleness can persist, independent of whether any single invalidation message was delivered. Reservation correctness is entirely unaffected -- the reservation path never depended on the invalidation having arrived.

The cache entry naturally refreshes on its next TTL expiry or rebuild cycle.

Always pair invalidation-based freshness with a TTL backstop rather than relying on invalidation delivery alone.

Seat-map browsing slows down or degrades; reservation and confirmation are unaffected, since they never depended on this cache.

Cache health checks/connection errors; a spike in booking-DB read traffic.

Do not blindly send all seat-map traffic to the booking database. Apply rate limiting, serve stale snapshots where acceptable, reduce refresh frequency, and shed nonessential read load to protect the database that the correctness-critical reservation path also depends on.

Once the cache recovers, read traffic returns to its normal path and DB load drops back down.

Load-test the booking database against a simulated full-cache-miss burst ahead of a known high-demand on-sale.

Depends on the defined policy: users already admitted may or may not be able to continue, and new admission may need to pause entirely.

Waiting-room health checks; admission-rate metric drops to zero unexpectedly.

Define explicitly, ahead of time, whether already-issued admission tokens remain valid during the outage and whether new admission pauses -- do not leave this ambiguous and decide ad hoc during the incident.

Once the waiting room recovers, admission resumes at its configured rate.

Run the waiting room with enough redundancy that it doesn't become a single point of failure for the entire on-sale.

None, if the token and booking path are both correctly bounded -- a replayed or expired token simply doesn't grant anything extra.

The token's own expiry/scope check fails, or booking idempotency recognizes a request as already handled.

The token should be bounded and scoped (time-limited, tied to a specific admission event); booking idempotency still applies fully underneath it -- a token replay does not bypass the reservation invariants.

An expired token requires re-admission through the waiting room, not a silent re-grant.

Never treat admission-token possession as equivalent to a reservation guarantee -- see Architecture Deep Dive -> Virtual Waiting Room.

Elevated latency and reservation-failure rate specifically for that event, disproportionate to overall platform traffic.

Per-event/per-partition load metrics spike well beyond the platform average.

Event-level sharding alone does not solve this -- the whole event lands on one shard. Finer partitioning (event -> section/block -> seat) and admission control specifically for that event are the responses -- see Architecture Deep Dive -> High-Demand Partitioning.

Load subsides once the event sells out or demand normalizes; sold-out detection (see below) should stop unnecessary further load quickly.

For known high-demand on-sales, apply finer partitioning and admission control proactively rather than discovering the hot-shard bottleneck live.

All but one of the concurrent requests for that seat fail -- expected, since only one can win a unique seat -- but a poorly designed client could make this worse by retrying uselessly.

Reservation conflict/failure-rate metric spikes for one specific seat, disproportionate to overall traffic.

Fail the losing requests quickly and clearly -- do not have clients blindly retry the exact same seat after a legitimate ownership conflict (see Trade-offs & Decisions -> Locking vs. Optimistic Concurrency). Point the user toward other available seats instead.

Contention subsides once the seat is HELD/CONFIRMED and the read model reflects that.

Design client UX to avoid encouraging rapid re-clicks on a seat that just failed -- surface the conflict clearly and suggest alternatives.

Without a fast sold-out signal, users continue attempting reservations and payments that are doomed to fail, wasting their own time and system capacity.

Available-seat count for the event reaches zero, accounting for currently active holds that may still expire and free up inventory.

Stop unnecessary admission/payment work promptly once inventory is genuinely exhausted -- while still correctly accounting for active holds that could still expire and briefly reopen availability.

If a hold expires and frees a seat after a sold-out signal was shown, the availability read model should reflect that reopening once it refreshes.

Distinguish 'zero seats available right now' from 'zero seats and zero active holds' when deciding whether to show a hard sold-out message or a 'seats may become available' message.

Users see the event as unavailable, become frustrated, and may abandon -- even though a wave of holds is about to expire and free up real inventory.

A conversion-rate metric (holds created vs. holds confirmed) drops sharply while active-hold count spikes.

Track hold-to-confirmation conversion as a first-class metric, not just raw hold counts. Admission policy can factor in expected conversion rate rather than treating every hold as equally likely to convert.

As holds expire, inventory frees up and becomes available again -- the system should reflect that promptly rather than continuing to look artificially sold out.

Consider admission or hold-policy adjustments (e.g. tighter admission during a known low-conversion pattern) if hold storms are a recurring problem for a given event type.

Users can select and hold seats but cannot complete payment -- if unaddressed, this locks inventory that can't actually convert to a sale.

Payment provider error/timeout rate spikes.

Pause or limit new holds, or adjust hold policy, rather than allowing inventory to remain locked indefinitely by holds that have no realistic path to payment during the outage.

Once the provider recovers, resume normal hold/payment flow; any holds that expired during the outage return their seats to availability normally.

Monitor payment provider health as an input to admission/hold policy, not just as a per-request retry concern.

Customers with ambiguous bookings wait longer than expected to learn whether their seat is confirmed.

Oldest-pending-age metric and mismatch-count metric both trend upward.

Alert on both the oldest pending case's age and the overall mismatch count -- a growing backlog is itself a signal, separate from any single case's resolution.

Scale reconciliation worker capacity, or triage by oldest/highest-impact cases first, until the backlog drains.

Treat reconciliation capacity as a first-class scaling concern, not an afterthought batch job -- see Architecture Deep Dive -> Booking Reconciliation.

Without a defined interaction, a cancelled booking's ticket could still scan as valid at the venue, or a used ticket could still show as active in the customer's app.

Cancellation request against a booking whose ticket has already been issued (or scanned, if the domain tracks that).

Where the product requires admission scanning, cancellation must invalidate/revoke the issued ticket as part of the same workflow -- not leave it independently valid.

If a ticket was already used before cancellation was processed, that's a policy exception to handle explicitly (e.g. no refund, or a manual review case) rather than a silent gap.

See Architecture Deep Dive -> Cancellation & Refund -- ticket revocation is part of the cancellation workflow whenever ticket validity is checked at the point of use.

In-flight holds/bookings during the failover window carry the same ambiguity risk as any other unresolved attempt, compounded by the question of which region has write authority for a given seat.

Regional health checks and failover automation; defined RPO/RTO targets for the booking database.

Define, ahead of any failure, which region owns write authority for a given event/partition's seats, and ensure fencing prevents the original region from resuming writes unexpectedly after it recovers.

Failover promotes a standby region using the same fencing discipline as any single-leader failover; in-flight holds/bookings are resolved via the same reconciliation mechanisms used for any other unresolved case.

Treat active-active multi-region scarce-seat reservation as an advanced extension requiring deliberate fencing design, not a default -- see Trade-offs & Decisions -> Single-Region Owner vs. Multi-Region Active-Active.

Ownership is per event or partition, so every seat has exactly one region that may allocate it. A hold's expires_at is an absolute timestamp, so a hold that lapsed during the failover window is already invalid when the new owner starts serving; it does not get a fresh TTL because the region changed. Reservation and confirmation still validate hold identity and version, so a late write from the old region, or an expiry task that outlived its hold, cannot release or confirm a seat that now belongs to a different hold. Payments left unresolved by the window follow the existing payment-after-expiry policy rather than a failover-specific one.