Splitwise System Design Failure, Recovery and Scaling Scenarios

Every write that creates financial effect — a new expense, a settlement, a split — has to survive being retried, and has to either fully succeed or leave no partial trace. These two incidents are the foundation everything else in this module depends on: a client that can't tell 'it failed' from 'the response got lost,' and a database that can crash mid-transaction.

Scenario 01 — Duplicate expense POST after timeout

Trigger: Asha's phone submits the ₹900 dinner expense (equal split across Asha/Ben/Cara), the request reaches the server and is fully processed, but the response times out on the way back — a flaky hotel Wi-Fi connection drops right after the write commits. Asha's client, unable to tell success from failure, automatically retries the identical POST a few seconds later, this time carrying the same client-generated idempotency key. Immediate technical effect: this is Vector F. The write path checks the idempotency key against a durable key-to-result ledger before creating any new expense row, splits, or ledger entries; the retried request is recognized as already-processed and is never re-executed as a second expense.

User-visible effect: the group ends up with exactly one ₹900 dinner expense, one set of three split rows, and one ledger effect — Asha +₹600, Ben −₹300, Cara −₹300 — and exactly one activity-feed entry, never two. The retried request receives the same success response the original would have returned, so Asha's client has no reason to believe anything went wrong. How to detect it: idempotency-key hit rate on the write path, with an alert on an unusually high duplicate-key rate from one client, which usually signals an overly aggressive client-side retry policy rather than a one-off network blip.

Containment: no halt — resolved inline, per request; other members' writes to the same or other groups are unaffected. Recovery sequence: none needed — the idempotency check is itself the recovery, returning the original committed result rather than creating a new one. If the retried request arrives with a different payload under the same key (for example a changed amount), that is treated as a documented conflict and rejected with an explicit error, never silently applied as if it were the original. How to prevent recurrence: require a client-generated idempotency key on every financially effective write, keep the key-lookback window at least as long as the client's maximum retry window, and treat 'same key, different payload' as a hard error rather than resolving it by guessing which payload was intended.

Scenario 02 — DB transaction partially fails (expense committed, splits/ledger don't)

Trigger: while committing Ben's ₹600 taxi expense (exact split Asha ₹100 / Ben ₹200 / Cara ₹300), the database process is killed, the connection drops, or a constraint violation fires after the expense header row is written but before the three split rows and the corresponding ledger entries are written — a process crash landing in the worst possible gap. Immediate technical effect: without an atomic transaction boundary, the database could be left with an expense row that exists but has no splits and no ledger effect, violating 'total paid = total cost' and 'zero-sum ledger' simultaneously — a half-written financial fact is worse than no fact at all.

User-visible effect with correct transaction boundaries: none of this is ever visible — the expense, its splits, and its ledger entries are written inside a single atomic transaction, so a crash at any point before commit leaves zero trace, not a partial one; a retry (idempotency-key-guarded, as in Scenario 01) completes the write cleanly on the next attempt. How to detect it: a continuous reconciliation check asserting every committed expense has exactly the split rows its amount requires and a matching zero-sum ledger contribution; any expense failing that check is a hard-alert, since it should be structurally impossible, not just unlikely.

Containment: no halt to other groups' or other members' writes; a failed transaction simply rolls back entirely and reports failure to the caller rather than leaving a partial write visible to reads. Recovery sequence: the client's idempotency-key retry (Scenario 01's mechanism) re-attempts the full write; because the first attempt never committed anything, the retry is a clean first success, not a duplicate. How to prevent recurrence: wrap expense-header, split-row, and ledger-entry writes in a single database transaction (or an outbox-pattern equivalent per Scenario 09), never allow a reader to observe an expense row before its splits and ledger effect are also durably committed, and run the reconciliation check continuously rather than only after an incident.

An expense is a mutable-looking document sitting on top of an immutable, versioned history. These three incidents are about what happens when two things try to change, or change and settle against, the same expense at nearly the same moment.

Scenario 03 — Two users edit the same expense (concurrent edit race)

Trigger: Asha and Ben both open the ₹900 dinner expense (currently version 4) to edit it at nearly the same time — Asha wants to correct the amount, Ben wants to change the split type — and both submit their edits within milliseconds of each other, each believing they are editing version 4. Immediate technical effect: this is Vector G. Both edit requests carry an expected-version precondition (version 4). The write path serializes the two requests; whichever commits first succeeds, creating version 5 and revoking version 4 as the prior version, atomically. The second request's expected-version check fails, since the current version is now 5, not 4.

User-visible effect: exactly one edit wins and becomes the new version 5; the other user receives an explicit conflict response (not a silent failure, and not a silent overwrite) along with the now-current version 5 so they can review it and resubmit their change against the new baseline if it still applies. Never both edits silently succeed, and never does one edit silently vanish without the submitting user being told. How to detect it: an expected-version-mismatch counter on the expense-edit endpoint, tracked per group, with a spike indicating either genuinely high concurrent-editing activity or a client bug that isn't refreshing its local version before allowing an edit.

Containment: no halt; resolved per expense via optimistic concurrency control — a compare-and-set against the expected version — never a global lock across the group. Recovery sequence: the losing client re-fetches the current (version 5) state, re-applies its intended change on top of that baseline if still desired, and resubmits as a new edit attempt with expected version 5. How to prevent recurrence: every expense edit must carry an expected-version precondition, enforced atomically at the database layer (not merely checked-then-written at the application layer, which reopens the same race), and every version transition must be append-only in the audit trail, never an in-place field update.

Scenario 04 — Edit races with a settlement on the same expense

Trigger: Cara is in the middle of recording her ₹200 repayment to Asha (Vector C) while, at nearly the same instant, Asha edits the underlying dinner expense's amount from ₹900 to ₹850 to correct an overcharge — two different kinds of write (a settlement and an expense edit) touching overlapping balance state concurrently, rather than two edits of the same type as in Scenario 03. Immediate technical effect: the settlement and the expense edit are distinct ledger-affecting events, each independently versioned and independently appended; the system does not attempt to merge them into one combined operation. Each is applied as its own atomic, ordered ledger entry against the group's running balance.

User-visible effect: both effects land — the edited expense amount (₹850) changes the balances it contributes, and Cara's ₹200 settlement is recorded as its own entry — and the final balance is the deterministic sum of applying both ledger entries in their committed order, never a race where one silently clobbers the other's contribution. If the expense edit specifically needs to reference 'balance at the time the settlement was recorded' for some derived display, that display is computed from the ordered ledger, never from a single mutable balance field two writers raced to update. How to detect it: a per-group balance-reconciliation job summing every ledger entry (expense splits and settlements alike) and comparing against the materialized balance, which would flag any writer-race-induced drift immediately.

Containment: no halt; both writes proceed through the same append-only, ordered ledger-entry mechanism, each independently atomic. Recovery sequence: none needed under correct design — recovery only becomes necessary if a bug let one write read a stale balance and compute an effect from it rather than appending an independent entry, in which case the fix is recomputing the materialized balance from the full ordered ledger (the same rebuild pattern as Scenario 14). How to prevent recurrence: never model 'the balance' as a single mutable field that a settlement and an edit both read-modify-write; model it as the sum of an append-only ledger, so that unrelated-but-concurrent financial events naturally compose instead of racing.

Scenario 05 — Expense deleted after a repayment already recorded against it

Trigger: after Cara's ₹200 repayment to Asha (Vector C) has already been recorded, Asha attempts to delete the original ₹900 dinner expense entirely — perhaps believing it was a duplicate — not realizing a real-world settlement has already happened partly on the strength of the balance that expense created. Immediate technical effect: a hard delete would retroactively invalidate the context the ₹200 repayment was made against, and would violate the immutable-audit-trail invariant regardless of whether a settlement depended on it; the system instead only ever supports a versioned reversal, which creates a new, explicit reversing entry rather than removing the original expense or its history.

User-visible effect: the dinner expense is marked reversed/voided (a new version, not an erasure) and its own ledger contribution is zeroed out going forward by an explicit, audited reversing entry — but Cara's ₹200 repayment remains exactly as recorded, still fully visible in the audit trail, and the group is shown both the original expense, its reversal, and the unaffected settlement as three distinct, timestamped, never-overwritten events. The group's current balance reflects all three honestly, even though one of them (the repayment) was made in a context that no longer fully applies. How to detect it: a pre-delete/pre-reversal check surfacing to the user that a settlement references this expense's balance-affecting window, so the person doing the reversal is not acting blind; after the fact, an audit query for 'reversed expenses with settlements recorded against their pre-reversal balance' flags the resulting situation for human review rather than resolving it automatically.

Containment: no halt; a hard delete is simply never an available operation on an expense with any ledger history — only a versioned reversal is. Recovery sequence: if the group genuinely wants the net effect reconciled (for example Cara's ₹200 reimbursed back), that happens through a new, separate, explicit settlement or adjustment entry, recorded going forward — never by rewriting the original repayment or the original expense's history. How to prevent recurrence: disallow hard deletes entirely for any expense with ledger effect; support reversal-and-replacement only, and surface a warning, not a block, when a reversal targets an expense that settlements have already referenced, so the decision stays informed and human-driven rather than silently automated.

Every expense's splits must sum to the total exactly, every amount is an integer number of paise, and no amount should ever be able to overflow or silently round away a few paise. These three incidents are about enforcing that arithmetic contract at the boundary, before anything is ever written to the ledger.

Scenario 06 — Invalid split sum submitted

Trigger: a client submits Ben's ₹600 taxi expense with an exact split of Asha ₹100 / Ben ₹200 / Cara ₹250 — summing to ₹550, not ₹600 — due to a client-side bug, a stale form value, or a user manually overriding one share without the UI recalculating the others. Immediate technical effect: server-side validation sums the submitted shares in integer paise and compares against the total amount in paise before any write is attempted; a mismatch rejects the request outright with an explicit error identifying the shortfall, rather than attempting to silently absorb or distribute the missing ₹50.

User-visible effect: no expense is created, no splits are written, no ledger effect occurs — the client is told exactly how much the submitted shares are short (or over) by, so it can correct the input before resubmitting. How to detect it: a split-sum-validation-rejection rate tracked per client version, since a sustained spike after a release strongly suggests a client-side calculation regression rather than isolated user error.

Containment: no halt; a per-request validation rejection, isolated to the one submission. Recovery sequence: none needed at the server — the client corrects its payload and resubmits; if the bug is systemic, the fix is in the client's split-calculation logic, not a server-side workaround that tries to guess the intended distribution. How to prevent recurrence: validate that shares sum to the total exactly, in integer minor units, as a hard precondition on every expense write — never a warning, never an auto-correction — and keep this check identical across every expense-creation surface (manual split, percentage split, recurring expense) so no entry point can bypass it.

Scenario 07 — Fractional rounding mismatch on a percentage split

Trigger: a percentage split divides ₹1,000 three ways at 33.33% / 33.33% / 33.34%, or an equal three-way split of ₹1,000 computes 333.33... paise per person, a value that is not representable as an integer number of paise; naively rounding each share independently (333, 333, 333) sums to only ₹999, one paise short of the ₹1,000 total. Immediate technical effect: the split-computation logic applies a deterministic remainder rule — compute each share's integer-paise floor, then distribute the leftover paise one at a time, in a fixed, documented order (for example by ascending member ID, or in submission order), until the remainder is exhausted, so the shares always sum to exactly ₹1,000 with no ambiguity about which member absorbs the extra paisa.

User-visible effect: the three shares come out as, illustratively, ₹333.34 / ₹333.33 / ₹333.33 (or whichever member the deterministic rule assigns the extra paisa to), summing to exactly ₹1,000 — never ₹999 or ₹1,001 — and the same input always produces the same per-member assignment of the odd paisa, so the split is reproducible and explainable if a user asks why their share is one paisa higher than another's. How to detect it: a split-sum-equals-total assertion run on every computed split before it's persisted, functioning as a safety net behind the remainder-rule logic itself; any failure here indicates a bug in the remainder distribution, not a legitimate rounding outcome.

Containment: not an incident requiring containment when the deterministic rule is correctly implemented — it's a designed-for case, not a failure mode; if the assertion above ever fails, the write is rejected rather than persisting an expense whose splits don't sum correctly. Recovery sequence: not applicable in the normal case; if a bug in the remainder logic is found after some splits were already persisted incorrectly, affected expenses are identified via the same sum-check run retroactively and corrected through a versioned reversal-and-replacement, never a silent in-place patch. How to prevent recurrence: implement percentage and equal splits with one single, well-tested, deterministic remainder-distribution function shared across every split type and every client, rather than letting each split type or each client reimplement its own rounding independently.

Scenario 08 — Very large amount / integer overflow risk

Trigger: a group records an unusually large expense — say a ₹50,00,000 (five-million-rupee) shared property deposit — or a bug in a client multiplies a normal amount by a large factor before submission, producing a paise value that, if stored in a narrow integer type, risks silent wraparound or truncation during an intermediate calculation such as multiplying a per-unit price by a large quantity. Immediate technical effect: amount fields are validated against an explicit, documented maximum (chosen well below any storage type's actual ceiling, with headroom) before being accepted, and all arithmetic on amounts is performed using a wide integer type throughout the write path — never an intermediate floating-point or narrow-integer computation that could lose precision or wrap around before the final value is validated.

User-visible effect: a legitimately large expense within the documented maximum is accepted and behaves exactly like any other expense; an amount exceeding the documented maximum (whether genuine or the product of a client bug) is rejected with an explicit 'amount exceeds allowed maximum' error, never silently truncated, wrapped to a negative or nonsensical value, or partially applied. How to detect it: a validation-rejection metric specifically for the amount-ceiling check, and a defensive assertion that no persisted amount or derived split value is ever negative or implausibly large relative to the group's own history, catching a bypass of the primary check.

Containment: no halt; a per-request rejection for anything exceeding the documented ceiling. Recovery sequence: none needed for a correctly rejected request; if a bug ever let an overflowed value through before this validation existed, the affected expense is identified via the defensive assertion above and corrected through a versioned reversal, never a silent data patch. How to prevent recurrence: store and compute all monetary values as integer minor units in a wide integer type end to end, validate every amount against an explicit documented maximum at the API boundary before any arithmetic is performed on it, and never perform intermediate monetary arithmetic in floating point or a narrow integer type at any layer of the stack.

Weekend Trip's balances are always derived from the immutable ledger, but the fast, queryable balance view most screens actually read — a projection — is a separate, rebuildable artifact fed by an outbox and an event stream. These six incidents live in the gap between 'the ledger is correct' and 'the projection reflects it.'

Scenario 09 — Duplicate event-bus delivery to the projection

Trigger: the transactional outbox relay delivers the ledger event for Asha's ₹900 dinner expense to the balance-projection consumer, but the consumer's acknowledgment back to the message bus is lost, so the at-least-once-delivery bus redelivers the identical event a second time. Immediate technical effect: the projector maintains its own applied-event-ID bookkeeping, separate from the ledger's own idempotency key, and checks every incoming event against it before applying; the redelivered event is recognized as already-applied and is a no-op rather than being summed into the projection a second time.

User-visible effect: the projected balances read Asha +₹600, Ben −₹300, Cara −₹300 exactly once — never doubled to +₹1,200/−₹600/−₹600 from an undetected double-apply. How to detect it: a per-event apply-count metric inside the projector, and a periodic reconciliation comparing the projection's balances against a fresh sum of the ledger's raw events for the same group.

Containment: no halt; resolved inline by the projector's own idempotency check. Recovery sequence: none needed when the check is correctly implemented; if a bug ever bypassed it, the affected group's projection is rebuilt from the ledger (Scenario 14's rebuild pattern) rather than patched in place. How to prevent recurrence: the projector must maintain applied-event-ID bookkeeping independent of any upstream idempotency check, since at-least-once bus delivery means the projector will see redelivered events as a matter of course, not as an edge case.

Scenario 10 — Event delivered out of order

Trigger: the dinner expense event (Vector A) and the taxi expense event (Vector B) are published to the bus in that order, but a partition rebalance or multi-path delivery causes the taxi event to reach the projector before the dinner event. Immediate technical effect: if the projector applied events strictly in arrival order with no sequence check, it could compute an intermediate balance state that never actually existed in the ledger's true order, and any per-event display (like 'balance after this expense') could show an impossible sequence.

User-visible effect with correct handling: the projector buffers or re-sequences events per group using each event's ledger-assigned sequence number before applying them, so the final projected balance is identical regardless of delivery order, and no intermediate, out-of-true-order state is ever exposed to a reader. How to detect it: an out-of-order-arrival counter on the projector's ingestion side, tracked per group partition, distinguishing this from Scenario 09's duplicate-delivery case.

Containment: no halt; resolved by the projector buffering briefly and reordering before apply, scoped to the specific group whose events arrived out of order. Recovery sequence: if a gap in sequence numbers persists beyond a bounded wait (suggesting a missing event rather than simple reordering), the projector requests a resync from the durable ledger rather than guessing or skipping ahead. How to prevent recurrence: every outbox event carries a monotonically increasing per-group sequence number, and the projector enforces strict in-order application per group, never applying event N+1 before confirming event N has been applied or explicitly skipped after a resync.

Scenario 11 — Outbox relay crashes mid-batch

Trigger: the transactional outbox relay process, responsible for reading newly committed ledger events and publishing them to the bus, crashes partway through publishing a batch — some events in the batch reached the bus, others didn't, and the relay's own bookkeeping of 'last published offset' may or may not have been durably updated before the crash. Immediate technical effect: the ledger itself is unaffected — the crash is entirely in the relay/publish stage — but the projection now risks either a gap (events never published) or, on restart, a replay of events already published once, depending on exactly where the relay's offset tracking landed relative to the crash.

User-visible effect: during the gap, the projection is stale for the affected group by exactly the unpublished events — a bounded staleness window, not data loss, since the ledger still has every event durably; once the relay restarts and resumes from its last durably recorded offset, any genuinely missed events are published (and any already-published ones that get republished are absorbed by Scenario 09's projector-side idempotency check). How to detect it: relay-lag and relay-health monitoring comparing the relay's last-published offset against the ledger's latest committed offset, plus standard process-crash/restart alerting.

Containment: no halt to ledger writes — the outbox pattern decouples committing a ledger event from publishing it, so a relay crash never blocks new expenses or settlements from being recorded. Recovery sequence: the relay restarts (or a replacement takes over) and resumes publishing from its last durably committed offset, relying on the projector's idempotency check to safely absorb any events republished due to uncertainty about exactly what was sent before the crash. How to prevent recurrence: make the relay's offset tracking itself durable and updated only after a publish is confirmed, run the relay under fast-restart supervision, and lean on projector-side idempotency as the safety net rather than trying to make the relay's own crash recovery perfectly exact.

Scenario 12 — Balance projection lags significantly behind the ledger

Trigger: the projector's consumption rate falls sustainably behind the ledger's event-production rate — insufficient consumer parallelism, a slow downstream dependency, or a burst of activity (many expenses added in quick succession during a trip) that outpaces normal projector throughput. Immediate technical effect: the gap between the ledger's latest committed offset and the projector's last-applied offset grows instead of staying bounded, and the balances most screens read reflect an increasingly old moment relative to the ledger.

User-visible effect: Asha might not see Cara's just-recorded ₹200 repayment reflected in her balance for longer than usual — not wrong, just late, and the staleness should be surfaced explicitly (a 'last updated' indicator) rather than presented as instantaneous. How to detect it: projector lag, converted to an estimated time delay, tracked continuously per group and in aggregate, not just checked reactively during an incident.

Containment: no halt to ledger writes — the ledger keeps accepting new expenses and settlements regardless of projector speed, since the two are deliberately decoupled; if lag threatens to breach a stated staleness budget, it's escalated as a capacity incident, not hidden from users. Recovery sequence: scale out projector consumer parallelism (partitioned so one group's events stay strictly ordered) or fix whatever slow dependency caused the lag; the projector drains its backlog in order, applying every event exactly once, same as live traffic. How to prevent recurrence: provision projector throughput with headroom above assumed peak event rate, monitor lag continuously against a stated staleness budget, and design the projector to scale horizontally across group partitions without breaking per-group event ordering.

Scenario 13 — Cache contains stale/old balances after an update

Trigger: a short-TTL cache sits in front of the balance projection to absorb repeated reads of the same group's balance screen, and Cara's ₹200 repayment to Asha is applied to the underlying projection, but the cache hasn't yet refreshed or been invalidated at the moment a member re-opens the app. Immediate technical effect: reads served from the cache during the gap between the projection's update and the cache's next refresh reflect the pre-repayment balances — an explicit, bounded staleness the cache layer trades for reduced read load on the projection store, not an unexpected bug.

User-visible effect: Ben might briefly see Cara still owing the pre-repayment amount before the next cache refresh or an explicit invalidation catches up, bounded by the cache's TTL and ideally labeled with a staleness indicator, never presented as guaranteed-instantaneous. How to detect it: periodic cache-vs-projection divergence checks confirming the gap never exceeds the expected TTL-bound window, plus cache refresh-latency monitoring.

Containment: no halt; accepted, bounded behavior unless the divergence window exceeds its stated bound, in which case it's treated as a cache-refresh-pipeline problem. Recovery sequence: the next scheduled refresh, or an explicit cache invalidation triggered by the settlement event itself if that optimization is in place, brings the cache back in line with the projection. How to prevent recurrence: choose a cache TTL deliberately against a stated staleness budget, and consider event-driven invalidation for balance-sensitive views rather than waiting out a fixed TTL blindly.

Scenario 14 — Projection disagrees with the ledger (drift detected)

Trigger: a bug in the projector's apply logic, an undetected partial failure, or memory corruption causes the materialized balance projection for Weekend Trip to quietly diverge from what the immutable ledger's event history actually implies — no crash, no visible error, just a projection that no longer agrees with its source of truth. Immediate technical effect: the group keeps reading from the live projection, which is now silently wrong — the most dangerous class of incident precisely because nothing visibly fails; a reconciliation job comparing the projection's balances against an independently computed sum of the raw ledger events is what actually surfaces the problem.

User-visible effect without detection: Weekend Trip's displayed balances don't actually sum to zero across members, or don't match what a manual tally of the ledger would produce — eroding trust well before anyone identifies a root cause. How to detect it: a continuous, scheduled reconciliation job per group (and sampled across all groups) that replays the ledger independently and diffs the result against the live projection; any nonzero diff is a hard alert, never a soft warning, since the projection having zero-sum balances internally doesn't guarantee it matches the ledger's actual history.

Containment: the affected group's projection is flagged, and if the divergence is severe enough to risk showing visibly wrong balances, reads fail over to a freshly rebuilt view rather than continuing to serve known-corrupted state. Recovery sequence: discard the suspect projection state for the affected group, and deterministically rebuild it from scratch by replaying every ledger event for that group in order through the same idempotent-apply logic as live traffic; resume serving the group only once the rebuilt projection's balances are confirmed zero-sum and matching the ledger. How to prevent recurrence: run the reconciliation job continuously in production, not only during an active incident, and keep the projector's rebuild-from-ledger path tested and fast, since it's both the detection mechanism's companion and the only trustworthy recovery path.

A group's expenses belong to that group, and a member's ability to act is scoped to groups they actually belong to. These four incidents probe that boundary — removal races, token replay, cross-group object access, and permission checks on edits.

Scenario 15 — Group member removed while a write targeting them is in flight

Trigger: Asha begins adding a new expense that includes Ben as a beneficiary at the same moment another member removes Ben from Weekend Trip — perhaps because Ben is believed to have already left the trip — so the membership-removal write and the expense-creation write race against overlapping state. Immediate technical effect: the expense-creation path checks membership validity as part of its write transaction, not merely at form-load time on the client; if Ben's removal commits first, the in-flight expense creation referencing Ben as a split participant is rejected with an explicit 'member not active in this group' error rather than silently completing with a now-removed member baked into the split.

User-visible effect: Asha's submission fails clearly, telling her Ben is no longer a group member, so she can decide how to proceed (re-add Ben, exclude him from the split, or cancel) rather than the system silently either dropping Ben from the split without telling her or letting a removed member retain balance-affecting involvement without anyone realizing it. How to detect it: a member-validity-check rejection rate at expense-creation time, and an audit check confirming no committed expense ever references a member who was not active in the group at the expense's recorded timestamp.

Containment: no halt to other groups or other members; the race is resolved per-request via the membership check inside the write transaction. Recovery sequence: Asha resubmits with updated group composition; if Ben's removal is later reversed (he's re-added), any expense that would have included him is a fresh submission, never a retroactive patch to a transaction that already failed cleanly. How to prevent recurrence: enforce membership validity as a transactional precondition on every write that references a member, not a client-side-only check performed once at form load, so a race against a concurrent removal can never slip through.

Scenario 16 — Invitation token replay

Trigger: an old Weekend Trip invitation link, already used once by Ben to join the group (or since expired, or since revoked after the trip's membership was finalized), is reused — perhaps found in an old message thread, or deliberately replayed by someone who intercepted it. Immediate technical effect: the invitation-acceptance endpoint checks the token against its own durable state (used, expired, or revoked) before granting membership; a replayed token in any of those states is rejected outright rather than silently granting membership a second time or to an unintended party.

User-visible effect: the person replaying the token sees an explicit 'this invitation is no longer valid' message and does not gain group membership or any visibility into the group's expenses; Ben's original, legitimate membership is completely unaffected by someone else's replay attempt. How to detect it: an invitation-token-rejection rate broken out by rejection reason (already used, expired, revoked), with a spike in 'already used' rejections for one token worth investigating as a potential token-leakage incident.

Containment: no halt; a per-request rejection, with no effect on the group's existing membership or any other invitation token. Recovery sequence: none needed for a correctly rejected replay; if investigation confirms a token was leaked and a real unauthorized join occurred before detection, the resolution is removing the unauthorized member (handled per Scenario 15's membership-change path) and issuing a fresh invitation to any legitimately intended recipient. How to prevent recurrence: make invitation tokens single-use and expiring by design, record token state transitions durably (issued → used/expired/revoked) so replay detection doesn't depend on fallible in-memory state, and never make token validity inferable purely from the token's own contents without a server-side state check.

Scenario 17 — Cross-group IDOR attempt

Trigger: a user who belongs to some other group entirely — not Weekend Trip — guesses or forges a sequential or otherwise predictable expense ID belonging to Weekend Trip's ₹900 dinner expense, and issues a request to view or mutate it directly by ID, bypassing any group-scoped navigation. Immediate technical effect: every expense-access and expense-mutation endpoint checks that the requesting user is an active member of the group the target expense belongs to, as a mandatory authorization step performed on the server for every request — never inferred from the fact that the client happened to know the ID, and never skipped because the ID itself 'looks like' it belongs to a valid expense.

User-visible effect: the request is rejected with an authorization error (ideally a generic 'not found' rather than a revealing 'forbidden,' to avoid confirming the ID's validity to an attacker probing for valid IDs), and no data about Weekend Trip's expense — not even its existence — is disclosed to the unauthorized requester. How to detect it: an authorization-rejection rate on object-access endpoints broken out by requester and target-group pairing, with a sustained pattern of one user probing many sequential or near-sequential IDs across groups they don't belong to flagged as a probable enumeration attempt.

Containment: no halt to Weekend Trip's own members' access; the rejection is scoped entirely to the unauthorized requester. Recovery sequence: none needed for a correctly rejected attempt; if an enumeration pattern is detected, the requesting account can be rate-limited or flagged for review independent of any single request's outcome. How to prevent recurrence: enforce object-level, group-membership-scoped authorization as a mandatory check on every single expense (and settlement, and member-list) access path, never rely on an ID being hard-to-guess as a substitute for an actual authorization check, and prefer non-sequential, non-enumerable identifiers as a defense-in-depth layer on top of — never instead of — the authorization check.

Scenario 18 — Unauthorized expense edit attempt

Trigger: a Weekend Trip member who is not a participant in a particular expense — say, a fourth member added to the group after the dinner expense was recorded, who was never a payer or beneficiary on it — or a member whose role is restricted from editing others' entries, attempts to edit the ₹900 dinner expense's amount or split. Immediate technical effect: the edit endpoint checks not just group membership (as in Scenario 17) but the specific authorization policy for mutating this expense — for example, requiring the editor to be the original payer, a beneficiary, or hold an explicit group-admin role — before the versioned-edit path (Scenario 03's mechanism) is even reached.

User-visible effect: the edit request is rejected with an explicit 'not authorized to edit this expense' error; the expense remains at its current version, completely untouched, and no version-conflict machinery is invoked at all, since the request never qualifies to compete for the compare-and-set in the first place. How to detect it: an authorization-rejection rate specifically on the edit path, distinct from Scenario 17's object-existence-scoped check, since this one fires even for members who do legitimately belong to the group.

Containment: no halt; a per-request rejection with zero effect on the expense's state or version. Recovery sequence: none needed; if the requester genuinely should have edit rights (for example, a group-admin role was supposed to be granted but wasn't), that's resolved through the group's own role-management flow, entirely separate from the expense-edit path itself. How to prevent recurrence: define and enforce an explicit, server-side authorization policy for who may edit a given expense (payer, beneficiary, or admin role — whichever the product chooses), apply it consistently across every edit entry point, and never let a client-side-only check (such as simply hiding an edit button in the UI) stand in for the server-side enforcement.

A settlement is a user's assertion until a payment provider confirms it, and that confirmation can be duplicated, delayed, lost, or reversed. These four incidents are about keeping 'settled' meaning exactly what it claims to mean — never showing a pending or failed payment as a confirmed one.

Scenario 19 — Payment webhook delivered duplicated

Trigger: Cara pays her ₹200 to Asha through an integrated payment provider; the provider's webhook confirming successful settlement is delivered to Weekend Trip's backend, but the acknowledgment back to the provider is lost, so the provider's own at-least-once retry policy redelivers the identical confirmation webhook a second time. Immediate technical effect: the webhook handler checks the provider's transaction ID against a durable idempotency ledger (the same pattern as Scenario 01, applied to inbound webhooks) before marking the settlement confirmed and appending its ledger effect; the redelivered webhook is recognized as already-processed and produces no second ledger entry.

User-visible effect: Cara's ₹200 settlement is recorded exactly once — Asha +₹300, Ben +₹100, Cara −₹400 — never twice, which would have incorrectly implied Cara paid ₹400 total. How to detect it: webhook-idempotency-hit rate per provider transaction ID, with an alert on an abnormal duplicate-delivery rate from a given provider integration, since that can indicate either normal provider retry behavior under network issues or a misconfigured acknowledgment response on the receiving end.

Containment: no halt; resolved inline per webhook delivery. Recovery sequence: none needed — the idempotency check is itself the recovery, and the handler returns an acknowledgment to the provider regardless, so the provider's own retry logic stops. How to prevent recurrence: require every inbound webhook handler to check the provider's transaction ID against a durable idempotency ledger before applying any ledger effect, and always acknowledge a webhook (even a duplicate) promptly so the provider's retry logic doesn't escalate unnecessarily.

Scenario 20 — Payment goes pending then fails

Trigger: Cara initiates her ₹200 repayment through the integrated payment provider; the provider immediately reports the payment as pending (common for bank-transfer-backed payment methods), and some time later the provider's authoritative follow-up reports the payment failed — insufficient funds, a bank-side rejection, or a timeout on the provider's end. Immediate technical effect: this is Vector J. While the payment is pending, the settlement is recorded in a pending state that does not reduce Cara's confirmed balance — Asha still shows Cara owing the full pre-repayment amount — specifically because a pending payment is never treated as settled. When the failure confirmation arrives, the pending settlement is marked failed, not reinterpreted as a late success, and still never reduces the confirmed balance.

User-visible effect: at every point in this sequence, Asha's confirmed balance (Cara owes ₹400 total per Vector B's prior state) is undisturbed; only once a payment is provider-confirmed as successful would the balance update. Cara sees her own payment attempt listed as pending, then failed, with a clear next step (retry payment) rather than a confusing state where the group's balance moved and then had to move back. How to detect it: a payment-state-transition metric tracking pending → failed rates per provider and per payment method, and an assertion that no confirmed-balance-affecting ledger entry exists for any settlement whose payment state is not 'succeeded.'

Containment: no halt; this is normal, expected payment-provider behavior that the settlement state machine is explicitly designed to handle. Recovery sequence: Cara is notified of the failure and can retry the payment, which creates a new payment attempt (its own idempotency-key-guarded write) rather than resurrecting the failed one; if the pending state had, due to a bug, already reduced the confirmed balance, the fix is a versioned reversal, never a silent balance patch. How to prevent recurrence: model settlement status as an explicit, versioned state machine (initiated → pending → succeeded/failed), and gate any confirmed-balance ledger effect strictly on reaching the succeeded state via the provider's own authoritative confirmation, never on the mere existence of a pending attempt.

Scenario 21 — Payment succeeds but the confirmation callback is lost

Trigger: Cara's ₹200 payment actually succeeds at the provider, but the webhook or callback meant to notify Weekend Trip's backend of that success is dropped entirely — a network partition, a backend outage at the exact moment of delivery, or a provider-side delivery failure with no automatic retry configured. Immediate technical effect: without a reconciliation mechanism, the settlement would remain stuck in 'pending' indefinitely in Weekend Trip's own state, even though the provider's own records show it succeeded — a disagreement between the two systems that nothing would ever resolve on its own.

User-visible effect without reconciliation: Cara's balance never updates despite her bank statement showing the payment went through — a confusing and trust-damaging mismatch between what Cara knows happened and what the app shows. How to detect it: a scheduled reconciliation job that polls the provider's API for the status of any settlement still pending past an expected confirmation window, rather than relying solely on the provider's callback ever arriving.

Containment: no halt; the affected settlement simply remains pending until reconciliation resolves it, with no impact on other settlements or other groups. Recovery sequence: the reconciliation job queries the provider directly, finds the payment actually succeeded, and applies the same success-transition logic the callback would have triggered — including the same idempotency-key check from Scenario 19, since the reconciliation path and the callback path both ultimately drive the same state transition and must not double-apply if the callback arrives late after reconciliation already resolved it. How to prevent recurrence: never treat a provider's push-based callback as the only path to learning a payment's outcome — always pair it with a pull-based reconciliation job as a safety net, and make both paths converge on the same idempotent state-transition logic.

Scenario 22 — Chargeback / refund after a confirmed settlement

Trigger: some time after Cara's ₹200 payment to Asha is confirmed succeeded and the group's balances have updated accordingly, the payment is reversed — a chargeback initiated by Cara's bank, or a refund issued by the provider for an unrelated dispute — arriving as a new, separate event from the provider well after the original settlement was already finalized. Immediate technical effect: the original settlement record is never edited or deleted; the chargeback is applied as a new, explicit, audited reversing ledger entry, dated to when the chargeback was recorded, undoing the settlement's balance effect going forward.

User-visible effect: Weekend Trip's balance updates to reflect that Cara's ₹200 is, in effect, owed again (Asha and Ben's shares adjust back accordingly), but the group's history shows three distinct, timestamped, never-overwritten events — the original expense, the original settlement, and the chargeback reversal — so anyone reviewing the group's history can see exactly what happened and when, never a rewritten past where the settlement simply appears to have never occurred. How to detect it: chargeback/refund webhooks or reconciliation findings from the provider, handled through the same idempotency-guarded inbound-event pattern as any other provider notification (Scenario 19's pattern).

Containment: no halt; the reversal is applied as a scoped, audited ledger entry affecting only this settlement's balance contribution. Recovery sequence: not an outage-style recovery — this is the designed-for handling of a legitimate real-world payment reversal; the group may separately decide, through normal group interaction, how Cara should resolve the now-reopened ₹200, but that resolution is a new, future event, never a retroactive rewrite of the chargeback itself. How to prevent recurrence: treat chargebacks and refunds as first-class settlement lifecycle events with their own reversing-entry semantics from the start, never as something requiring an ad hoc manual balance correction when they occur.

Weekend Trip settles in INR, but a bug could still try to net a foreign-currency amount against it, an exchange rate could shift after the fact, and a scheduled recurring expense's own clock can misfire. These three incidents guard the edges of what 'the ledger' is allowed to assume stays constant.

Scenario 23 — Multi-currency accidental netting attempt

Trigger: a bug — perhaps in a future feature allowing an expense to be recorded in a foreign currency for a member who paid abroad — attempts to sum a USD-denominated expense amount directly against Weekend Trip's INR ledger balance without an explicit conversion step, treating the numeric values as if they were fungible. Immediate technical effect: the ledger enforces that every entry carries an explicit currency code, and any aggregation or netting operation across entries first asserts that all entries share the same currency; an attempt to sum or net entries across different currencies without an explicit, recorded conversion event is rejected as a hard error rather than silently producing a numerically-summed but financially meaningless total.

User-visible effect: Weekend Trip's balances remain correctly denominated entirely in INR; if a USD expense is ever introduced, it must go through an explicit conversion event — a recorded exchange rate and a rate policy (Scenario 24) — before it contributes any value to the INR ledger, and the UI makes that conversion step visible rather than silently blending currencies behind the scenes. How to detect it: a currency-consistency assertion run on every aggregation query, and a hard validation error at write time for any entry whose currency doesn't match the ledger's established currency for that group without an accompanying conversion record.

Containment: the offending write or aggregation is rejected before it can affect any balance. Recovery sequence: none needed for a correctly rejected attempt; if a currency-mixing bug ever let an unconverted value through, the fix is identifying every entry lacking a proper conversion record and correcting them through a versioned reversal and properly converted replacement, never a silent numeric patch. How to prevent recurrence: require an explicit currency code on every monetary field end to end, make cross-currency aggregation structurally impossible without passing through an explicit conversion step, and never let a UI or API default a missing currency field to 'assume same as group currency.'

Scenario 24 — Exchange rate changes after an expense was recorded with an old rate snapshot

Trigger: following Scenario 23's safe path, a foreign-currency expense is properly recorded with an explicit conversion event capturing the exchange rate at that moment; sometime later, the market exchange rate moves, and a question arises — should the expense's INR-equivalent value in Weekend Trip's ledger update to reflect the new rate? Immediate technical effect: the conversion event and the rate it captured are immutable, exactly like any other ledger entry — the INR-equivalent value recorded at the time of the expense is permanent and does not silently re-float with later market-rate movements, since the ledger represents what was agreed and recorded at a specific point in time, not a live valuation.

User-visible effect: the group's INR balances remain exactly what they were computed as using the original recorded rate; if the group wants to apply a different rate (for example, settling at today's rate instead of the original one), that requires a new, explicit, separately audited adjustment event with its own rate policy decision — never an automatic, invisible re-conversion of the original entry. How to detect it: this isn't a failure to detect reactively — it's a policy question settled by design; what's monitored is that every conversion event's rate and timestamp are immutably recorded and that no process ever attempts to mutate a historical conversion's rate in place.

Containment: not applicable as an incident; the immutability of the recorded rate is itself the containment. Recovery sequence: if a rate was recorded incorrectly (a genuine data-entry or integration bug, not a market movement), the correction follows the same versioned-reversal-and-replacement path as any other expense correction, with a new conversion event carrying the corrected rate, never an in-place edit of the original rate value. How to prevent recurrence: document and enforce the rate policy explicitly (e.g., 'rate is locked at time of expense, re-conversion requires an explicit adjustment') so there's never ambiguity about whether a ledger entry's value is expected to float with the market.

Scenario 25 — Recurring expense scheduled job runs twice for the same period

Trigger: Weekend Trip has a recurring expense configured (say, a shared weekly accommodation charge), and the scheduler responsible for generating that period's expense instance fires twice for the same period — a scheduler failover that doesn't coordinate cleanly with the previous instance, or a retry after a job-runner timeout that doesn't actually reflect a failure. Immediate technical effect: the scheduled-job handler derives a deterministic idempotency key from the recurring-expense definition's ID and the specific period it's generating for (for example, 'recurring-expense-42 : week-of-2026-10-12'), and checks that key against the same durable idempotency ledger used for client writes (Scenario 01's mechanism) before creating the period's expense instance.

User-visible effect: exactly one expense instance is created for that week, regardless of how many times the scheduler fired for it — the second firing's idempotency-key check recognizes the period as already-generated and produces no second expense. How to detect it: a scheduler-firing-count metric per recurring-expense-definition-and-period pair, with any count above one investigated even though the idempotency check should make the outcome harmless, since repeated firings can still indicate a scheduler coordination bug worth fixing.

Containment: no halt; resolved inline by the same idempotency mechanism as any other write. Recovery sequence: none needed — the duplicate firing is absorbed cleanly; if a bug ever predates this check and created two instances for one period, the fix is reversing the duplicate instance through the standard versioned-reversal path, never a silent deletion. How to prevent recurrence: derive a deterministic, period-scoped idempotency key for every scheduled-job-generated write, and ensure the scheduler's own failover/retry logic can never assume 'no response yet' means 'never ran' without consulting that same idempotency ledger first.

Not every incident touches the ledger directly — a notification outage, an unavailable receipt store, a database failover, and a backup gap are all operational failures that have to stay scoped to their own layer without ever being allowed to corrupt or lose financial state.

Scenario 26 — Notification service is down while expenses are still being created

Trigger: the service responsible for sending push notifications and activity-feed alerts (for example, 'Asha added a ₹900 expense') experiences an outage, while expense creation, splitting, and ledger writes continue operating normally and independently. Immediate technical effect: because notification delivery is architecturally decoupled from the write path that commits expenses, splits, and ledger entries — notifications are published as their own downstream, best-effort event off the back of an already-committed write, never a precondition for the write to succeed — the outage has zero effect on whether expenses are correctly recorded, only on whether members are promptly told about them.

User-visible effect: Ben and Cara don't receive a push alert about Asha's new dinner expense while the outage lasts, but opening the app shows the expense and updated balances correctly, since reads are served from the ledger/projection, not from whether a notification was ever sent. How to detect it: notification-delivery success-rate and service-health monitoring tracked entirely separately from write-path and ledger-correctness metrics, so a notification outage is never conflated with a financial-correctness incident.

Containment: no halt to expense creation or any ledger-affecting write; notifications queue up or are simply dropped per the notification service's own retry/backoff policy, isolated entirely to that service. Recovery sequence: once the notification service recovers, it can optionally catch up on a backlog of queued notifications, or simply resume sending new ones going forward, depending on how stale a backlog of old alerts would be considered useful; nothing in the ledger or projection ever needs to recover anything, since nothing there was ever affected. How to prevent recurrence: keep notification delivery as a downstream, best-effort consumer of already-committed events, never a dependency the write path blocks on, and monitor it independently so its outages are visibly scoped to 'members weren't told promptly,' never conflated with 'the expense wasn't recorded.'

Scenario 27 — Receipt storage is unavailable when a user tries to attach a receipt

Trigger: Ben tries to attach a photo of the taxi receipt to his ₹600 expense at the exact moment the object-storage service backing receipt uploads is unavailable — a storage-provider outage, a network partition to that specific dependency, or a quota/throttling error on the storage side. Immediate technical effect: the expense-creation and receipt-attachment paths are deliberately separate operations; the expense itself (amount, split, ledger effect) can commit successfully and independently of whether its optional receipt attachment succeeds, since a receipt is supplementary evidence, not a required field for the expense's financial correctness.

User-visible effect: Ben's ₹600 taxi expense is recorded correctly with its full split and ledger effect; the receipt-attachment step fails with an explicit, retryable error, and Ben can retry attaching the receipt later (to the same, already-created expense) without needing to recreate or edit the financial details of the expense itself. How to detect it: receipt-upload error-rate and storage-service health metrics, monitored separately from expense-write success rate, so a storage outage is never misattributed as a broader write-path failure.

Containment: no halt to expense creation; receipt attachment simply fails independently and is retryable on its own. Recovery sequence: once storage recovers, Ben (or anyone with edit access) retries the receipt-attachment step against the already-existing expense; this is treated as its own idempotent operation (same pattern as Scenario 01) so a retry of the attachment itself can't create duplicate receipt records. How to prevent recurrence: architect receipt storage as a decoupled, optional dependency of expense creation, never a transactional precondition for it, and make the attach-receipt operation independently retryable and idempotent.

Scenario 28 — Database primary failover during a write

Trigger: the primary database node handling a write — say, Asha's ₹900 dinner expense commit — fails or is forcibly failed over to a replica mid-transaction, due to a hardware fault, an operator-initiated maintenance failover, or a network partition isolating the primary. Immediate technical effect: a transaction that had not yet committed at the moment of failover is simply never considered to have happened — the new primary (promoted replica) either has the transaction if it was durably committed and replicated before the failover, or does not have it at all; there is no in-between, partially-applied state, because the database's own transactional durability guarantee (not applied to clients until committed and acknowledged) is what Splitwise's write path relies on.

User-visible effect: if the commit hadn't been acknowledged to Asha's client before the failover, her client treats the request as failed or uncertain and retries it (idempotency-key-guarded, per Scenario 01), safely producing exactly one committed expense regardless of whether the original attempt actually landed just before the failover or not at all. If the commit had already been acknowledged, the new primary has it, and nothing is lost. How to detect it: database failover events are themselves alertable infrastructure events, and the write path's own client-side timeout/retry behavior around a failover window is monitored for an elevated retry rate, which is expected and handled, not alarming on its own.

Containment: in-flight writes during the failover window either complete cleanly against the new primary or fail cleanly and get retried; no write is left half-applied, because failover itself doesn't preserve partial transaction state — the database's atomicity guarantee covers exactly this case. Recovery sequence: the client's normal idempotency-key retry logic (Scenario 01) resolves any uncertain-outcome write automatically, without any special-cased 'post-failover reconciliation' logic needed beyond what idempotency already provides. How to prevent recurrence: rely on the database's own documented durability and failover guarantees rather than attempting to build weaker, ad hoc assumptions about partial-commit visibility, and ensure every client-facing write path is already idempotency-key-guarded so a failover-induced retry is indistinguishable from an ordinary network-timeout retry.

Scenario 29 — Backup restore is missing the most recent writes (RPO gap)

Trigger: following a catastrophic primary-and-replica loss, the team restores from the most recent backup, but that backup predates the actual point of failure by some interval — a stated recovery-point-objective (RPO) gap — because backups are taken on a schedule rather than continuously, or the continuous replication stream feeding a point-in-time-recovery target was itself interrupted shortly before the failure. Immediate technical effect: any expense, split, settlement, or ledger entry committed after the backup's point-in-time but before the failure is genuinely absent from the restored database — a real, honest data gap, not a silently hidden one, bounded by the stated RPO.

User-visible effect: members whose writes fell inside the gap (for example, a settlement Cara recorded minutes before the failure) see those writes missing after the restore, and the group is explicitly notified of the restore point and the specific window at risk, rather than silently presenting a restored-but-incomplete history as if it were complete. How to detect it: comparing the restored database's latest committed transaction timestamp against the outage's actual failure timestamp makes the gap's exact size immediately knowable, and any outbox/event-log replication that was running independently (Scenario 09's pipeline) is checked for whether it preserved a more complete record than the primary's own backup did.

Containment: the restored database is brought up read-only or clearly labeled while the gap is being assessed, rather than immediately treated as if it were fully caught up. Recovery sequence: if the durable event bus or outbox log retained events from within the gap window (because it's replicated independently of the primary database, per Scenario 11's durability model), those events are replayed forward into the restored database to close some or all of the gap; whatever cannot be recovered this way is a genuine, documented, communicated loss bounded by the stated RPO, never silently absorbed into the restored state as if it had always been there. How to prevent recurrence: state an explicit RPO target up front, back it with continuous replication or frequent incremental backups rather than relying on infrequent full snapshots alone, and keep the durable event log/outbox replicated independently of the primary database specifically so it can serve as a second recovery source with a tighter effective RPO than the database's own backup schedule.

Weekend Trip is a three-person group, but the same design has to hold up for a group with thousands of expenses, a sudden surge of new signups, and a dashboard that has to summarize a user's entire financial life across many groups at once. These five incidents are about load, not correctness per se — though every containment and recovery here still has to respect the correctness invariants above.

Scenario 30 — A hot group with an unusually large number of expenses causes query slowness

Trigger: an atypical group — a long-running shared household, or a group used as an informal shared-expense tracker over years — accumulates an unusually large number of expenses (tens of thousands), far beyond a typical trip group like Weekend Trip, and a query touching that group's full expense history or recomputing its balance from scratch starts taking noticeably longer than a normal group's equivalent query. Immediate technical effect: a query pattern that scans or aggregates proportionally to the group's total expense count — rather than using an already-maintained, incrementally updated balance projection, or a properly indexed, paginated history query — degrades roughly linearly (or worse, if poorly indexed) with that one group's size, while every other, smaller group's queries remain unaffected.

User-visible effect: members of the unusually large group experience slower balance loads or slower history scrolling than members of a typical trip group, without any other group being affected at all — a scoped, group-specific slowdown rather than a systemic one. How to detect it: per-group query-latency percentiles, with an outlier-detection alert on any single group whose query latency diverges sharply from the overall distribution, correlated against that group's expense count.

Containment: no halt to other groups; the slow query path is identified and scoped to the one hot group. Recovery sequence: ensure the balance view is served from the incrementally maintained projection (never recomputed from the full raw ledger on every read), add or correct indexing on the expense-history query's filter and sort columns, and paginate history queries rather than ever loading a group's full expense list in one response. How to prevent recurrence: design every per-group read path to scale with the data actually being displayed (a page of recent history, a pre-maintained balance) rather than the group's total historical size, and load-test against an intentionally large synthetic group during development, not only against typical-sized ones.

Scenario 31 — Viral user growth causes a sudden 10× traffic spike

Trigger: a feature goes unexpectedly viral — a social-sharing moment, a press mention, or a sudden spike in new sign-ups inviting friends into groups — and overall platform traffic (sign-ups, group creation, expense writes, and balance reads together) spikes roughly 10× above baseline within a short window. Immediate technical effect: every layer of the write and read path — API servers, the database, the outbox relay, the projector, and any cache tier — sees proportionally higher load simultaneously, risking saturation at whichever layer has the least headroom first, typically the database's write capacity or the API tier's connection pool.

User-visible effect without admission control: API latency could degrade broadly, and in the worst case, write timeouts could increase, which (per Scenario 01) the client's idempotency-key retry logic handles safely but which is still a degraded experience; without capacity management, this could look to users like widespread slow loading rather than a correctness problem. How to detect it: aggregate request-rate, API latency percentiles, and database connection-pool saturation crossing high-water marks together, distinguished from Scenario 30's single-group-scoped slowness by being broad and simultaneous across many groups.

Containment: admission control/rate-limiting at the API tier to protect the database from being overwhelmed — never silently dropping a write, since a throttled client gets an explicit, retryable rejection rather than risking a silently lost request — and horizontal auto-scaling of stateless API servers to absorb read traffic. Recovery sequence: once traffic subsides or additional capacity is brought online, normal latency returns; a post-incident check confirms every accepted write was durably committed exactly once via the idempotency ledger, with no retry-induced duplicates. How to prevent recurrence: provision capacity headroom above an assumed peak multiplier (see Scale Walkthroughs), make API servers horizontally scalable statelessly, and ensure the database and outbox/projection pipeline can absorb a sustained multiplier of normal write volume without requiring a redesign under pressure.

Scenario 32 — A dashboard query fans out across a user's many groups inefficiently

Trigger: a user who belongs to an unusually large number of groups (dozens of trip groups, household groups, and recurring-event groups accumulated over years) opens a 'total balance across all my groups' dashboard view, which has to gather that user's current balance from every one of those groups before it can compute and display an aggregate. Immediate technical effect: if the dashboard query fetches each group's balance with a separate round-trip, or worse, recomputes each group's balance from its raw ledger rather than reading its maintained projection, the total latency for one user's dashboard load scales with that user's group count — a fan-out cost structurally similar to a very-large-friends-list rank query in other domains.

User-visible effect: users with many groups experience a noticeably slower dashboard load than users with just one or two groups, proportional to their group count rather than to any shared platform load. How to detect it: dashboard-load latency correlated against the requesting user's group count, and a fan-out-size distribution tracked so an unusually high group count — or a bug inflating the apparent fan-out — is visible before it causes a latency outlier.

Containment: batch the per-group balance fetches into a single, properly indexed query (keyed by user ID across the balance-projection table) rather than issuing one round-trip per group, and cap or paginate how many groups contribute to a single dashboard render for pathologically large counts. Recovery sequence: not an outage — a latency-tail concern; if a specific user's fan-out proves consistently too slow even after batching, the fix is a precomputed, incrementally maintained per-user aggregate balance view, updated as any of their groups' balances change, rather than recomputing the aggregate from scratch on every dashboard load. How to prevent recurrence: design the dashboard query to read from the already-maintained per-group balance projections via one batched query from the start, and keep the per-group balance projection itself as the one place balance correctness logic lives, so the dashboard is a cheap aggregation layer on top of it, never a parallel recomputation path.

Scenario 33 — The event queue (bus) backs up and grows unbounded

Trigger: the durable event bus carrying ledger events from the outbox relay to the balance-projector consumer experiences sustained producer-outpacing-consumer imbalance — a burst of expense activity across many groups simultaneously, or a sustained projector slowdown (Scenario 12) that never catches back up — causing the queue's backlog to grow continuously rather than draining within any bounded window. Immediate technical effect: an unbounded backlog risks the bus's own storage capacity being exhausted, which, depending on the bus's configuration, could mean either rejecting new publishes (backpressuring the outbox relay) or, worse, silently evicting the oldest unconsumed messages if retention is misconfigured — the latter being an unacceptable, undetected ledger-to-projection data loss risk if it were ever allowed to happen.

User-visible effect: if the bus backpressures the relay rather than dropping messages, ledger writes themselves remain unaffected (the outbox pattern still durably records every event in the primary database regardless of bus health) but projection staleness grows further behind than Scenario 12's normal-lag case, an escalating version of the same symptom. How to detect it: bus queue-depth and retention-utilization metrics crossing high-water marks, which is a more severe and more urgent version of the projector-lag alert from Scenario 12, since it threatens actual message loss if retention is exceeded, not merely staleness.

Containment: the bus's retention policy is configured generously enough, and monitored closely enough, that it never actually reaches the point of evicting unconsumed messages — if queue depth approaches that threshold, scale out projector consumers immediately and, if needed, throttle new-expense admission at the write path as a last resort to buy the queue time to drain, always preferring to slow writes over risking event loss. Recovery sequence: scale projector consumer parallelism to drain the backlog faster than it's growing, and once drained, confirm via reconciliation (Scenario 14's check) that every event was applied exactly once with nothing lost. How to prevent recurrence: size bus retention against a worst-case projector-outage duration, not just normal operation, provision projector throughput with headroom above assumed peak event rate, and treat 'queue depth approaching retention limit' as a more urgent escalation tier than ordinary lag monitoring.

Scenario 34 — Debt simplification is run on a very large group (many members, many balances)

Trigger: a group far larger than Weekend Trip — say, a hundred-member event or club group with a correspondingly large and tangled set of pairwise balances — requests a debt-simplification view, which has to compute a reduced set of suggested transfers that nets everyone's balance to zero using as few transfers as reasonably possible. Immediate technical effect: simplification algorithms that minimize the number of transfers exactly are computationally expensive as member count grows, so at large member counts the system uses a bounded-cost heuristic (for example, greedily matching the largest creditor against the largest debtor repeatedly) that produces a good, net-position-preserving simplification quickly, rather than an expensive exact-minimum-transfer-count solution that could degrade response time unpredictably as the group grows.

User-visible effect: the suggested simplified transfers for a very large group may use a few more transfers than a theoretically optimal minimum would, but every suggested transfer set still nets every member's position to exactly zero relative to their pre-simplification balance — the one property that must never be sacrificed for speed. Simplification output is always clearly presented as a suggestion, never a forced or legally binding transfer. How to detect it: simplification-computation latency tracked against group size/member count, and a post-computation assertion that the suggested transfer set's net effect per member exactly matches their pre-simplification balance, run on every simplification result regardless of group size.

Containment: no halt; for very large groups, the heuristic path is used automatically rather than attempting the expensive exact computation and risking a slow or timed-out response. Recovery sequence: not an outage — a performance/quality trade-off; if the heuristic's transfer count proves unacceptably high for some large groups, the fix is a better heuristic or a bounded-time exact solver with a hard cutoff, never relaxing the net-position-preserving correctness check itself. How to prevent recurrence: choose a simplification algorithm with an explicit, bounded time complexity as a function of member count from the start, validate every simplification result's net-position-preservation as a mandatory post-computation check regardless of which algorithm path was used, and never present simplification's suggested transfers as anything other than optional and non-binding.

The last two incidents are the worst-case tier: a ledger record itself found corrupted or missing, and a full regional outage requiring Weekend Trip's data to be recovered somewhere else entirely. Both depend on the same foundation every other scenario in this module has been reinforcing — a durable, replicated, replayable ledger that projections and balances can always be rebuilt from.

Scenario 35 — Ledger event corruption or a missing record is discovered

Trigger: a routine reconciliation check (Scenario 14's job) or a manual audit discovers that one specific ledger event — say, the original ledger entry for Ben's ₹600 taxi expense — is either corrupted (a field that doesn't parse, or a stored amount that doesn't match its own recorded splits) or appears to be missing entirely relative to what the audit trail's surrounding sequence numbers imply should exist. Immediate technical effect: this is treated as the most serious class of incident in the system, since it threatens the one thing every other recovery path in this module depends on — the ledger's own trustworthiness as the authoritative source of truth; the affected record and any projection derived from it are immediately flagged as unverified pending investigation, rather than continuing to serve reads from potentially-corrupted downstream state as if nothing were wrong.

User-visible effect: the affected group's balance view may be marked as under review while investigation proceeds, rather than confidently displaying a balance that might be computed from corrupted or missing input; once the investigation concludes, either the original record is restored from a verified replica or durable backup (if the corruption was isolated to one copy and a clean copy exists elsewhere), or, if genuinely unrecoverable, an explicit, clearly documented and audited gap-acknowledgment entry is recorded — never a silently fabricated replacement value standing in for data that can't actually be verified. How to detect it: continuous checksum or structural-integrity validation on ledger records, cross-replica consistency checks (comparing the same record across replicated copies), and the same reconciliation job from Scenario 14, extended to flag not just projection-vs-ledger drift but internal ledger-record integrity issues.

Containment: the specific affected group (or, in the worst case, the affected shard of groups) is isolated for investigation; unaffected groups continue operating normally, since the blast radius is scoped to wherever the corruption or gap was actually found, confirmed by checking other records near it rather than assumed to be limited. Recovery sequence: restore the affected record from the least-corrupted available source — a healthy replica, a recent backup, or, if the event was also captured in the durable outbox/event-bus stream before corruption occurred in the primary store, replay it from there; recompute and verify every downstream projection depending on the restored record; if no recoverable copy exists anywhere, record the gap explicitly and permanently in the audit trail rather than guessing at a replacement value, and communicate the specific gap to the affected group. How to prevent recurrence: run continuous integrity validation in production rather than only during scheduled audits, maintain multiple independently verifiable copies of every ledger record (replication plus the separately durable event stream), and treat any detected integrity violation as a top-severity incident regardless of how small the affected scope appears to be.

Scenario 36 — Region-wide outage requiring full disaster recovery

Trigger: a catastrophic infrastructure event — a full data-center or cloud-region outage — takes down the primary database, the outbox relay, the projector, and the cache tier simultaneously, well beyond any single-component failure covered elsewhere in this module. Immediate technical effect: there is no fast same-region failover option left; recovery depends entirely on whatever was replicated to a separate region ahead of time — a cross-region database replica or recent backup, and, critically, the durable event log/outbox stream, if it was independently replicated cross-region, which lets any gap between the database replica's last-known state and the actual point of failure be closed by replaying events forward.

User-visible effect: the platform is explicitly unavailable for both reads and writes for a genuinely longer, honestly communicated window while failover and verification proceed — never a silent partial service serving possibly-incomplete or incorrect balances in the meantime. How to detect it: region-health monitoring from outside the affected region (since the affected region's own monitoring can't be trusted to self-report during its own outage), confirmed independently before declaring a disaster-recovery event rather than failing over prematurely on a transient blip.

Containment: the platform is marked unavailable explicitly rather than attempting to serve degraded or partial service from a possibly-incomplete failover target; no write is accepted anywhere until a new authoritative region is confirmed consistent. Recovery sequence: promote the cross-region database replica (or restore from the most recent cross-region backup) as the new authoritative primary; replay any event-log/outbox entries that exist beyond that replica's last-known state to close the gap, exactly as in Scenario 29's RPO-gap recovery, just at a larger scale; run full invariant checks across every group — zero-sum balances, no orphaned splits, every expense's splits summing to its total — before reopening writes; reopen service and communicate the specific recovery point and any residual, bounded data-loss window to affected users. How to prevent recurrence: replicate both the primary database and the durable event log/outbox stream across regions, not just one or the other, state explicit RPO and recovery-time-objective (RTO) targets for a full regional failure up front, and rehearse full cross-region failover drills regularly rather than discovering gaps in the plan only during a real outage.

The 36 scenarios above describe what breaks; this closing chapter puts illustrative numbers on how the design behaves as Weekend Trip's pattern is multiplied across far more users and groups. Every number below is a stated assumption for this educational exercise, not a verified production benchmark for Splitwise or any real shared-expense service.

Growth from 100k to 1M to 10M active users

Assumption: start from an illustrative baseline of 100,000 monthly active users, each belonging to an average of 3 active groups and recording an average of 2 expenses per week. That's roughly 100,000 × 2 ÷ 7 ≈ 28,600 expense-writes/day at baseline. Growing to 1 million active users (10×) scales that proportionally to roughly 286,000 expense-writes/day, and growing to 10 million (100× baseline) scales it to roughly 2.86 million expense-writes/day. None of these per-user behavioral assumptions (3 groups, 2 expenses/week) are measured — they are illustrative inputs chosen to make the arithmetic concrete.

Peak write amplification: splits and ledger events per expense

Assumption: a single expense write is never just one row — illustratively, one expense header row, an average of 3 split rows (matching Weekend Trip's 3-member group size as a representative average), 1 ledger-affecting outbox event, and 1 activity-feed entry, for roughly 6 downstream writes/events per committed expense. At the 10M-MAU, 2.86M-expenses/day assumption above, that's roughly 2.86M × 6 ≈ 17.2 million total row/event writes per day attributable to expense creation alone, before settlements, edits, or reversals are added on top. This amplification factor is exactly why the atomic-transaction boundary in Scenario 02 and the outbox pattern in Scenarios 09–11 matter more as volume grows, not less.

Read-to-write ratio

Assumption: a shared-expense app is typically read-dominant day to day — illustratively, members check group balances and activity feeds on the order of 20–40× more often than they create a new expense or settlement, since checking 'who owes what' is a much lower-friction action than recording a transaction. At the 10M-MAU / 33 expense-writes/sec average assumption above, a 30× read-to-write ratio implies roughly 990 balance/history reads/sec on average — the exact workload shape that makes the cache-and-projection discipline from Scenarios 12–14 the dominant day-to-day cost driver rather than the write path itself.

Hot-group skew and cache effectiveness

Assumption: group activity is never evenly distributed — illustratively, assume the top 1% of groups (an active household, a long-running club group) account for roughly 20% of total read traffic, a Pareto-style skew typical of social/collaborative apps. A short-TTL cache in front of the balance projection, illustratively sized to a 30-second refresh window, could absorb the vast majority of that hot-group read volume, since a 30-second staleness window matters far less for a passive balance check than it would for, say, a live financial trading feed. The remaining long tail of low-traffic groups sees proportionally lower cache hit rates but also contributes far less aggregate load, making a uniform cache policy reasonable rather than needing per-group tuning.

Group partitioning and database sharding

Assumption: because almost every read and write is naturally scoped to one group (an expense, a split, a settlement all belong to exactly one group), partitioning the database by group ID is a close-to-ideal sharding key — illustratively, sharding 10 million active users' groups across, say, 64 database shards by a hash of group ID would put an average of roughly 1.5 million users' worth of group data per shard (since users with multiple groups may span shards), with cross-shard queries needed only for user-level dashboard aggregation (Scenario 32), never for any single group's own reads or writes. This is a materially simpler partitioning story than a single-global-leaderboard-style system, precisely because group scope is a natural, almost-always-respected boundary.

Projection rebuild time at scale

Assumption: if a projector can replay, illustratively, 5,000 ledger events/sec during a rebuild (Scenario 14's or Scenario 35's recovery path), and a single very active group has accumulated, illustratively, 50,000 lifetime ledger events (the Scenario 30 hot-group case), rebuilding that one group's projection from scratch would take roughly 50,000 ÷ 5,000 = 10 seconds — fast enough to be a routine, low-drama recovery operation. Rebuilding every group's projection platform-wide after a Scenario 36-style full regional disaster, at the 17.2M-events/day volume assumption above accumulated over, say, a illustrative 2-year retention window (roughly 12.5 billion events), would take roughly 12.5 billion ÷ 5,000 ≈ 2.9 million seconds (~34 days) if done by a single rebuild stream — which is exactly why a real disaster-recovery plan would parallelize rebuild across many concurrent group-scoped streams rather than ever relying on one sequential replay, and why the RTO target in Scenario 36 must be set with that parallelization capacity explicitly in mind.

Notification fan-out cost

Assumption: each committed expense illustratively triggers a notification to every other active member of its group — for a representative 3-member group like Weekend Trip, that's 2 notification sends per expense (to the 2 non-submitting members). At the 2.86M-expenses/day (10M-MAU) assumption above, that's roughly 5.72 million notification sends/day, or an average of roughly 66 sends/sec — a workload that, per Scenario 26, is entirely decoupled from the write path and can be scaled or degraded independently without ever risking the correctness of the underlying expense record.

Database indexing strategy

Assumption: the two query shapes that dominate read volume — 'current balance for group X' and 'recent expense history for group X, paginated' — are illustratively served by an index on (group_id, created_at) for history pagination and a maintained, separately-indexed balance-projection table keyed directly by group_id for balance reads, so neither query shape ever requires a full-table scan or an unindexed aggregation regardless of total platform size. The authorization checks from Scenarios 15–18 (is this user an active member of this group) are illustratively served by an index on (group_id, user_id, status) on the membership table, since that check runs on effectively every single request touching group data.

RPO/RTO targets

Assumption: for an illustrative, non-binding example of how these targets might be stated: a recovery-point objective of a few seconds for the primary database (via continuous or near-continuous replication) and a recovery-time objective of under an hour for a single-component failure (Scenario 28's style), versus a looser illustrative RPO of a few minutes and an RTO of several hours for a full regional disaster (Scenario 36), reflecting that cross-region recovery genuinely takes longer to verify safely than a same-region failover. These numbers are stated here purely as an illustration of the kind of target a real design document would commit to explicitly — never as an actual SLA for any real system.