Architecture

Incident & Postmortem Analysis

Incidents expose where the real architecture, operating model, and assumptions differ from what the team expected. A strong postmortem maps the causal chain, identifies systemic weaknesses, and turns the incident into better prevention, containment, detection, and recovery.

Mangesh MandlikLast updated: 2 November 2026

What Is Incident & Postmortem Analysis? Blameless Postmortem Basics

Incident postmortem, causal analysis, and systemic corrective actions

Incident analysis is the practice of understanding not just what failed in production, but why the system was capable of failing that way. A postmortem, or post-incident review, turns that understanding into concrete follow-up so the organisation learns from the event instead of merely surviving it.

A blameless postmortem does not mean no accountability. It means avoiding hindsight-driven personal blame while still examining whether decisions were reasonable given the information, constraints, and signals available at the time. You still assign owners, improve weak decisions, and close gaps in process, testing, architecture, and operations.

Blameless

Focus on system conditions, decision context, and evidence rather than punishing individuals for good-faith actions.

Multiple causes

Complex incidents often have branching causal chains, several contributing factors, and amplifiers that worsened impact.

Action items

Corrective actions can improve prevention, containment, detection, recovery, testing, runbooks, or ownership clarity — not just architecture diagrams.

Visual Story

Normal → trigger → cascade → detection → postmortem fix applied

Incident & Postmortem Analysis

Pre-Incident

0 / 0

⚠️Up next14:32 — Trigger

⤢ Expand for synchronized narration timeline

Scene 5 shows the same trigger happening again, but a circuit breaker now contains it. The value of the postmortem is not the document; it is the changed system behaviour.

Problem It Solves

Why incidents recur without structured analysis

Without structured postmortems, teams often fix the immediate symptom and move on. Restarting a service may restore traffic, but it does not explain why the service became vulnerable to a slow dependency, why the issue was not detected sooner, or why recovery still required manual intervention.

Postmortem analysis solves this by separating the trigger from the deeper conditions that allowed the incident to spread, identifying which gaps were architectural versus operational, and producing follow-up that addresses the whole failure class rather than a single visible symptom.

Incident Analysis Vocabulary

Use precise terms for what happened and why it mattered

Trigger

The specific event that started the incident, such as a bad deploy, a dependency slowdown, or a failover.

Proximate cause

The most immediate technical cause of the failure users experienced.

Contributing factors

Other conditions that made the incident more likely or worse, even if they were not the primary trigger.

Latent or systemic conditions

Pre-existing weaknesses that existed before the incident and were only exposed when the trigger occurred.

Impact amplifier

Something that increased blast radius or severity, such as shared infrastructure, broad retries, or missing bulkheads.

Detection gap

Why the team did not know about the issue sooner, including missing telemetry, noisy alerts, or unclear ownership.

Recovery gap

Why recovery took as long as it did after detection, such as weak runbooks, missing automation, or risky rollback steps.

Five Whys Root Cause Analysis

One useful technique, not the whole postmortem methodology

Five Whys is a root cause analysis technique that asks “why?” repeatedly, starting from the symptom. It can help peel back layers from symptom to deeper condition, but complex incidents rarely follow one perfect straight line. Branching causes, parallel failures, and impact amplifiers often matter just as much as the neatest single chain.

Why #	Observation
1	Checkout was unavailable for 23 minutes.
2	The checkout service ran out of database connections.
3	A slow third-party payment API held connections open.
4	There was no timeout on the payment API HTTP client.
5	The service was never reviewed for missing timeouts before production.

In this example, the why-chain highlights missing timeouts and review gaps. A fuller postmortem would also ask whether connection-pool isolation, fallback behaviour, third-party dependency handling, alerting, and rollback options contributed in parallel.

How Architects Think

Look for branching causes, not just a single culprit

Some incidents look obvious in hindsight, but many genuinely surprising incidents were not reasonably predictable in advance from the information teams had at the time. Good postmortems avoid pretending perfect foresight was available. Instead, they ask whether the decisions, safeguards, and escalations were reasonable for the evidence that was actually visible before the failure.

Corrective actions should match the gap they address. Sometimes the right answer is an architectural change such as a timeout, circuit breaker, isolation boundary, or idempotent workflow. Sometimes it is better detection, a clearer runbook, a test that reproduces the failure mode, or clearer ownership so recovery happens faster next time.

What makes a postmortem action item systemic rather than superficial?

A systemic action item removes or materially reduces the underlying condition that made this class of failure possible, so it is likely to prevent similar-but-not-identical incidents in the future. A superficial action item only patches the exact symptom from this incident, leaving the broader weakness intact. For example, adding a timeout to one payment client may be superficial if twelve other outbound clients still have no timeout; creating a shared client standard, test coverage, and review check for outbound timeouts is more systemic because it changes the class of risk.

What would change your decision? If the same failure class keeps recurring despite prior fixes, revisit whether those fixes targeted prevention, containment, detection, or recovery at the right level of abstraction. That feedback loop connects directly to Reliability & Resilience and future Architecture Reviews.

Illustrative Scenario

Illustrative checkout outage caused by a slow dependency

Illustrative timeline: 14:32 — a payment provider begins responding slowly (P99 rises to roughly 8s from about 120ms). 14:33 — checkout worker threads block waiting for payment responses. 14:35 — the connection pool is exhausted; checkout starts returning 503s. 14:41 — the on-call engineer is paged. 14:55 — the team disables one payment path and partial service returns. 15:10 — the provider rolls back the bad configuration and recovers (the same time the Visual Story shows).

Illustrative impact: the impact window runs 14:32–14:55, about 23 minutes, with checkout returning 503s from 14:35 until partial service returns at 14:55 (about 20 minutes of full unavailability). In this illustrative scenario, the team assumes roughly $6,000 of lost orders per minute, which gives an estimate of about $140,000 (23 min × $6,000 ≈ $138,000), and a 100% customer-facing error rate during the worst part of the outage.

Possible corrective actions: (1) timeouts on external calls, (2) circuit breakers or bulkheads around the payment integration, (3) a dedicated connection pool or isolation boundary for payment work, (4) lower alert thresholds for saturation signals, and (5) an incident drill or test that simulates slow third-party dependencies before peak traffic.

Trade-offs

What postmortems optimise for

Operational fix vs systemic fix

Updating a runbook or pager playbook can improve recovery immediately. Architectural, testing, or ownership changes may take longer but reduce the likelihood or blast radius of the same failure class returning in a different form.

Depth of analysis vs time to finish the postmortem

A review conducted while data and recollection are still fresh is useful, but rushing to a simplistic answer can miss important contributing factors. Aim for timely analysis without collapsing complexity into a single storyline.

Public summary vs internal candor

Customer-facing summaries need clarity and trust. Internal postmortems need enough detail to expose decision context, systemic weaknesses, and uncomfortable trade-offs honestly.

Blamelessness vs accountability

Blameless analysis avoids hindsight-driven punishment for good-faith decisions. It still requires named owners, careful decision analysis, and follow-through on corrective actions.

What Can Go Wrong

Postmortem anti-patterns

Blame-focused postmortem

Focuses on individual mistakes instead of the system conditions and decision context that made the mistake consequential.

Only monitoring as action item

Improves detection only. Sometimes that is valuable, but it does not automatically prevent or contain the next failure.

Postmortem theater

Document goes through the motions, action items are not tracked, and the same failure class returns months later.

Stopping at Why 1

Postmortem identifies the visible outage symptom but never reaches the proximate cause, contributing factors, or latent conditions.

No ownership of action items

Actions are assigned to “the team” with no named owner, due date, or review checkpoint.

Fixing the instance, not the class

One narrow symptom is patched, but similar dependencies, workflows, or review gaps remain unchanged elsewhere in the system.

When To Use / When Not To Use

✓ A full postmortem is especially valuable when

→ Customers or internal users experienced meaningful impact
→ An SLO, error budget, or compliance commitment was breached
→ Recovery required unusual coordination, improvisation, or manual mitigation
→ The same class of incident has recurred or nearly recurred

⚠️ Scope the effort carefully when

→ The event was low impact and a lightweight review will capture the learning
→ The same obvious follow-up is already underway and deeper analysis adds little new signal
→ Teams confuse “blameless” with “no owner” or “no accountability”
→ The review is used to write a neat story instead of an evidence-backed one

Architect's Challenge

A postmortem reveals that a checkout outage was caused by a slow payment API exhausting the thread pool. The team writes an action item: 'Add monitoring for payment API latency'. Is this the right architectural response?

Remember· 30-second mental model

A good postmortem explains how the incident unfolded, why the system allowed it, and which corrective actions will reduce the broader failure class next time.

1Five Whys is one useful technique, but complex incidents often have branching causes and multiple contributing factors.
2Separate trigger, proximate cause, contributing factors, latent conditions, impact amplifiers, detection gaps, and recovery gaps.
3Blameless analysis avoids hindsight-driven punishment while still examining decisions, ownership, and accountability.
4Corrective actions can improve prevention, containment, detection, recovery, testing, runbooks, and ownership clarity — not just architecture structure.
5A systemic action item changes the conditions that made the failure possible, not only the exact symptom seen this time.

Key lesson

‘Add monitoring’ may be a useful supporting action. It becomes insufficient when the real gap is that one slow dependency can still take down the whole service.