Incident Management & Postmortems

Deep-Dive

INCIDENT MANAGEMENT REFERENCE

Click any stage to explore concepts, templates, and production-tested guidance.

🚨

Severity Levels (SEV1–4)

A shared vocabulary for "how bad is this" that decides who gets paged, how fast, and whether the CEO needs to know before breakfast.

Must Know

Must Know▾

Level	Definition	Response
SEV1	Full outage or major data loss risk, all users affected	Page immediately, IC assigned, exec comms
SEV2	Significant degradation, subset of users/features affected	Page on-call, IC assigned, internal status updates
SEV3	Minor degradation, workaround exists, low user impact	Ticket + async investigation, no page
SEV4	Cosmetic or negligible impact	Backlog, next sprint

Getting Severity Wrong

🔴Under-declaring: calling a full checkout-flow outage a SEV3 because "it's probably fine" — the single most common way response gets delayed

🟠Over-declaring: every SEV1 trains the team to stop trusting the page — reserve it for what actually meets the bar

📟

Alert Routing & Escalation

An alert nobody sees isn't detection — it's a log line. Routing and escalation policy is what turns a Prometheus rule into a phone ringing.

Important

Important▾

Escalation policy (Grafana OnCall style)

yaml

escalation_chain: billing-ops-primary

steps:

- {notify: on-call-primary, wait: 5m}

- {notify: on-call-secondary, wait: 5m}

- {notify: team-lead, wait: 10m}

- {notify: engineering-manager}

Common Tools

🔵PagerDuty / Opsgenie: commercial, mature escalation + scheduling

🔵Grafana OnCall: open-source, integrates natively with Grafana/Alertmanager

🎯

Golden Signals & Detection Sources

Where the first sign of trouble actually comes from — and why "a customer tweeted" is a detection-gap incident all by itself.

Monitoring

Recommended▾

Detection Sources, Ranked

🟢Golden-signal alerting (latency, traffic, errors, saturation) — fastest, cheapest

🔵Synthetic monitoring — catches issues before real users hit them

🟠Customer reports / support tickets — means every automated layer above already failed

Closing the Gap

1

Every postmortem should log which of these three caught it

2

A customer-report-first incident is itself an action item: add the missing alert

🎖️

The Incident Commander Role

One person owns the call. Not the smartest engineer in the room — the person coordinating everyone else so the smartest engineer can actually think.

Must Know

Must Know▾

What the IC Does (and Doesn't Do)

🔴Owns the decision to declare, escalate, or resolve — even without full information

🔴Coordinates who's investigating what — prevents three people debugging the same log line

🟠Doesn't have to be the one typing the fix — IC is a coordination role, not the top debugger

💬

War Room Practice

A dedicated channel, a scribe capturing the timeline in real time, and clear roles — so the postmortem writes itself instead of being reconstructed from memory three days later.

Important

Important▾

Slack incident bot pattern

slack

/incident declare "Billing API 5xx spike" --sev2

# creates #inc-2026-09-14-billing-api, pages IC, opens status doc

/incident timeline "root cause identified: bad config push"

# auto-logs a timestamped entry — becomes the postmortem timeline

Core Roles

🔵Incident Commander — coordinates, decides

🔵Scribe — logs every action and timestamp in real time

🔵Comms Lead — owns status page and stakeholder updates, frees the IC to coordinate

📢

Status Pages & Stakeholder Communication

Internal Slack updates and the public status page are different audiences with different needs — conflating them causes either panic or silence.

Communication

Recommended▾

Internal vs External

🔵Internal: technical detail, real-time, in the war room channel

🔵External: plain language, impact-focused, on a fixed cadence (e.g. every 30 min for SEV1)

Cadence Rule of Thumb

1

SEV1: update every 30 minutes even if the update is "still investigating"

2

Silence reads as "nobody's working on it" — say something even with nothing new

📝

Blameless Postmortem Structure

A consistent template that separates what happened from who was on call — every incident, same shape, so nobody re-invents the format at 2am.

Must Know

Must Know▾

postmortem-template.md

markdown

# Incident: Billing API 5xx Spike (SEV2)

## Summary — 2 sentences, plain language

## Impact — who/what, how long, how many requests

## Timeline — timestamped, from alert to resolution

## Root Cause & Contributing Factors

## What Went Well / What Didn't

## Action Items — owner, deadline, ticket link

🕰️

Timeline Reconstruction

The single most valuable — and most skipped — section. A vague timeline produces a vague root cause.

Important

Important▾

Sources to Reconcile

sources

1. Alertmanager fired-alert history (exact timestamps)

2. War-room chat log (what people said, when)

3. Deploy/change history (what changed right before)

4. Metrics dashboards (when the graph actually bent)

Rule

1

Every timeline entry needs a source — "we think" doesn't belong in a timeline

2

Write it within 24 hours, while chat logs and memory are both still fresh

🔍

Root Cause vs Contributing Factors

"The engineer pushed a bad config" is a trigger, not a root cause. The five-whys trap is stopping at the first human action instead of the systemic gap that let it ship.

Analysis

Recommended▾

Five Whys, Done Right

🔵Why did billing go down? → Bad config pushed

🔵Why did the bad config ship? → No validation step in the pipeline

🔵Why no validation step? → Never prioritized against feature work

🔴Stop at the systemic gap (missing validation), not at "the engineer made a mistake"

✅

Writing Actionable Action Items

"Improve monitoring" is not an action item — it's a wish. An action item has an owner, a deadline, and a definition of done.

Must Know

Must Know▾

Vague vs Actionable

🔴❌ "Improve alerting for the billing service"

🟢✅ "Add a p99 latency alert on /checkout at 800ms — @priya, due Sep 20, JIRA-4521"

SMART Check

1

Specific, Measurable, Assigned, Realistic, Time-boxed

2

If you can't tell when it's "done," rewrite it before the meeting ends

📉

Tracking to Closure

Action items don't die in a dramatic way — they die quietly in a backlog nobody re-opens. This is where most postmortem processes actually fail.

Important

Important▾

1

Every action item gets a real ticket, linked from the postmortem doc — not just a bullet point

2

Track a single metric: % of action items closed within 30 days — report it monthly

3

An action item still open after 2 quarters is a decision to accept the risk — make that explicit, don't let it just fade

⚖️

Risk-Based Prioritization

Not every action item deserves the same urgency as the incident that spawned it — prioritize by recurrence risk and blast radius, not by how memorable the incident was.

Planning

Recommended▾

Prioritize By

🔴High recurrence risk + high blast radius → this sprint

🟠High recurrence, low blast radius (or vice versa) → next sprint

🟢Low + low → backlog, revisit if it recurs

🤝

Blameless Culture in Practice

Blameless doesn't mean nobody's accountable — it means the postmortem's job is fixing the system, and accountability conversations (if any) happen somewhere else entirely.

Must Know

Must Know▾

Signs It's Actually Blameless

🔵Names are in the timeline for context, not in the "root cause" section

🔵The engineer who caused it is comfortable presenting the postmortem themselves

Common Failure Mode

1

Leadership says "blameless" but asks "whose change was this" in the review meeting — culture is what leaders do, not what the template says

📊

MTTR / MTTD / MTBF

The standard reliability metrics — useful in aggregate over quarters, dangerous the moment someone's bonus depends on them.

Important

Important▾

Metric	Meaning	Watch Out For
MTTD	Mean Time To Detect — alert fired vs. issue started	Gamed by lowering alert thresholds, not fixing detection
MTTR	Mean Time To Resolve — alert fired vs. resolved	Gamed by declaring "mitigated" early, before truly fixed
MTBF	Mean Time Between Failures — how often incidents recur	Skewed hard by one outlier month — track median too

🗓️

Postmortem Review Cadence

One postmortem is a data point. A monthly review of all of them is where the actual patterns — the same root cause wearing a different costume — become visible.

Process

Recommended▾

1

Monthly incident review: read every postmortem from the last 30 days side by side

2

Look for repeated root causes across different incidents — that's your real priority list

3

Report action-item closure rate and MTTR trend to leadership — this is what earns the team time to do the work

📄 Copy-Paste Postmortem Template

01Summary — two sentences, plain language, no jargon

02Impact — who was affected, for how long, quantified (requests, revenue, users)

03Timeline — every entry timestamped and sourced (alert log, chat, deploy history)

04Root Cause & Contributing Factors — stop at the systemic gap, not the first human action

05What Went Well / What Didn't — detection speed, comms quality, tooling gaps

06Action Items — owner, deadline, linked ticket, SMART-checked

Decision Guide

WHAT SEVERITY IS THIS?

A quick lookup for the call that has to happen in the first 60 seconds.

Situation	Severity	Why
Checkout completely down, all regions	SEV1	Full revenue-impacting outage, every user affected
One region degraded, others fine	SEV2	Significant but partial — subset of users, no full outage
Non-critical feature broken, workaround exists	SEV3	Real bug, low urgency — ticket, not a page
Typo in an internal dashboard label	SEV4	Cosmetic, zero user impact
Uncertain — could be either SEV1 or SEV2	Declare SEV1	Downgrading later is easy; upgrading a slow response isn't

INCIDENT MANAGEMENT
& POSTMORTEMS

THE INCIDENT LIFECYCLE

ANATOMY OF ONE INCIDENT

INCIDENT MANAGEMENT REFERENCE

WHAT SEVERITY IS THIS?

RESPONSE CHEATSHEET

THE INCIDENT LIFECYCLE

ANATOMY OF ONE INCIDENT

INCIDENT MANAGEMENT REFERENCE

WHAT SEVERITY IS THIS?

RESPONSE CHEATSHEET

INCIDENT MANAGEMENT
& POSTMORTEMS