Observability & Operations
Incident Management & Postmortems
An incident is managed in two phases — restore service first, understand root cause second — and a blameless postmortem is how an organization actually learns from an outage instead of just being relieved it is over.
- Mitigate before you diagnose: rolling back a bad deploy or failing over to a healthy region ends user impact immediately, even before the root cause is known
- An incident commander owns coordination and communication during an incident so responders can focus on fixing instead of status-updating
- A severity scale drives response consistently — who gets paged, whether execs are notified, whether a public status page updates — instead of deciding ad hoc each time
- Blameless postmortems focus on what about the system allowed this, rather than who made the mistake — punishing the person who tripped an incident just teaches everyone to hide the next one
- A postmortem's real output is action items with owners and dates — a document that identifies causes but assigns no follow-up changes nothing
- Timeline reconstruction — what was known, when, and what was done about it — is often more valuable than the root cause itself, since it reveals detection and response gaps that recur across unrelated incidents
| Severity | Impact | Who is paged | Comms cadence |
|---|---|---|---|
| SEV1 | full outage or data loss | on-call plus incident commander plus leadership | continuous, public status page |
| SEV2 | major feature degraded | on-call plus incident commander | regular internal updates |
| SEV3 | minor, contained impact | on-call | ticket, no broad comms |