An incident checklist turns a stressful outage into a repeatable process instead of improvised panic. At minimum it should cover detection, communication, mitigation, and follow-up.
The checklist
- Detect: automated alert fires (uptime monitor, error-rate spike, etc.)
- Acknowledge: on-call person confirms they’ve seen it, within an agreed SLA (e.g. 5 minutes)
- Communicate: post an initial status page update, even if it just says "investigating"
- Diagnose: check recent deploys, server metrics, and error logs, in that order
- Mitigate: roll back a bad deploy, restart a stuck service, or fail over — whichever restores service fastest, even before root cause is fully known
- Update: post status page updates at regular intervals until resolved
- Resolve: confirm the fix with monitoring, not just visually
- Post-mortem: document root cause and a concrete prevention step within a few days
FAQ
Should every outage get a public post-mortem?
Not necessarily public, but every outage above a certain severity threshold should get an internal one — skipping this step is how the same outage repeats.