For engineers who own production: run incidents calmly and defend reliability on-call.
Reach for this when you own a service on-call and want incidents to be routine, not chaos. It carries you through the whole lifecycle - instrument and de-noise your signals, call severity fast, work the runbook, keep customers honestly informed on the status page and run the external crisis comms (first statement inside the hour, speculation never), hand the shift off with a completeness checklist, write the blameless postmortem on the 48-hour clock with the five-whys-stops-at-process rule, and turn SLO error budgets into automatic team decisions. Templates and severity thresholds included at every step. Built for engineers and SREs who carry the pager.
Click to play with sound.
Arranged in the author's recommended order. Walk through them in sequence, or open any one on its own.
Classifies incident severity (SEV1-4) using impact, scope, and urgency signals and decides who to page.
Writes an operational runbook for a service covering symptoms, diagnostic checks, mitigations, and escalation paths.
Writes customer-facing incident status updates for each lifecycle stage – investigating, identified, monitoring, resolved – with cadence rules and plain-language templates.
Drafts and sequences external communications during an incident or crisis – a first statement within the hour, acknowledge-what-you-know-without-speculating discipline, a single-spokesperson rule, channel sequencing from status page to email to social.
Produces a complete on-call shift handoff note – shift summary, open incidents, watch items with paging thresholds, silenced alerts, in-flight changes, and escalation contacts – checked against a completeness checklist.
Reduces alert noise by making alerts actionable, symptom-based, and tied to SLOs. Identifies and kills flappy, duplicate, and cause-based alerts.
Defines and applies an SLO error-budget policy that gates feature work versus reliability work based on remaining budget.
Instruments systems with the three pillars – structured logs, RED/USE metrics, and distributed traces – correlated by one trace ID, with cardinality budgets and symptom-based alerts.
Writes blameless incident postmortems with a log-reconstructed timeline, multi-causal contributing factors, and owned, dated action items.