LLM Best Practices

Blameless Postmortems

3 min read

View markdown source

Overview

A postmortem turns an incident into fixes that stop the same class of failure from recurring. It is the last step of the response process in incident-response, after service is restored. This page covers when to write one, how to keep it blameless, the section order, and a template to copy.

Write one within 48 hours for every SEV1 and escalated SEV2

Every SEV1 and every SEV2 that required on-call escalation gets a postmortem. SEV3 incidents get a brief written summary if a systemic pattern is suspected.

Assume good intent in every finding

Blameless means the process assumes every person acted with the best information they had at the time. Blame produces defensiveness; defensiveness produces incomplete timelines; incomplete timelines produce repeated incidents. Atlassian and Google SRE both treat blamelessness as a prerequisite for honest postmortems, not a nicety.

Use a fixed five-part structure

Write every postmortem in the same order so readers can compare incidents:

  1. Timeline. Ordered list of events from first alert to full resolution, sourced from logs and tool history, not memory.
  2. Root cause. The technical condition that caused the failure. Use “five whys” to get past the proximate cause. Refer to systems and processes, not individuals.
  3. Impact. Users affected, duration, data or revenue exposure.
  4. What worked. Explicitly note detection speed, tooling, or coordination that held up well. Reinforce those behaviors.
  5. Action items. Each item has an owner, a due date, and a category: mitigative (prevents this specific failure) or preventative (addresses the class of failure). Add unresolved gaps to pre-launch-checklist for future deploys.

Do not close the postmortem until every action item has an owner. Items without owners do not get done.

Start from a template

Copy this skeleton into the incident’s doc so every postmortem has the same sections. Source the timeline from observability data and alert history in error-tracking.

# Postmortem: <short title> (SEV<n>, <YYYY-MM-DD>)
 
## Summary
One paragraph: what broke, for whom, for how long.
 
## Timeline (UTC)
- HH:MM alert fired: <source>
- HH:MM acknowledged by <role>
- HH:MM mitigation: <action>
- HH:MM resolved
 
## Root cause
<technical condition; systems and processes, not people>
 
## Impact
Users affected, duration, data or revenue exposure.
 
## What worked
<detection, tooling, coordination worth keeping>
 
## Action items
| Item | Owner | Due | Type (mitigative / preventative) |
|------|-------|-----|----------------------------------|

Backlinks