LLM Best Practices

Blameless Postmortems

2 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

A postmortem turns an incident into fixes for the whole class of failure. It is the last step after service is restored in incident-response. Use the structure below so postmortems are comparable across incidents.

Write one within 48 hours for every SEV1 and escalated SEV2

Draft while logs and memory are fresh. For a SEV3, write a short summary only when you suspect a systemic pattern.

Assume every person acted on the best information they had

Blame produces defensiveness, defensiveness produces incomplete timelines, and incomplete timelines produce repeat incidents. Google’s SRE book and Atlassian’s incident handbook both treat blamelessness as a precondition for an honest postmortem. Describe systems and processes, not people: “the deploy script did not check for a pending migration”, not “Sam forgot”.

Use five parts in the same order every time

  1. Timeline: events from first alert to full resolution, taken from logs, alert history (error-tracking), and traces (observability), not memory.
  2. Root cause: the technical condition that caused the failure. Keep asking why until you pass the proximate cause.
  3. Impact: users affected, duration, data or revenue exposure.
  4. What worked: detection, tooling, and coordination worth keeping.
  5. Action items: each with an owner, a due date, and a type, either mitigative (prevents this specific failure) or preventative (addresses the class).

Do not close the postmortem until every action item has an owner; unowned items do not get done. Turn recurring gaps into checks in pre-launch-checklist.

Copy this template

# Postmortem: <short title> (SEV<n>, <YYYY-MM-DD>)
 
## Summary
What broke, for whom, for how long.
 
## Timeline (UTC)
- HH:MM alert fired: <source>
- HH:MM acknowledged by <role>
- HH:MM mitigation: <action>
- HH:MM resolved
 
## Root cause
<technical condition; systems and processes, not people>
 
## Impact
Users affected, duration, data or revenue exposure.
 
## What worked
<detection, tooling, coordination worth keeping>
 
## Action items
| Item | Owner | Due | Type (mitigative / preventative) |
|------|-------|-----|----------------------------------|

Backlinks