How to Actually Run a Post Mortem for a Failed Deploy
Post Mortem Idaho 4 — The Practical Version
Most post mortems are a waste of time. You've seen them. Someone writes a vague document, nobody reads it past the summary, and the same thing happens again three months later because there were no concrete action items attached to owners with deadlines. I spent way too many years watching this cycle play out in different shapes, and the one thing that made it stop being useless was treating the format itself as a discipline, not a checkbox. Here's how I structure a post mortem for something like a failed deployment, outage, or system failure — specifically adapted for what we're calling the Post Mortem Idaho 4 framework. It's not rocket science. It's just a template that forces you to say what actually happened before you get comfortable with the sanitized version.
Step 1: Write the Timeline Before Anything Else
I don't mean "the bug was reported at 2pm." I mean minute-by-minute from the moment the first symptom appeared. Every alert. Every on-call page. Every manual intervention. Every status update sent to Slack. Pull the incident channel logs. Pull the PagerDuty timeline. Pull the Grafana screenshots. Get it all in one ordered list. The reason this matters is that your memory lies to you. Within 48 hours, people will start editing the story in their heads. The engineer who missed the alert will remember they were "deep in focus work." The manager who delayed escalation will remember they "thought it would self-heal." The timeline is the only thing that stays still. For Post Mortem Idaho 4 specifically, the timeline needs to capture the moment the deployment pipeline was triggered and the exact second monitoring flagged the anomaly. Those two data points lock in whether this was a deployment issue or a runtime issue, and most teams skip that distinction at their peril.
Step 2: Identify the Root Cause Without Naming a Person
This is where most teams fail. They write "human error" and move on. That's not a root cause. That's a lazy description of a symptom. Your job is to trace the failure back to the condition that made the human error possible. Was there a missing guardrail in the deploy script? Was the test coverage insufficient for the changed path? Was the alert threshold set too high to catch the degradation? I once spent three weeks chasing a flaky integration failure that kept resurfacing after deployments. The Post Mortem Idaho 4 process forced me to map every test that ran in CI against every test that ran in staging. What we found was that a shared test fixture had been modified in a parallel PR, and the CI pipeline was using a cached version while staging had the new one. No human did anything wrong. The system allowed two environments to operate on different code for six hours. That was the real failure.
Get the Full Details
Step 3: Define the Impact in Numbers, Not Feelings
Don't write "significant customer-facing impact." Write "47 users experienced 429 errors between 14:32 and 15:17 UTC, representing 3.2% of active sessions during that window. Revenue loss estimated at $1,840 based on transaction volume during the affected period." Even if the numbers are rough, they're more useful than adjectives. They tell you whether this was a blip or a structural problem. Every action item needs an owner, a due date, and a definition of done. "Improve monitoring" is not an action item. "Add a synthetic check to the /health endpoint that pings every 30 seconds and alerts if response time exceeds 200ms for three consecutive checks, owned by @jsmith, due 2024-03-15" is an action item. For Post Mortem Idaho 4, I've found that the most common failure mode is writing action items that require a human to remember to do something. You need to bake the fix into the system itself. A guardrail, a test, an automated check. Something that makes the failure impossible rather than just unlikely.
What This Doesn't Fix
Post Mortem Idaho 4 assumes you have access to logs, monitoring data, and a team that will be honest in the retrospective. If your org culture punishes people for reporting mistakes, this process produces bad data and you'll get bad results. No template fixes that. You need leadership to explicitly decouple blame from consequences for anyone documenting a failure. Without that, you'll spend your time whitewashing the timeline instead of understanding it. Also, the process takes time. A proper Post Mortem Idaho 4 takes between 2 and 4 hours depending on the severity of the incident. If your on-call rotation doesn't protect people from being expected to write these immediately after a 3am page, they'll produce shallow documents just to clear the ticket. Build in the recovery time. It's not optional.
The One Thing I Wish Someone Had Told Me Earlier
The post mortem document is not the output. The output is the change it forces you to make in your systems. A beautifully written Post Mortem Idaho 4 that results in zero engineering changes is worse than nothing — it creates the illusion of improvement while the actual problems remain untouched. Track every action item. Follow up on them at the next retrospective. If something was supposed to be fixed and it's still broken three months later, that's a separate failure that deserves its own document. I keep the action items in a public spreadsheet linked from the post mortem. Anyone can see what's open, what's done, and what's stalled. Transparency is the only thing that keeps these from becoming graveyard documents.
