The first fifteen minutes of an incident decide most of what happens in the next four hours. Teams that handle them well — through habit, not heroics — recover faster, do less damage in flight, and produce better post-mortems. Teams that handle them badly compound the original problem until someone senior steps in to restart the response.
The discipline that separates the two is unglamorous and learnable.
The roles to fill immediately
Every non-trivial incident needs three roles. They can be three people or they can be one person juggling them, but the roles must be explicit.
Incident commander. Owns the response. Does not fix the problem; coordinates the people fixing it. Their job is to keep the response on track, run the timeline, and call escalations.
Communications lead. Owns external and internal updates. Drafts the status page entries, the customer emails, the executive summaries. This is a real job during an incident; do not let it default to the incident commander.
Subject-matter expert(s). The people actually fixing the problem. They report to the incident commander.
Naming these roles in the first five minutes of an incident is the single highest-leverage habit a team can build.
The first fifteen minutes, in order
- 0–2 min: Acknowledge the page. Open the incident channel. State, in writing, "I am the incident commander."
- 2–5 min: Confirm impact. What is broken, for whom, since when? Update the status page even if the answer is "investigating".
- 5–10 min: Form a hypothesis. Pull in subject-matter experts. Start a timeline document. Decide if you need to escalate.
- 10–15 min: Bound the blast radius. Roll back if the change is recent. Disable a feature flag if there is one. Stop the bleeding before you understand the cause.
The temptation in the first fifteen minutes is to debug. Resist it. Stop the bleeding first, debug later.
What good post-mortems do
Post-mortems should produce changes to the system, not changes to people. The questions to answer are: what allowed this to happen, what allowed it to get this bad, what shortened the response, and what would have shortened it more.
The post-mortems that pay back are the ones that produce concrete follow-up work and the team actually does it. If you find yourself writing similar post-mortems month after month, the system is not learning. Fix the loop, not the next incident.