What do you do to prevent the same kind of incident from happening a second time?

Question

Accepted Answer

Run a **blameless postmortem** that finds the systemic cause, then convert it into **concrete, owned, and tracked** action items — and verify they actually ship. The goal is not to find who to blame; it's to change the system so the same class of failure becomes impossible or self-detecting.

## The core loop

1. **Blameless postmortem.** Build an honest timeline: what happened, when we detected it, how we responded, what the impact was. Blame kills the truth you need.
2. **Dig to the systemic cause**, not the trigger. "An engineer pushed a bad config" is a trigger; "there was no validation or staged rollout for configs" is the cause. Use the **Five Whys** but stop at the system, never at a person.
3. **Generate prevention layers**, not a single fix:
   - **Prevent** — make the failure impossible (validation, types, guardrails).
   - **Detect** — catch it sooner (alerts, SLOs, synthetic checks).
   - **Mitigate** — limit blast radius (rollback, flags, circuit breakers).
4. **Assign owners and due dates.** An action item without an owner is a wish. Track them like real work in the backlog.
5. **Verify and close the loop.** Did the fix ship? Would it have caught the original? Did MTTR and detection actually improve?

## Trade-offs and pitfalls

- **Postmortems that produce documents, not changes** — the most common failure. No follow-through means you'll meet the same incident again.
- **Over-fixing the specific symptom** while the class of bug remains open. Fix the category, not just this instance.
- **Alert fatigue** from piling on detection without tuning; noisy alerts get ignored and the next incident slips through.
- **Blame culture** drives reporting underground; you lose the near-misses that are your cheapest lessons.
- Treat **repeat incidents as a signal of process failure**, not bad luck — track recurrence as a metric.

## Why it matters

Incidents are tuition you've already paid; the only waste is not learning from them. A team that systematically prevents recurrence gets more reliable every quarter, while a team that just firefights relives the same outages forever. Building this learning loop — and the safety to be honest in it — is one of the most durable things a Tech Lead can leave behind.