Me kake yi don hana irin incident ɗaya faruwa a karo na biyu?

Question

Accepted Answer

Gudanar da **blameless postmortem** da ke gano systemic cause, sannan a maida shi zuwa **takamaiman, mai mallaka, da tracked** action items — kuma a tabbatar da gaske suna ship. Manufar ba gano wanda za a zarga ba; canza tsarin ne don irin gazawar ya zama ba zai yiwu ba ko self-detecting.

## Core loop

1. **Blameless postmortem.** Gina timeline mai gaskiya: abin da ya faru, lokacin da muka gano shi, yadda muka amsa, menene tasiri. Zargi yana kashe gaskiyar da kake buƙata.
2. **Tona zuwa systemic cause**, ba trigger ba. "Injiniya ya tura bad config" trigger ne; "babu validation ko staged rollout ga configs" shi ne dalili. Yi amfani da **Five Whys** amma tsaya a tsarin, ba a mutum ba.
3. **Samar da prevention layers**, ba gyara ɗaya ba:
   - **Prevent** — sa gazawa ba zai yiwu ba (validation, types, guardrails).
   - **Detect** — kama shi da wuri (alerts, SLOs, synthetic checks).
   - **Mitigate** — iyakance blast radius (rollback, flags, circuit breakers).
4. **Sanya masu mallaka da kwanakin ƙarewa.** Action item ba tare da mai mallaka ba fata ce. Track su kamar ainihin aiki a backlog.
5. **Tabbatar da rufe loop.** Gyaran ya ship? Zai kama na asali? MTTR da detection da gaske sun inganta?

## Trade-offs da kuskure

- **Postmortems da ke samar da takardu, ba canje-canje ba** — gazawa mafi yawa. Babu bin diddigi yana nufin za ka sake haɗu da incident ɗaya.
- **Over-fixing takamaiman alama** yayin da nau'in bug ya rage a buɗe. Gyara nau'in, ba kawai wannan misalin ba.
- **Alert fatigue** daga tara detection ba tare da tuning ba; alerts masu hayaniya ana watsi da su kuma incident na gaba yana zamewa.
- **Blame culture** yana tura rahoto ƙasa; kana rasa near-misses da su ne darussanka mafi arha.
- Ɗauki **repeat incidents a matsayin sigina na process failure**, ba rashin sa'a ba — track recurrence a matsayin metric.

## Me ya sa yake da muhimmanci

Incidents tuition ne da ka riga ka biya; kawai ɓarna ita ce rashin koyo daga gare su. Ƙungiya da ke hana sake faruwa cikin tsari tana zama mafi aminci kowane kwata, yayin da ƙungiya da kawai ke firefight tana sake rayuwar outages ɗaya har abada. Gina wannan learning loop — da amincin yin gaskiya a cikinsa — yana ɗaya daga cikin abubuwa masu dorewa da Tech Lead zai iya barewa.