Gidan yanar gizonka na production ya faɗi yanzu. Yaya za ka magance shi?

Question

Accepted Answer

Fifiko na farko shi ne **maido da sabis, sannan a nemo dalili** — rage tasiri yana gabanin diagnosis. Zan ayyana incident, in rarraba ayyuka a fili, kuma in tura zuwa mafi sauri da aminci na dawowa, ina sadarwa cikin dukan hanyar.

## Daidaita da farko

- **Ayyana incident** kuma a buɗe channel ɗaya (war room / Slack thread) don kada haɗin kai ya watse.
- **Rarraba ayyuka**: **Incident Commander** don yanke shawara, **comms owner** don sabunta stakeholder, da **responders** don bincike. A matsayin Tech Lead sau da yawa ina ɗaukar IC don injiniyoyi su mai da hankali kan matsalar.
- **Nemi mafi saurin gyara mai juyawa.** Idan deploy yana da alaƙa da outage, **roll back** da farko sannan a yi tambaya daga baya — maido da masu amfani ya fi zama daidai.

## Yi diagnosis a lokaci ɗaya

- Duba **bayyanannun sigina**: dashboards, error rates, sabbin deploys, canje-canjen infra, hauhawar traffic, certs da suka ƙare.
- Ƙirƙiri hasashe, gwada mafi arha da farko, kuma **kada ka canza abubuwa biyar a lokaci ɗaya** — ba za ka san abin da ya yi aiki ba.

## Yi sadarwa a koyaushe

Shiru yana haifar da firgici. Ina aika sabuntawa bisa kullin tsari ko da babu labari:

```
[14:05] Ana bincike — checkout ya faɗi, ~40% na masu amfani sun shafa. Sabuntawa na gaba 14:20.
[14:20] An gano: bad deploy. Ana rolling back yanzu. ETA mintuna 10.
[14:35] An maido da sabis. Ana sa ido. Postmortem zai biyo baya.
```

## Bayan dawowa

- Tabbatar da cikakken dawowa, ba kawai "yana kallon mafi kyau ba."
- Gudanar da **blameless postmortem** cikin 'yan kwanaki: timeline, root cause, abin da ya sa ya yi jinkirin ganowa/gyarawa, da **takamaiman ayyuka da masu mallaka**.
- Sakamakon shi ne ingantawar tsari (mafi kyawun alerts, guardrails, rollback automation) — ba sunan da za a zarga ba.

## Kuskure

- **Yin debugging kafin rage tasiri** yayin da masu amfani ke shan wahala.
- **Babu mai yanke shawara ɗaya**, don haka mutane biyar suna hasashe a lokaci ɗaya.
- **Yin shiru** ga stakeholder.
- **Zargin mutane**, wanda ke kashe gaskiyar da incident na gaba suka dogara da ita.

## Me ya sa yake da muhimmanci

Outages ba makawa ne; yadda kake gudanar da su yana ayyana amanar ƙungiya da kwarin gwiwar abokin ciniki. Haɗin kai mai natsuwa bisa ayyuka tare da bin diddigi mara zargi yana mai da mummunar rana zuwa tsari mai ƙarfi — kuma yana nuna wa injiniyoyinka cewa yana da aminci a yi sauri saboda ana magance gazawa a matsayin tsari, ba farautar mayya ba.