Migrate database na 10TB ba tare da downtime ba: menene shirin?

Question

IT Interview Practice · Accepted Answer

**Dual-write zuwa tsoho da sabo, backfill history a background, yi amfani da CDC don kama tail, verify da shadow reads, sannan ka flip reads cikin atomic — da rollback path a kowane mataki.** Ba ka taɓa saukar da app ba; kana gudanar da databases biyu a layi ɗaya har sabon ya tabbata iri ɗaya, sannan ka cut over.

## Gine-gine

```text
        ┌── write ──▶ OLD DB (source of truth) ──CDC (Debezium)──┐
 App ───┤                                                        ▼
        └── write ──▶ NEW DB ◀──── backfill job (historical rows)
                        │
   shadow reads (compare OLD vs NEW) ─▶ diff metrics ─▶ atomic cutover ─▶ NEW is truth
                                                          (flip flag; rollback = flip back)
```

## Yadda yake aiki

1. **Dual-write.** App yana rubuta kowane canji zuwa OLD da NEW. Sabbin rows suna zaman sync daga yanzu; history kawai ce ta rasa.
2. **Backfill.** Background job yana kwafin historical rows a batches (throttled don kare prod). Yi amfani da `INSERT ... ON CONFLICT DO NOTHING` domin ya zama idempotent kuma mai aminci a sake gudanarwa.
3. **CDC catch-up.** Dual-write na iya rasa rows da aka rubuta a lokacin switch-on. **Debezium** yana stream WAL/binlog na OLD DB yana replay kowane canji, yana rufe gibin da backfill ba zai iya ba.
4. **Shadow reads.** Karanta daga NEW a layi ɗaya da OLD, **kwatanta sakamako**, kuma ka fitar da diff metric. Zero diffs a kan lokaci = NEW abin dogaro ne.
5. **Cutover.** Flip feature flag domin reads (sannan writes) su tafi NEW. Canjin config ne, ba deploy ba — nan take, kuma **mai iya juyawa da flip baya**.

## Ra'ayin hira

Mai yin hirar yana son **yadda kake tabbatar da babu data loss da babu downtime**, kuma ko kana da **rollback**. Amsoshi masu kyau suna ambaton dual-write; amsoshi masu kyau sosai suna ƙara **CDC don kama race window**, **shadow reads don verify kafin dogaro**, da **atomic, reversible flag flip**. Tarkon gargajiya shi ne "backfill sannan switch" ba tare da verification ba — kana gano rashin daidaituwa bayan users sun buge ta, ba tare da hanyar dawowa ba.

## Sulhu & kurakurai

- **Dual-write consistency.** Idan write na NEW ya gaza amma OLD ya yi nasara, sun kauda. Mai da OLD authoritative kuma ka bar CDC ya gyara NEW, maimakon gazawar request na user.
- **Cost & complexity.** Gudanar da databases biyu + CDC + backfill mai tsada ne kuma mai nauyi a operations — barata kawai ga manyan, critical stores.
- **Schema changes** a lokacin migration suna ninka haɗari; freeze schema har cutover.
- **Backfill load.** Unthrottled backfill na iya cika prod I/O kuma ya haifar da outage — akasin manufa; batch kuma rate-limit.
- **Rollback window.** Ci gaba da dual-writing (yanzu NEW→OLD) na kwanaki bayan cutover domin ka iya flip baya idan latent bug ya bayyana. Sai kawai ka decommission OLD.
- **Lokacin da ba za a yi ba:** ƙaramin DB da maintenance window ba ya buƙatar wannan — `pg_dump`/restore na mintuna 30 mai sauƙi ne kuma mafi aminci fiye da CDC pipeline.

Gine-gine

Yadda yake aiki

Ra'ayin hira

Sulhu & kurakurai

Tambayoyi masu alaƙa