Hamisha database ya 10TB bila downtime: mpango ni upi?

Question

IT Interview Practice · Accepted Answer

**Dual-write kwa old na new, backfill history nyuma ya pazia, tumia CDC kupata tail, thibitisha kwa shadow reads, kisha flip reads kwa atomic — na njia ya rollback kwa kila hatua.** Kamwe hushushi app; unaendesha databases zote mbili sambamba hadi mpya iwe imethibitika kuwa sawa kabisa, kisha cut over.

## Usanifu

```text
        ┌── write ──▶ OLD DB (source of truth) ──CDC (Debezium)──┐
 App ───┤                                                        ▼
        └── write ──▶ NEW DB ◀──── backfill job (historical rows)
                        │
   shadow reads (compare OLD vs NEW) ─▶ diff metrics ─▶ atomic cutover ─▶ NEW is truth
                                                          (flip flag; rollback = flip back)
```

## Jinsi inavyofanya kazi

1. **Dual-write.** App huandika kila mabadiliko kwa OLD na NEW zote. Rows mpya hubaki sync tangu sasa; history pekee ndiyo inayokosekana.
2. **Backfill.** Background job hunakili rows za kihistoria kwa batches (throttled kulinda prod). Tumia `INSERT ... ON CONFLICT DO NOTHING` ili iwe idempotent na salama kuendesha tena.
3. **CDC catch-up.** Dual-write inaweza kukosa rows zilizoandikwa wakati wa switch-on. **Debezium** ikistream WAL/binlog ya OLD DB hufanya replay ya kila mabadiliko, ikifunga pengo ambalo backfill haiwezi.
4. **Shadow reads.** Soma kutoka NEW sambamba na OLD, **linganisha matokeo**, na toa diff metric. Zero diffs kwa muda = NEW inaaminika.
5. **Cutover.** Flip feature flag ili reads (kisha writes) ziende NEW. Ni config change, si deploy — papo hapo, na **inayoweza kurudishwa kwa kuflip tena**.

## Mtazamo wa mahojiano

Mhojiwa anataka **jinsi unavyohakikisha hakuna data loss na hakuna downtime**, na kama una **rollback**. Majibu mazuri hutaja dual-write; bora sana huongeza **CDC kunasa race window**, **shadow reads kuthibitisha kabla ya kuamini**, na **atomic, reversible flag flip**. Mtego wa kawaida ni "backfill kisha switch" bila uthibitisho — unagundua mismatch baada ya watumiaji kuipiga, bila njia ya kurudi.

## Maelewano na hitilafu

- **Dual-write consistency.** Ikiwa NEW write inashindwa lakini OLD inafanikiwa, zinatofautiana. Fanya OLD iwe authoritative na uruhusu CDC kurekebisha NEW, badala ya kushindisha request ya mtumiaji.
- **Gharama na ugumu.** Kuendesha databases mbili + CDC + backfill ni ghali na nzito kiuendeshaji — huhalalishwa tu kwa stores kubwa, muhimu.
- **Schema changes** wakati wa migration huzidisha hatari; freeze schema hadi cutover.
- **Backfill load.** Backfill isiyo throttled inaweza kujaza prod I/O na kusababisha outage — kinyume cha lengo; batch na rate-limit.
- **Rollback window.** Endelea kufanya dual-writing (sasa NEW→OLD) kwa siku baada ya cutover ili uweze kuflip tena ikiwa latent bug itajitokeza. Ndipo tu utengue OLD.
- **Wakati usifanye:** DB ndogo yenye maintenance window haihitaji hili — `pg_dump`/restore ya dakika 30 ni rahisi na salama zaidi kuliko CDC pipeline.

Usanifu

Jinsi inavyofanya kazi

Mtazamo wa mahojiano

Maelewano na hitilafu

Maswali yanayohusiana