CDC husync vipi data kuvuka systems?

Question

IT Interview Practice · Accepted Answer

**Change Data Capture (CDC)** hustream kila insert/update/delete kutoka source database — kwa kawaida kwa **kusoma transaction log yake** — na huzichapisha kama events ambazo systems nyingine huzitumia, ili downstream stores zibaki sync bila dual-writes au polling.

```text
  Postgres/MySQL          Debezium            Kafka topic          sinks
 ┌────────────┐  reads   ┌─────────┐  emit   ┌──────────┐  consume ┌───────────┐
 │ WAL/binlog │ ───────▶ │ connector│ ──────▶ │ per-table│ ───────▶ │ search    │
 │ (tx log)   │          │          │  events │  ordered │          │ warehouse │
 └────────────┘          └─────────┘          └──────────┘          │ cache     │
                                                                     └───────────┘
```

## Kwa nini log-based CDC

Unaweza sync kwa polling ("rows zilizobadilika tangu T") au kwa kuandika kwenye systems mbili kwa wakati mmoja (**dual-write**), lakini polling hukosa fast updates na dual-writes huvunjika kiatomiki — ikiwa write ya pili inashindwa unapata divergence. **Log-based CDC** badala yake hufuatilia write-ahead log ya database yenyewe (Postgres **WAL**, MySQL **binlog**), rekodi ya kudumu ambayo DB hutumia kwa crash recovery. **Debezium** ni connector wa kawaida: husoma log na kuchapisha kila mabadiliko kwenda **Kafka**, topic moja kwa kila jedwali — ikikamata *kila* mabadiliko bila load kwenye queries na bila rows zilizokosekana.

## Ordering na idempotency

- **Ordering:** mabadiliko kwa row ile ile lazima yafike kwa mpangilio la sivyo ungetumia thamani ya zamani mwisho. Kafka huhakikisha mpangilio **ndani ya partition**, hivyo partition kwa primary key — mabadiliko yote ya row moja huingia partition moja, kwa commit order.
- **Delivery ni at-least-once:** kwenye retry au connector restart event ile ile inaweza kutumwa mara mbili. Hivyo **sinks lazima ziwe idempotent** — weka write key kwa primary key + log offset (LSN) ili kutumia tena event iwe no-op. Kamwe usidhani exactly-once.

## Mtazamo wa mahojiano

Mhojiwa hukagua kama unajua CDC hushinda dual-writes na *kwa nini* (atomicity, hakuna mabadiliko yaliyokosekana), na kama unaelewa **at-least-once + idempotent sinks**. Nzuri: "Debezium husoma WAL kwenda Kafka." Bora sana: unaongeza partition-by-key kwa ordering na idempotent sinks kwa duplicates. Mtego: kudhani exactly-once delivery, au kupuuza kwamba DDL change kwenye source inaweza kuvunja consumers.

## Maelewano na aina za hitilafu

- **Schema evolution:** column iliyoongezwa/kuondolewa upstream hubadilisha umbo la event — tumia **schema registry** na mabadiliko yanayolingana ili consumers zisivunjike.
- **Initial snapshot:** CDC huona mabadiliko *tangu sasa* tu; unahitaji snapshot ya mara moja ya rows zilizopo, kisha ubadilishe kuwa streaming.
- **Log retention:** ikiwa source itafuta log yake kabla connector hajaisoma, unapoteza mabadiliko — fuatilia connector lag.
- **Backpressure:** sink ya polepole huruhusu Kafka lag kukua; pima retention na consumers kwa peak change volume.

Kwa nini log-based CDC

Ordering na idempotency

Mtazamo wa mahojiano

Maelewano na aina za hitilafu

Maswali yanayohusiana