Ta yaya CDC ke sync data a kan systems?

Question

IT Interview Practice · Accepted Answer

**Change Data Capture (CDC)** yana stream kowane insert/update/delete daga source database — yawanci ta **karanta transaction log ɗinsa** — kuma yana buga su a matsayin events da sauran systems ke consume, domin downstream stores su zauna sync ba tare da dual-writes ko polling ba.

```text
  Postgres/MySQL          Debezium            Kafka topic          sinks
 ┌────────────┐  reads   ┌─────────┐  emit   ┌──────────┐  consume ┌───────────┐
 │ WAL/binlog │ ───────▶ │ connector│ ──────▶ │ per-table│ ───────▶ │ search    │
 │ (tx log)   │          │          │  events │  ordered │          │ warehouse │
 └────────────┘          └─────────┘          └──────────┘          │ cache     │
                                                                     └───────────┘
```

## Me ya sa log-based CDC

Za ka iya sync ta polling ("rows da suka canja tun T") ko ta rubuta zuwa systems biyu a lokaci guda (**dual-write**), amma polling yana rasa fast updates kuma dual-writes suna karyewa atomically — idan write na biyu ya gaza kana samun divergence. **Log-based CDC** maimakon haka yana bin write-ahead log na database (Postgres **WAL**, MySQL **binlog**), durable record da DB ke amfani da shi don crash recovery. **Debezium** shi ne standard connector: yana karanta log kuma yana buga kowane canji zuwa **Kafka**, topic ɗaya ga kowane table — yana kama *kowane* canji ba tare da load a queries ba kuma babu missed rows.

## Ordering & idempotency

- **Ordering:** canje-canje ga same row dole su iso cikin oda ko za ka apply old value na ƙarshe. Kafka yana ba da tabbacin oda **a cikin partition**, don haka partition da primary key — duk canje-canje ga row ɗaya suna sauka a partition ɗaya, cikin commit order.
- **Delivery at-least-once ne:** a retry ko connector restart same event na iya iso sau biyu. Don haka **sinks dole su zama idempotent** — key write da primary key + log offset (LSN) domin sake-apply na event ya zama no-op. Kada ka taɓa ɗauka exactly-once.

## Ra'ayin hira

Mai yin hirar yana bincike ko ka san CDC yana doke dual-writes kuma *me ya sa* (atomicity, babu missed changes), kuma ko ka fahimci **at-least-once + idempotent sinks**. Mai kyau: "Debezium yana karanta WAL zuwa Kafka." Mai kyau sosai: ka ƙara partition-by-key don ordering da idempotent sinks don duplicates. Tarko: ɗaukar exactly-once delivery, ko yin watsi da cewa DDL change a source na iya karya consumers.

## Sulhu & hanyoyin gazawa

- **Schema evolution:** column da aka ƙara/cire upstream yana canja event shape — yi amfani da **schema registry** da compatible changes domin consumers kada su karye.
- **Initial snapshot:** CDC yana ganin canje-canje *daga yanzu* kawai; kana buƙatar one-time snapshot na existing rows, sannan ka canja zuwa streaming.
- **Log retention:** idan source ya purge log ɗinsa kafin connector ya karanta shi, kana rasa canje-canje — monitor connector lag.
- **Backpressure:** slow sink yana bar Kafka lag ya girma; size retention da consumers don peak change volume.

Me ya sa log-based CDC

Ordering & idempotency

Ra'ayin hira

Sulhu & hanyoyin gazawa

Tambayoyi masu alaƙa