Kepriye cara retensi data lan panyimpenan dalam Kafka?

Question

Accepted Answer

Kafka **nyimpen event** kanggo dina periode sing bisa dikonfigurasi (utawa ukuran), nyimpen ing partition log sanajan sawise konsumsi — ngaktifake replay lan pirang-pirang consumer. Memahami retensi lan manajemen log penting kanggo nggunakake Kafka lan ngelola panyimpenan.

## Retensi — event disimpen

```text
Unlike traditional queues, Kafka RETAINS events (doesn't delete on consumption):
  → events stay in the partition log for a configured RETENTION period (e.g. 7 days) or
    SIZE limit
  → enables: REPLAY (re-read history), multiple consumers reading independently, late
    consumers catching up
  → after retention expires → old segments are deleted (or compacted)
→ Kafka is a durable LOG, not just a transient queue
```

## Konfigurasi retensi

```text
→ TIME-based → retention.ms (keep events for N time, e.g. 7 days)
→ SIZE-based → retention.bytes (keep up to N bytes per partition)
→ configured per topic; events past the limit are removed (oldest first)
→ choose based on replay needs, consumer lag tolerance, and storage capacity
```

## Pemadatan log

```text
LOG COMPACTION → an alternative retention mode: keep the LATEST value per KEY (not all events):
  → retains the most recent record for each key, removing older ones for that key
  → useful for: changelog/state topics (the current state per key matters, not full history)
  → e.g. a topic of "user → latest profile" → keep the latest per user
→ vs time/size retention (keep recent history) — compaction keeps latest-per-key forever
```

## Mengapa penting

Memahami retensi data lan panyimpenan ing Kafka iku penting amarga **retensi minangka fitur Kafka sing paten lan penting** sing nuduhake kapabilitas lan panyimpenan, dadi pengetahuan sing relevan kanggo nggunakake lan nglayani Kafka.

Retensi event Kafka (nyimpen ing partition log sanajan sawise konsumsi) minangka pembeda utama saka antri tradisional lan ngaktifake kapabilitas paten Kafka.

Memahami **retensi** — yen Kafka nyimpen event kanggo periode utawa ukuran sing dikonfigurasi (ora mbusak sawise konsumsi), ngaktifake **replay** (maca ulang riwayat), pirang-pirang consumer sing maca kanthi mandiri, lan consumer sing telat bisa nyusul — nerangake apa Kafka minangka log sing tahan lan ora antri sing singkat, lan kepriye kapabilitas replay lan multi-consumer iku gawe.

Memahami **konfigurasi retensi** — retensi adhedhasar wektu (retention.ms, contone 7 dina) lan ukuran (retention.bytes) retensi, dikonfigurasi saben topik, milih adhedhasar kebutuhan replay, toleransi consumer lag, lan kapasitas panyimpenan — iku pengetahuan praktis penting kanggo ngelola carane dawa data disimpen lan panyimpenan sing ana karo. Kepriye cara retensi data lan panyimpenan dalam Kafka?

Memahami **pemadatan log** — mode retensi alternatif sing nyimpen **nilai pungkasan saben kunci** (tinimbang kabeh event), migunani kanggo topik changelog/state kang kasusah sang kasusah kasusah kasusah kasusah kasusah kasusah kasusah kasusah matur upami "pangguna → profil paling anyar," nyimpen paling anyar saben pangguna — minangka fitur Kafka sing paten lan mapan kanggo topik sing adhedhasar state, kontras karo retensi wektu/ukuran.

Memahami retensi lan pemadatan penting kanggo nggunakake kapabilitas Kafka (replay, topik state) lan ngelola panyimpenan (retensi nuduhake carane akeh data disimpen).

Konsep-konsep iki nuduhake funksionalitas (apa sing bisa diwaca lan direplay dening consumer) lan operasi (manajemen panyimpenan).

Karana retensi minangka fitur Kafka sing paten lan penting (ngaktifake replay lan pirang-pirang consumer, mbedakake Kafka saka antri) sing nuduhake kapabilitas lan panyimpenan, lan amarga memahami konfigurasi retensi lan pemadatan log iku relevan kanggo nggunakake kapabilitas Kafka lan ngelola panyimpenan, memahami retensi data lan panyimpenan ing Kafka iku pengetahuan sing mapan, praktis-relevan — pusat kapabilitas paten Kafka (replay, pirang-pirang consumer) lan ngelola panyimpenan, karo pemadatan log minangka fitur mapan kanggo topik state, penting kanggo nggunakake lan nglayani Kafka kanthi efektif.