How does data retention and storage work in Kafka?

Question

Accepted Answer

Kafka **retains events** for a configurable period (or size), keeping them in the partition logs even after consumption — enabling replay and multiple consumers. Understanding retention and log management is important for using Kafka and managing storage.

## Retention — events are kept

```text
Unlike traditional queues, Kafka RETAINS events (doesn't delete on consumption):
  → events stay in the partition log for a configured RETENTION period (e.g. 7 days) or
    SIZE limit
  → enables: REPLAY (re-read history), multiple consumers reading independently, late
    consumers catching up
  → after retention expires → old segments are deleted (or compacted)
→ Kafka is a durable LOG, not just a transient queue
```

## Retention configuration

```text
→ TIME-based → retention.ms (keep events for N time, e.g. 7 days)
→ SIZE-based → retention.bytes (keep up to N bytes per partition)
→ configured per topic; events past the limit are removed (oldest first)
→ choose based on replay needs, consumer lag tolerance, and storage capacity
```

## Log compaction

```text
LOG COMPACTION → an alternative retention mode: keep the LATEST value per KEY (not all events):
  → retains the most recent record for each key, removing older ones for that key
  → useful for: changelog/state topics (the current state per key matters, not full history)
  → e.g. a topic of "user → latest profile" → keep the latest per user
→ vs time/size retention (keep recent history) — compaction keeps latest-per-key forever
```

## Why it matters

Understanding data retention and storage in Kafka is valuable because **retention is a distinctive, important Kafka feature** affecting capabilities and storage, so it's relevant knowledge for using and operating Kafka.

Kafka's retention of events (keeping them in partition logs even after consumption) is a key differentiator from traditional queues and enables Kafka's distinctive capabilities.

Understanding **retention** — that Kafka keeps events for a configured period or size (not deleting on consumption), enabling **replay** (re-reading history), multiple consumers reading independently, and late consumers catching up — clarifies why Kafka is a durable log rather than a transient queue, and how its replay and multi-consumer capabilities work.

Understanding **retention configuration** — time-based (retention.ms, e.g. 7 days) and size-based (retention.bytes) retention, configured per topic, choosing based on replay needs, consumer lag tolerance, and storage capacity — is important practical knowledge for managing how long data is kept and the associated storage.

Understanding **log compaction** — an alternative retention mode that keeps the **latest value per key** (rather than all events), useful for changelog/state topics where the current state per key matters (like "user → latest profile," keeping the latest per user) — is a distinctive, valuable Kafka feature for state-oriented topics, contrasting with time/size retention.

Understanding retention and compaction is important for using Kafka's capabilities (replay, state topics) and managing storage (retention affects how much data is stored).

These concepts affect both functionality (what consumers can read and replay) and operations (storage management).

Since retention is a distinctive, important Kafka feature (enabling replay and multiple consumers, distinguishing Kafka from queues) that affects capabilities and storage, and since understanding retention configuration and log compaction is relevant for using Kafka's capabilities and managing storage, understanding data retention and storage in Kafka is valuable, practically-relevant knowledge — central to Kafka's distinctive capabilities (replay, multiple consumers) and to storage management, with log compaction being a valuable feature for state topics, important for using and operating Kafka effectively.