How does Kafka work internally (log storage, ZooKeeper/KRaft)?

Question

Accepted Answer

Internally, Kafka stores data as an **append-only log** on disk (organized in segments), uses efficient I/O techniques, and manages cluster metadata via **ZooKeeper** (historically) or **KRaft** (now). Understanding the internals deepens understanding of Kafka's behavior and performance.

## The commit log storage

```text
Each partition is an append-only LOG stored on disk, split into SEGMENTS (files):
  → new events are APPENDED to the end (sequential writes → fast)
  → events are immutable once written; identified by OFFSET
  → old segments are deleted (retention) or compacted
  → an INDEX maps offsets to file positions (fast lookups)
→ the append-only log is the core of Kafka's design (durable, sequential, efficient)
```

## Efficiency techniques

```text
✓ SEQUENTIAL I/O → appending to a log is sequential disk access → very fast (vs random I/O)
✓ ZERO-COPY (sendfile) → transfer data from disk to network WITHOUT copying through the app
  → highly efficient reads
✓ OS PAGE CACHE → Kafka relies on the OS cache (recent data in memory) rather than its own
  cache → fast reads of recent data
✓ Batching, compression → efficient I/O
→ these make Kafka extremely fast despite using disk
```

## Cluster coordination: ZooKeeper → KRaft

```text
Kafka needs to manage: cluster METADATA, broker membership, leader ELECTION, configs:
  → ZOOKEEPER (historically) → a separate system Kafka used for coordination/metadata
  → KRAFT (KIP-500, now) → Kafka's OWN built-in consensus (Raft) → removes the ZooKeeper
    dependency → simpler ops, better scalability (the modern approach)
→ Kafka is moving to KRaft (self-managed metadata, no separate ZooKeeper)
```

## Why it matters

Understanding how Kafka works internally is valuable senior-level knowledge because it **deepens understanding of Kafka's behavior and performance**, reflecting sophisticated knowledge.

Understanding Kafka's internals explains why it behaves and performs as it does.

Understanding **the commit log storage** — each partition being an append-only log on disk split into segments, with events appended sequentially (fast writes), identified by offset, immutable once written, and old segments deleted or compacted, with an index for fast offset lookups — explains Kafka's core storage design (the append-only log being fundamental to its durability and efficiency).

Understanding **efficiency techniques** — **sequential I/O** (appending being fast sequential disk access), **zero-copy** (transferring data from disk to network without copying through the application, for efficient reads), **OS page cache** (relying on the OS cache for fast reads of recent data), and batching/compression — explains *why Kafka is so fast despite using disk*, a key insight (Kafka achieves high performance through these design choices, not by avoiding disk).

Understanding **cluster coordination** — that Kafka manages metadata, broker membership, and leader election via **ZooKeeper** historically or **KRaft** now (Kafka's own built-in Raft consensus that removes the ZooKeeper dependency for simpler operations and better scalability, the modern approach) — reflects awareness of an important architectural evolution in Kafka (the move to KRaft, removing the long-standing ZooKeeper dependency).

This internal understanding (log storage, efficiency techniques, coordination evolution) deepens comprehension of Kafka's performance, durability, and operations, distinguishing those who understand Kafka deeply.

Since understanding Kafka's internals (the append-only log storage, the efficiency techniques like sequential I/O and zero-copy that make it fast, and the ZooKeeper-to-KRaft coordination evolution) deepens understanding of its behavior, performance, and operations, and since this reflects sophisticated Kafka knowledge, understanding how Kafka works internally is valuable senior-level knowledge — deepening understanding of Kafka's performance (why it's fast despite using disk), durability (the log), and operations (KRaft), reflecting the sophisticated understanding that distinguishes deep Kafka expertise, and useful for reasoning about Kafka's behavior and the modern KRaft architecture.