What are the core concepts of Kafka (topics, partitions, brokers)?

Question

Accepted Answer

Kafka's core concepts include **topics** (categories of events), **partitions** (the units of parallelism within topics), **brokers** (the servers), **producers** (write), and **consumers** (read). Understanding these is fundamental to working with Kafka.

## Core concepts

```text
TOPIC → a named category/stream of events (like a "channel" — e.g. "orders", "clicks")
  → producers write to topics; consumers read from them
PARTITION → a topic is split into PARTITIONS (ordered, append-only logs):
  → the unit of PARALLELISM and scaling; events distributed across partitions
  → order is guaranteed WITHIN a partition (not across partitions)
BROKER → a Kafka SERVER (stores data, serves clients); a CLUSTER = multiple brokers
PRODUCER → writes (publishes) events to topics
CONSUMER → reads (subscribes to) events from topics
```

## Partitions and offsets

```text
→ each partition is an ordered, immutable sequence of events
→ each event has an OFFSET (its position in the partition)
→ consumers track their offset (where they've read to) → can resume, replay
→ events stay in the partition (retained); consumers read at their own pace
→ a partition KEY determines which partition an event goes to (same key → same partition
  → ordering for that key)
```

## How it fits together

```text
Producer → writes to a TOPIC (→ a PARTITION based on key) → stored on a BROKER (replicated)
  → Consumer reads from the partition (tracking its offset)
→ topics organize events; partitions enable parallelism/scale; brokers store/serve;
  offsets track position
```

## Why it matters

Understanding Kafka's core concepts is fundamental because **topics, partitions, brokers, producers, and consumers are the building blocks of Kafka**, so grasping them is essential for any Kafka work.

These concepts form the foundation of how Kafka works.

Understanding **topics** (named categories/streams of events that producers write to and consumers read from) is the basic organizing concept.

Understanding **partitions** is particularly important: topics are split into partitions (ordered, append-only logs) that are the **unit of parallelism and scaling**, with the crucial property that **order is guaranteed within a partition but not across partitions** — a key concept affecting how Kafka scales and orders events.

Understanding **brokers** (the Kafka servers, forming a cluster), **producers** (writing events), and **consumers** (reading events) completes the basic architecture.

Understanding **partitions and offsets** — each partition being an ordered immutable sequence where events have **offsets** (positions), consumers tracking their offset to resume and replay, events being retained so consumers read at their own pace, and the **partition key** determining which partition an event goes to (same key → same partition → ordering for that key) — is essential for understanding Kafka's data model and ordering guarantees.

Understanding **how it fits together** (producers writing to topics/partitions stored on replicated brokers, consumers reading while tracking offsets) provides the complete picture.

These concepts are the vocabulary and mechanics of Kafka that everything else builds on.

Since topics, partitions, brokers, producers, consumers, and offsets are the fundamental building blocks of Kafka (with partitions and ordering being especially important concepts), and since understanding them is essential for any Kafka work, understanding Kafka's core concepts is essential, foundational knowledge — the building blocks of Kafka necessary for working with it, central to understanding how Kafka organizes, scales, orders, and stores events, and the foundation for all Kafka topics.