How do producers and consumers work in Kafka?

Question

Accepted Answer

**Producers** publish (write) events to Kafka topics, and **consumers** subscribe to (read) events from them. Understanding how they work — including partitioning, offsets, and consumer groups — is fundamental to using Kafka.

## Producers — writing events

```text
PRODUCERS publish events to TOPICS:
  → send a record (key, value) to a topic
  → the key determines the PARTITION (same key → same partition → ordering for that key);
    no key → distributed (round-robin)
  → can configure ACKNOWLEDGMENTS (acks) for durability (wait for replicas to confirm)
  → batching, compression for throughput
```

## Consumers — reading events

```text
CONSUMERS subscribe to topics and read events:
  → read from partitions, tracking their OFFSET (position) → resume from where they left off
  → read at their OWN PACE (events are retained); can re-read (replay) from any offset
  → COMMIT offsets to record progress (so they don't reprocess on restart)
```

## Consumer groups (key for scaling)

```text
CONSUMER GROUP → consumers working together to consume a topic:
  → each partition is consumed by ONE consumer in the group → parallel processing
    (distribute partitions across the group's consumers)
  → add consumers (up to the partition count) to scale consumption
  → if a consumer fails, its partitions are REBALANCED to others (fault tolerance)
DIFFERENT GROUPS → each group reads ALL the events independently (pub/sub: many groups
  each get the full stream)
```

## Why it matters

Understanding how producers and consumers work is fundamental because they're **how data flows in and out of Kafka**, so it's essential for using Kafka.

Producers write events and consumers read them — the basic data flow.

Understanding **producers** — publishing events to topics, with the **key determining the partition** (same key going to the same partition for ordering, no key distributing round-robin), configuring acknowledgments for durability, and batching for throughput — covers how data enters Kafka.

Understanding **consumers** — reading events while tracking their **offset** (position) to resume from where they left off, reading at their own pace since events are retained, being able to **replay** from any offset, and committing offsets to record progress — covers how data is read, including Kafka's distinctive replay capability.

Most importantly, understanding **consumer groups** is essential for Kafka's scaling model: consumers working together where **each partition is consumed by one consumer in the group** (enabling parallel processing, scaling by adding consumers up to the partition count), with automatic **rebalancing** if a consumer fails (fault tolerance), while **different consumer groups each read all events independently** (enabling pub/sub where multiple groups each get the full stream).

This consumer-group model is fundamental to how Kafka scales consumption and supports both queue-like (within a group) and pub/sub (across groups) patterns.

Since producers and consumers are how data flows in and out of Kafka, and since understanding their workings — partitioning, offsets, replay, and especially consumer groups (the key to scaling and the queue/pub-sub duality) — is essential for using Kafka, understanding how producers and consumers work is essential, foundational Kafka knowledge — fundamental to the data flow in Kafka, with consumer groups being especially important for understanding Kafka's scaling and consumption model, and necessary for anyone using Kafka.