What are offsets in Kafka and how do they work?

Question

Accepted Answer

An **offset** is a unique, sequential ID for each event within a partition — marking its position. Consumers track and commit offsets to know what they've consumed, enabling resuming, replay, and reliable processing.

## What offsets are

```text
OFFSET → the position/ID of an event within a PARTITION:
  → each partition is an ordered log; events get incrementing offsets (0, 1, 2, ...)
  → an offset uniquely identifies an event's position in its partition
  → events are RETAINED → the log keeps events (offsets remain valid for the retention period)
```

## How consumers use offsets

```text
→ a consumer reads events in order and tracks its CURRENT offset (where it has read to)
→ COMMITTING an offset → records "I've processed up to here" → on restart, resume from
  the committed offset (don't reprocess everything)
→ consumers can SEEK to any offset → REPLAY old events, skip ahead, reprocess from a point
→ each consumer GROUP tracks its own offsets per partition (independent progress)
```

## Offset management and delivery semantics

```text
WHEN/HOW you commit offsets affects DELIVERY GUARANTEES:
  → commit AFTER processing → AT-LEAST-ONCE (a crash before commit → reprocess → duplicates;
    need idempotency)
  → commit BEFORE processing → AT-MOST-ONCE (a crash after commit but before processing →
    message lost)
  → EXACTLY-ONCE → requires more (transactions, idempotent processing)
→ auto-commit (periodic) vs manual commit (control) → choose based on guarantees needed
```

## Why it matters

Understanding offsets is fundamental because they're **how Kafka tracks consumption progress and enables its distinctive capabilities** (resuming, replay, delivery guarantees), so it's essential Kafka knowledge.

An offset — the position of an event within a partition — is central to how consumers work with Kafka.

Understanding **what offsets are** (sequential IDs marking each event's position in a partition's ordered log, with events retained so offsets remain valid) is the foundation.

Understanding **how consumers use offsets** — tracking their current offset (where they've read to), **committing offsets** to record progress (so they resume from the committed point on restart rather than reprocessing everything), and being able to **seek to any offset to replay** old events or reprocess from a point (Kafka's distinctive replay capability), with each consumer group tracking its own offsets independently — is essential for understanding how Kafka consumption, resumption, and replay work.

Most importantly, understanding **offset management and delivery semantics** is crucial: **when and how you commit offsets affects delivery guarantees** — committing after processing gives **at-least-once** (a crash before commit causes reprocessing/duplicates, requiring idempotency), committing before gives **at-most-once** (a crash after commit but before processing loses the message), and **exactly-once** requires more (transactions, idempotent processing).

This connection between offset management and delivery guarantees is a key practical concept for building reliable Kafka consumers.

Understanding auto-commit vs manual commit (control over guarantees) reflects practical offset management.

Since offsets are how Kafka tracks consumption progress and enable its distinctive capabilities (resuming, replay) while their management determines delivery guarantees (at-least-once, at-most-once, exactly-once), and since understanding offsets is essential for building reliable consumers, understanding offsets in Kafka is essential, foundational knowledge — central to how Kafka tracks consumption and enables replay, and crucially connected to delivery guarantees through offset management, essential for building reliable Kafka consumers and a core Kafka concept.