What are Kafka's delivery semantics (at-least-once, exactly-once)?

Question

Accepted Answer

Kafka supports different **delivery guarantees** — **at-most-once**, **at-least-once**, and **exactly-once** — that determine whether messages can be lost or duplicated. Understanding these and how to achieve them is important for building reliable systems.

## The three delivery semantics

```text
AT-MOST-ONCE → messages may be LOST but never duplicated:
  → commit offset BEFORE processing → if processing fails, the message is skipped (lost)
  → for: when occasional loss is acceptable and duplicates are not (rare)
AT-LEAST-ONCE → messages are never lost but may be DUPLICATED:
  → commit offset AFTER processing → if a crash occurs before commit, the message is
    reprocessed (duplicate) → requires IDEMPOTENT processing
  → the common default; safe (no loss) but handle duplicates
EXACTLY-ONCE → each message is processed exactly once (no loss, no duplicates):
  → the strongest, but hardest; requires Kafka transactions + idempotent producers
```

## Achieving the guarantees

```text
→ AT-LEAST-ONCE → process then commit; make consumers IDEMPOTENT (handle duplicates safely
  via dedup keys, idempotent operations) — practical and common
→ EXACTLY-ONCE → Kafka's transactional API + idempotent producer (enable.idempotence) →
  produce + commit offsets atomically (exactly-once within Kafka, e.g. Kafka Streams)
  ⚠️ exactly-once is complex and has overhead; often AT-LEAST-ONCE + idempotency is simpler
    and sufficient
```

## Practical guidance

```text
✓ Most systems use AT-LEAST-ONCE + idempotent processing (simple, reliable)
✓ Use EXACTLY-ONCE when duplicates are truly unacceptable AND idempotency is hard
✓ Producer acks (acks=all) → durability (don't lose on the produce side)
→ understand the trade-offs; design for the guarantee your use case needs
```

## Why it matters

Understanding Kafka's delivery semantics is valuable because **delivery guarantees determine reliability** (whether messages can be lost or duplicated), so understanding them is important for building reliable systems.

The choice of delivery semantics directly affects correctness.

Understanding the **three delivery semantics** — **at-most-once** (messages may be lost but not duplicated, by committing the offset before processing — for when occasional loss is acceptable), **at-least-once** (messages never lost but possibly duplicated, by committing after processing, requiring idempotent processing — the common, safe default), and **exactly-once** (each message processed exactly once, the strongest but hardest, requiring transactions and idempotent producers) — is essential for understanding the reliability trade-offs.

Understanding **how to achieve the guarantees** — at-least-once by processing then committing and making consumers **idempotent** (handling duplicates via dedup keys and idempotent operations — practical and common), and exactly-once via Kafka's transactional API and idempotent producer (producing and committing offsets atomically), with the important caveat that exactly-once is complex with overhead while **at-least-once plus idempotency is often simpler and sufficient** — reflects the practical approach.

Understanding the **practical guidance** — that most systems use at-least-once plus idempotent processing (simple and reliable), using exactly-once only when duplicates are truly unacceptable and idempotency is hard, and using producer acks for durability — reflects sound design judgment.

The key insight is that delivery semantics involve trade-offs (loss vs duplicates vs complexity), and the common, practical choice is at-least-once with idempotency, which connects to the broader importance of idempotency in distributed systems.

Since delivery guarantees determine whether messages can be lost or duplicated (directly affecting reliability) and understanding the semantics, how to achieve them (especially at-least-once with idempotency as the practical default), and the trade-offs is important for building reliable systems, understanding Kafka's delivery semantics is valuable, practically-important Kafka knowledge — central to reliability, important for designing systems with the right guarantees (commonly at-least-once with idempotency), and reflecting the understanding needed to build reliable Kafka-based systems.