How do batching and compression improve Kafka throughput?

Question

Accepted Answer

**Batching** (grouping messages) and **compression** (compressing data) are key techniques for high Kafka **throughput** — reducing per-message overhead and network/storage costs. Understanding them helps achieve Kafka's high performance.

## Batching — group messages

```text
BATCHING → the producer groups multiple messages into a single BATCH before sending:
  → fewer, larger requests instead of many tiny ones → much higher THROUGHPUT (less
    per-request overhead, fewer network round-trips)
  → controlled by: batch.size (batch size limit) + linger.ms (wait time to accumulate a batch)
  → trade-off: slightly higher LATENCY (waiting to batch) for much higher throughput
→ batching is essential for high-throughput producing
```

## Compression — compress batches

```text
COMPRESSION → compress message batches before sending (and store them compressed):
  → compression types: snappy, lz4, zstd (fast), gzip (higher ratio, slower)
  → benefits: LESS network bandwidth, LESS storage on brokers, often HIGHER throughput
    (less data to transfer outweighs CPU cost)
  → batches compress well (more data → better ratio) → batching + compression work together
  → consumers decompress; brokers store compressed (efficient end-to-end)
```

## Putting it together

```text
→ batch.size + linger.ms → accumulate batches → compress them → send efficiently
→ result: dramatically higher throughput vs sending tiny uncompressed messages
→ tune batch size/linger and compression type for your throughput/latency/CPU balance
→ the main producer-side throughput levers (with acks for durability)
```

## Why it matters

Understanding how batching and compression improve throughput is valuable because they're **key techniques for high Kafka performance**, so it's relevant knowledge for achieving Kafka's throughput.

Kafka achieves very high throughput, and batching and compression are central to how.

Understanding **batching** — the producer grouping multiple messages into a single batch before sending (fewer, larger requests instead of many tiny ones, dramatically increasing throughput by reducing per-request overhead and network round-trips), controlled by batch.size and linger.ms, with the trade-off of slightly higher latency for much higher throughput — clarifies a fundamental throughput technique.

Understanding **compression** — compressing message batches (with types like snappy, lz4, and zstd for speed or gzip for higher ratio), reducing network bandwidth and broker storage and often increasing throughput (since less data to transfer outweighs the CPU cost), with batches compressing well (more data, better ratio) so batching and compression work together — clarifies another key technique.

Understanding **putting it together** — accumulating batches, compressing them, and sending efficiently, dramatically increasing throughput versus tiny uncompressed messages, and tuning batch size, linger, and compression type for the throughput/latency/CPU balance — reflects how these techniques combine as the main producer-side throughput levers.

These techniques are essential for high-throughput Kafka producing, and understanding them (and their trade-offs, like batching's latency cost) is important for performance tuning.

Since batching and compression are key techniques for high Kafka throughput (reducing per-message overhead and network/storage costs) and understanding how they work, their trade-offs, and how they combine is relevant for achieving Kafka's high performance, understanding how batching and compression improve throughput is valuable, practically-relevant knowledge — key techniques for high Kafka throughput, important for performance tuning, and central to understanding how Kafka achieves its high performance on the producer side, with the throughput-vs-latency trade-off being a useful consideration.