What are Redis Streams?

Question

Accepted Answer

**Redis Streams** are an append-only log data structure — like a durable message queue or event log — supporting **consumer groups**, message persistence, acknowledgments, and replay. They're Redis's answer to reliable messaging, more robust than Pub/Sub (which is fire-and-forget).

## Append-only log of messages

```bash
# add entries to a stream (each gets an auto ID: timestamp-sequence)
XADD events * type "signup" user "ann"     # * = auto-generate ID; returns e.g. 1700000000-0
XADD events * type "login" user "bob"

XLEN events                                 # number of entries
XRANGE events - +                           # read all entries (- to +)
XREAD COUNT 10 STREAMS events 0             # read entries (from ID 0)
```

Streams are an **append-only log**: entries are added with `XADD` (each with a unique, time-ordered ID) and persist in the stream — unlike Pub/Sub, messages aren't lost; they're stored and can be re-read.

## Consumer groups — reliable, scalable processing

```bash
# create a consumer group to process the stream
XGROUP CREATE events workers 0

# consumers read messages (each message goes to ONE consumer in the group)
XREADGROUP GROUP workers consumer1 COUNT 1 STREAMS events >

# acknowledge a processed message (so it's not redelivered)
XACK events workers 1700000000-0
# unacknowledged messages can be reclaimed (XPENDING/XCLAIM) → no message lost on crash
```

**Consumer groups** distribute messages across multiple consumers (scaling processing), with each message going to one consumer, **acknowledgments** (`XACK`) confirming processing, and **pending/reclaim** (`XPENDING`/`XCLAIM`) ensuring messages aren't lost if a consumer crashes — reliable, at-least-once delivery.

## Streams vs Pub/Sub

```text
PUB/SUB  → fire-and-forget; offline subscribers MISS messages; no persistence/replay
STREAMS  → PERSISTENT log; messages stored; consumer groups; acknowledgments; replay
  → Streams = when you need RELIABLE delivery, message history, and scalable consumers
  → Pub/Sub = simple real-time broadcast where missing messages is OK
Streams are similar to a lightweight Kafka (log + consumer groups).
```

## Why it matters

Understanding Redis Streams is valuable for **reliable messaging and event processing** within Redis, so it's useful knowledge — particularly because Streams address the key limitations of Pub/Sub.

While Pub/Sub is simple but fire-and-forget (offline subscribers permanently miss messages, no persistence), **Streams provide reliable messaging**: an **append-only persistent log** where messages are stored (not lost) and can be re-read/replayed, plus **consumer groups** that distribute messages across multiple consumers for scalable processing, **acknowledgments** (`XACK`) to confirm processing, and **pending-message reclaim** (`XPENDING`/`XCLAIM`) so messages aren't lost when a consumer crashes — delivering at-least-once, reliable, scalable message processing.

Understanding **Streams vs Pub/Sub** (Streams for reliable delivery, history, and scalable consumers; Pub/Sub for simple real-time broadcast where loss is acceptable) is important for choosing the right Redis messaging primitive.

Recognizing that Streams are **similar to a lightweight Kafka** (log-based with consumer groups) helps frame their role — providing event-streaming/message-queue capabilities within Redis without a separate broker for moderate needs, while dedicated brokers (Kafka, RabbitMQ) remain appropriate for larger-scale or more demanding messaging.

Since reliable messaging and event processing are common needs, and since Redis Streams provide a robust, persistent, consumer-group-based solution (overcoming Pub/Sub's fire-and-forget limitation) within Redis, understanding Redis Streams — the append-only log, consumer groups with acknowledgments and reclaim, and how they compare to Pub/Sub and Kafka — is valuable, practically-relevant knowledge for building reliable messaging and event-driven features with Redis, the right choice when you need delivery guarantees and message history that Pub/Sub can't provide.