How do you produce and consume messages in Kafka (basic usage)?

Question

Accepted Answer

Basic Kafka usage involves **producing** messages to a topic and **consuming** them — typically via a client library (Java, Python, etc.). Understanding the basic produce/consume flow is the practical entry point to using Kafka.

## Producing messages

```js
// using a Kafka client (e.g. KafkaJS for Node)
const producer = kafka.producer();
await producer.connect();
await producer.send({
  topic: 'orders',
  messages: [
    { key: 'user-123', value: JSON.stringify({ orderId: 1, amount: 100 }) },
    // key determines the partition (same key → same partition → ordering)
  ],
});
// → publishes the message to the 'orders' topic
```

## Consuming messages

```js
const consumer = kafka.consumer({ groupId: 'order-processors' });   // a consumer GROUP
await consumer.connect();
await consumer.subscribe({ topic: 'orders', fromBeginning: false });
await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const data = JSON.parse(message.value.toString());
    // process the message...
    // offset is auto-committed (or commit manually for control)
  },
});
```

## Key practical points

```text
✓ Use a client LIBRARY for your language (Java, KafkaJS, confluent-kafka-python, etc.)
✓ Producers: set a KEY for ordering/partitioning; configure acks for durability
✓ Consumers: use a GROUP ID; the group distributes partitions; commit offsets
✓ Handle SERIALIZATION (JSON, Avro, Protobuf) — often a schema registry for structured data
✓ Handle errors, retries; process idempotently (at-least-once → possible duplicates)
```

## Why it matters

Understanding basic produce/consume usage is valuable because it's the **practical entry point to using Kafka**, so it's necessary hands-on knowledge for working with Kafka.

Knowing how to produce and consume messages is the fundamental skill for using Kafka in applications.

Understanding **producing messages** — connecting a producer and sending records (key, value) to a topic, with the **key determining partitioning and ordering** — is the basic write operation.

Understanding **consuming messages** — connecting a consumer with a **group ID**, subscribing to a topic, and processing messages (with offsets committed automatically or manually) — is the basic read operation, with the consumer group being how consumption scales.

Understanding the **key practical points** — using a client library for your language (Java, KafkaJS, Python, etc.), setting keys for ordering and configuring acks for durability on producers, using group IDs and committing offsets on consumers, handling **serialization** (JSON, Avro, Protobuf, often with a schema registry for structured data), and handling errors, retries, and **idempotent processing** (since at-least-once delivery means possible duplicates) — reflects the practical considerations for real Kafka usage.

These practical points (keys, groups, serialization, idempotency) connect the conceptual knowledge (partitions, consumer groups, delivery semantics) to actual code.

Since producing and consuming messages is the fundamental practical skill for using Kafka in applications, and since understanding the basic flow and the practical points (keys, groups, serialization, idempotency) is necessary hands-on knowledge, understanding basic Kafka produce/consume usage is valuable, practically-relevant knowledge — the practical entry point to using Kafka, connecting concepts to code, and necessary for anyone actually building applications with Kafka.