Kafka's architecture consists of a cluster of brokers (servers) storing partitioned, replicated topics, with producers writing and consumers reading, coordinated by metadata management (historically ZooKeeper, now KRaft). Understanding the architecture clarifies how Kafka works as a distributed system.
The components
BROKERS → Kafka SERVERS forming a CLUSTER:
→ store topic partitions (the data); serve producers and consumers
→ a cluster of multiple brokers → distributes data and load; scales horizontally
TOPICS / PARTITIONS → topics split into partitions, distributed across brokers
PRODUCERS → write events to topics (to partition leaders)
CONSUMERS (in groups) → read events from partitions
COORDINATION → metadata, cluster state, leader election:
→ historically ZOOKEEPER; now KRAFT (Kafka's own built-in consensus, removing ZooKeeper)
