Stream Processing with Kafka

Prerequisites: Docker and Docker Compose, and Python 3.10+.

Kafka 4.0 (released 2025) made a change that's easy to miss if you learned Kafka from an older tutorial: ZooKeeper isn't deprecated, it's removed entirely. Kafka 4.x runs exclusively in KRaft mode, where the metadata quorum lives inside Kafka itself via Raft consensus among controller nodes, instead of a separate ZooKeeper cluster. If a setup guide still shows a zookeeper: service in its Docker Compose file, it's describing a pre-4.0 architecture.

You'll run a single-node KRaft broker, write a Python producer that publishes order events to a topic, and a consumer group that processes them with manual offset commits — the pattern that gives you control over exactly when a message is considered "done" rather than trusting Kafka's auto-commit timer.