Intermediate

CDC Pipeline with Debezium

Capture row-level changes from a Postgres database in real time using Debezium on Kafka Connect, streaming inserts, updates, and deletes into Kafka topics without querying the source database directly.

~3h

0 / 7 steps

🚀

Intro

Before We Start

Prerequisites: Docker and Docker Compose, and basic SQL / Postgres familiarity.

Change Data Capture reads a database's transaction log (Postgres's write-ahead log, MySQL's binlog) directly, rather than repeatedly polling tables for what changed — which means it sees every insert, update, and delete with minimal load on the source database and none of the missed-changes-between-polls problem that a polling approach has. Debezium is the standard open-source CDC implementation, built as a set of Kafka Connect connectors.

You'll run Kafka in KRaft mode, add a Kafka Connect worker with the Debezium PostgreSQL connector, enable logical replication on a Postgres database, and watch row-level changes flow into Kafka topics as structured JSON events in real time.

🔨

Stand up Kafka, Connect, and Postgres

🔨

Create a table and register the Debezium connector

🔨

Watch change events flow into Kafka

🔨

Understand the initial snapshot

🎯

Secret Mission

Secret Mission: Land CDC events into a Delta table

🧹

Wrap Up

Before You Go

Pro

Test what you just learned

Self-testing is one of the best ways to retain new skills. Unlock project quizzes to check your understanding.

0 / 7 complete