Beginner

ETL Pipeline with Airflow

Build a scheduled Extract-Transform-Load pipeline in Apache Airflow 3 using the TaskFlow API and Assets, moving data from a CSV source through transformation into a Postgres warehouse table.

~2.5h

0 / 7 steps

🚀

Intro

Before We Start

Prerequisites: Python 3.10+, Docker and Docker Compose, and basic SQL familiarity.

Airflow 3.0, released in 2025, is the biggest architectural change in the project's history: DAG authoring moved to a stable, decoupled airflow.sdk namespace, task execution runs through a new Task Execution API rather than talking to the metadata database directly, and DAGs are now versioned so a run always executes against the code that was live when it started, rather than whatever the DAG file looks like today. If your only exposure to Airflow is older 2.x tutorials, expect several import paths to have moved.

You'll build a daily ETL DAG using the TaskFlow API (the @task decorator style introduced in 2.0 and still the recommended way to author most pipelines) that extracts from a CSV, transforms it in Python, and loads it into Postgres — then upgrade it to use Airflow 3's Assets feature so a downstream DAG can react automatically when this one produces fresh data.

🔨

Stand up Airflow 3 locally with Docker Compose

🔨

Write the ETL DAG with the TaskFlow API

🔨

Add retries, alerting, and a data-quality gate

🔨

Upgrade to Asset-aware, event-driven scheduling

🎯

Secret Mission

Secret Mission: Add a backfill and inspect DAG versioning

🧹

Wrap Up

Before You Go

Pro

Test what you just learned

Self-testing is one of the best ways to retain new skills. Unlock project quizzes to check your understanding.

0 / 7 complete