Data Pipeline Observability
Instrument an Airflow and dbt pipeline with OpenLineage so every run automatically emits structured lineage events into Marquez, giving you an impact-analysis graph and run history without hand-built dashboards.
Prerequisites: a working Airflow installation (from the ETL Pipeline project) and Docker for running Marquez.
"Observability" for a data pipeline means more than logs and a green checkmark — it means being able to answer "what upstream change caused this table to look wrong" and "what breaks downstream if I change this column" without manually tracing code across DAGs and dbt projects. OpenLineage is the open specification that solves this: it defines a standard JSON event format for describing pipeline runs and the datasets they read and write, and tools across the ecosystem (Airflow, Spark, dbt, Flink) can emit events in that format without knowing anything about each other.
You'll run Marquez (the reference open-source backend for OpenLineage) locally, configure the Airflow OpenLineage provider to emit events automatically with no DAG code changes required, and trace a real cross-DAG dependency through the resulting lineage graph.
Test what you just learned
Self-testing is one of the best ways to retain new skills. Unlock project quizzes to check your understanding.
Log in to unlock0 / 7 complete