Data Engineer
Move data reliably at scale: pipelines, warehouses, streaming.
30+
completed
~23h
total time
9
projects
ETL Pipeline with Airflow
Build a scheduled ETL pipeline in Apache Airflow 3 using the TaskFlow API and Assets, moving CSV data through transformation into Postgres.
Build a Data Warehouse on Snowflake
Model a small star-schema warehouse in Snowflake, load data with COPY INTO and stages, and build an incremental pipeline with Streams and Tasks.
Stream Processing with Kafka
Stand up a KRaft-mode Kafka broker, write a producer and consumer group, and implement exactly-once-style processing with manual offset commits.
Batch Processing with Spark
Process a multi-file dataset with PySpark's DataFrame API, applying explicit schemas, broadcast joins, and partitioning strategies.
Data Quality and Validation Framework
Define, run, and document automated data-quality checks with Great Expectations, gating a pipeline so it fails loudly on bad data.
Building a Lakehouse with Delta Lake
Add ACID transactions, schema enforcement, time travel, and upserts to a plain Parquet data lake using Delta Lake on top of PySpark.
Orchestrating dbt Transformations
Build a small dbt Core project with staging and mart models, add tests, and trigger it from Airflow as part of a scheduled pipeline.
CDC Pipeline with Debezium
Capture row-level changes from a Postgres database in real time using Debezium on Kafka Connect, streaming them into Kafka topics.
Data Pipeline Observability
Instrument an Airflow and dbt pipeline with OpenLineage so every run emits structured lineage events into Marquez.