Building a Lakehouse with Delta Lake

Prerequisites: completion of the Batch Processing with Spark project (or equivalent PySpark familiarity).

A plain Parquet-files-in-a-folder data lake has a real gap: there's no transaction log, so a job that fails halfway through a write can leave partial, inconsistent files behind, and there's no clean way to see "what did this table look like yesterday" or safely upsert into it. Delta Lake closes that gap by adding a transaction log (the _delta_log folder alongside your Parquet files) that gives you ACID transactions, time travel, and schema enforcement on top of storage you already have — this is the technical core of what "lakehouse" means as a term.

You'll create a Delta table, evolve its schema safely, query historical versions with time travel, and run an upsert with MERGE — the operation that's genuinely awkward on plain Parquet and straightforward on Delta.