Batch Processing with Spark
Process a multi-file dataset with PySpark's DataFrame API, applying explicit schemas, broadcast joins, and partitioning strategies, and inspect how Spark's query planner executes it under Adaptive Query Execution.
Prerequisites: Java 17+ installed, Python 3.10+, and 8GB+ RAM for local Spark.
Spark's core value proposition hasn't changed since its earliest versions — distribute a computation across many machines instead of one — but the current API surface (PySpark 4.1.x as of mid-2026) sits firmly on the DataFrame API, with the older RDD API considered a low-level escape hatch rather than the primary interface. Spark 4.x requires Java 17 or later and is built exclusively against Scala 2.13, both of which trip up people following older setup guides.
You'll process a multi-file CSV dataset with explicit schemas (avoiding an expensive full-file schema-inference scan), join it against a small lookup table using a broadcast join, and use Spark's built-in UI to see how Adaptive Query Execution — on by default since Spark 3.2 — actually reshapes the query plan at runtime based on real data statistics rather than a static plan.
Test what you just learned
Self-testing is one of the best ways to retain new skills. Unlock project quizzes to check your understanding.
Log in to unlock0 / 8 complete