Install PySpark locally
Install PySpark on Windows, WSL or Ubuntu, connect it to VS Code, then verify a real local session.
PYSPARK FOR PYTHON DEVELOPERS
Build your first PySpark pipeline on your own machine.
Start with the free guide to understand Spark, DataFrames and lazy execution. When you are ready to build something, the Starter Kit gives you the code, data, tests and exercises.
No subscription. The kit code is tested with PySpark 4.2.0.
reading sales.csv joining customers.csv writing output/enriched_sales transaction_id customer gross 101 Alice 275.55 102 Bob 126.00 103 Charlie 475.82 104 David 95.72 4 valid transactions
START HERE
Read the beginner guide free
Understand the mental model before worrying about cluster tuning.
PySpark for Beginners: Mastering the Basics
The guide explains what Spark does, how a local cluster works, why transformations are lazy, and how to create and process your first DataFrame.
- Install PySpark and start a local Spark session
- Create a DataFrame and inspect its schema
- Read CSV data and add calculated columns
- See the input and output for every example
Free PDF. No payment details required. Your email is not shared or sold.
WHEN YOU WANT TO BUILD IT
The PySpark Starter Kit
A small working project that reads messy CSV input, applies explicit schemas, removes incomplete and duplicate sales, joins customer data, calculates totals and writes Parquet.
pyspark-starter-kit/ |-- guides/ |-- data/ |-- notebooks/ |-- src/pipeline.py |-- tests/ |-- exercises/ |-- solutions/ `-- cheatsheets/
FREE TUTORIALS
Solve one PySpark problem at a time
Stop letting Spark guess CSV types
Define a schema and make malformed input fail where you can diagnose it.
See lazy evaluation in code
Build transformations, trigger an action and inspect the execution plan.
Turn CSV input into Parquet
Write typed columnar output and read it back without inferring the schema again.
Remove duplicate rows by transaction ID
Use dropDuplicates() with a business key and check which records remain.
Drop rows with missing required values
Apply dropna() to required columns without discarding useful optional data.
Keep unmatched rows in a PySpark left join
Join sales to customer data while retaining transactions without a lookup match.
Test a PySpark DataFrame with pytest
Create one Spark session, collect a small result and assert the rows that matter.
PAID TUTORIALS
Choose your next developer guide
Browse the other developer guides
Each guide shows its length and exactly what it teaches.
GitHub Actions for Python CI/CD, Part 1
£4.9911 pages. Build a GitHub Actions workflow for a Python project and run Pylint whenever code is pushed.
View guideGitHub Actions for Python CI/CD, Part 2
£4.9915 pages. Add automated pytest checks to the pipeline, then deploy the tested output to an AWS S3 bucket.
View guideSQL Grouping Sets, Rollup and Cube
£4.998 pages. Use grouping sets, rollup and cube to produce subtotals and multi-level reporting results in SQL.
View guideAgent Toolkit for AWS
£4.9915 pages. See how the Agent Toolkit can plan, reason about and automate AWS-focused development work.
View guideFrom Local App to Public Website
£4.9910 pages. Publish a local app with GitHub Pages, here.now or Hugging Face Spaces and compare when each route fits.
View guideFine-Grained Image Editing
£4.996 pages. Write structured and JSON-style instructions for more precise edits in Nano Banana Pro.
View guide