Home / Tutorials / Local installation

PYSPARK SETUP

How to install PySpark locally on Windows, VS Code or Ubuntu

Tested with PySpark 4.2.0, Python 3.13 and Java 21

A successful pip install does not prove that Python can start Spark's Java process. This guide sets up an isolated environment, connects it to VS Code and tests an actual Parquet write.

Choose the setup that matches your computer

Use native Windows if you want the quickest route into a Python script. Use WSL when you want a Linux environment on a Windows computer. Ubuntu and macOS can follow the same terminal workflow. Docker is useful for an isolated check, but it adds another tool when you are still learning.

The examples below use PySpark 4.2.0. It supports Python 3.10 or newer and requires Java 17 or newer. Apache's official PySpark installation page is the source of truth when a later release changes those requirements.

1. Check Python and Java

Open PowerShell, a WSL shell or an Ubuntu terminal and run:

python --version
java -version

On some Ubuntu installations, the Python command is python3. If java is not recognised, install a current JDK, close the terminal and open it again. Then check where Java is available:

# Windows PowerShell
Get-Command java

# WSL, Ubuntu or macOS
which java

2. Create a virtual environment and install PySpark

A virtual environment keeps PySpark and its Python dependencies separate from other projects.

Windows PowerShell

py -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install pyspark==4.2.0

If PowerShell blocks the activation script, Python's virtual environment documentation recommends enabling locally created scripts for your user account:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

WSL or Ubuntu

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install pyspark==4.2.0

If Ubuntu says that venv is unavailable, install the package that matches your Python version with the system package manager, then create the environment again.

macOS

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install pyspark==4.2.0

3. Use the PySpark environment in VS Code

Open the project folder in VS Code. Install Microsoft's Python extension, then open the Command Palette and choose Python: Select Interpreter. Select the interpreter inside .venv.

If you use WSL, first open the folder through the WSL extension. The bottom-left corner of VS Code should identify the WSL distribution, and the selected interpreter should end in .venv/bin/python. A Windows interpreter cannot use packages installed inside WSL, and the reverse is also true.

Confirm which Python VS Code will run:

python -c "import sys; print(sys.executable)"
python -c "import pyspark; print(pyspark.__version__)"

4. Start a local Spark session

Create setup_check.py in the project folder:

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .master("local[*]")
    .appName("SetupCheck")
    .getOrCreate()
)

print(f"Spark version: {spark.version}")
print(f"Master URL: {spark.sparkContext.master}")

rows = [(1, "ready"), (2, "working")]
frame = spark.createDataFrame(rows, ["id", "status"])
frame.write.mode("overwrite").parquet("output/setup_check")
print(f"Rows written: {spark.read.parquet('output/setup_check').count()}")

spark.stop()

Run it from the activated terminal:

python setup_check.py

Spark prints several log lines. The useful output is:

Spark version: 4.2.0
Master URL: local[*]
Rows written: 2

local[*] tells Spark to use the available processor cores on your machine. Reading back two rows proves that the Python process, Java process and local file output all work together. Apache's DataFrame quickstart is a useful next reference.

Common installation problems

Java gateway process exited before sending its port number

Run java -version in the same terminal that runs the Python script. If it fails, Java is not on that environment's path. WSL needs Java installed inside WSL, even when Java is already installed on Windows.

JAVA_HOME is not set

Most current JDK installers make java available without a manual variable. If another tool requires JAVA_HOME, set it to the JDK directory, not to the bin directory. Open a new terminal after changing it.

VS Code says there is no module named pyspark

VS Code is using a different Python interpreter. Run python -c "import sys; print(sys.executable)" in its terminal and select that same .venv interpreter from Python: Select Interpreter.

PySpark starts but cannot write files on Windows

First try a short project path that you can write to, such as C:\code\spark-test. If native Windows file-system errors continue, run the project in WSL and keep it in the Linux file system, such as ~/code/spark-test.

Python version mismatch

Delete and recreate .venv with the intended Python version. Installing PySpark into one interpreter does not make it available to every Python installation on the computer.

What to learn next

Once the setup check passes, learn how to define an explicit PySpark schema, then build a complete CSV-to-Parquet pipeline. Those examples use the same local session you just verified.