Home / Tutorials / Lazy evaluation

PYSPARK EXECUTION

PySpark lazy evaluation explained with a DataFrame example

Example tested with PySpark 4.2.0

A DataFrame transformation describes work. An action asks Spark for a result and causes that work to run.

Build a DataFrame

sales = spark.createDataFrame(
    [
        (101, 250.50, 25.05),
        (102, 120.00, 6.00),
        (103, 450.75, 25.07),
    ],
    ["transaction_id", "net_amount", "tax_amount"],
)

Add transformations

from pyspark.sql import functions as F

high_value = (
    sales
    .withColumn(
        "gross_amount",
        F.col("net_amount") + F.col("tax_amount"),
    )
    .filter(F.col("gross_amount") >= 200)
)

`withColumn()` and `filter()` return a new DataFrame description. Spark has enough information to construct a logical plan, but your code has not requested rows yet.

Inspect the plan, then run it

high_value.explain(mode="formatted")

print("rows:", high_value.count())

`explain()` prints Spark's plan. `count()` is the action that executes it. For this input, the final line is:

rows: 2

Other common actions include `show()`, `collect()`, `write.parquet()` and `first()`. If a transformation appears to do nothing, check whether your program has reached an action.