Home / Tutorials / Missing values

PYSPARK DATA CLEANING

Drop rows with nulls in required columns

Example tested with PySpark 4.2.0

Do not discard a valid sale because an optional note is empty. Name the columns that make the row usable.

Start with required and optional data

sales = spark.createDataFrame(
    [
        (101, 250.50, 25.05, "priority"),
        (102, 120.00, None, "standard"),
        (103, 450.75, 25.07, None),
    ],
    ["transaction_id", "net_amount", "tax_amount", "note"],
)

Check only the required columns

required = ["transaction_id", "net_amount", "tax_amount"]

valid_sales = sales.dropna(subset=required)
valid_sales.orderBy("transaction_id").show()

+--------------+----------+----------+--------+
|transaction_id|net_amount|tax_amount|    note|
+--------------+----------+----------+--------+
|           101|     250.5|     25.05|priority|
|           103|    450.75|     25.07|    NULL|
+--------------+----------+----------+--------+

Transaction 102 is removed because tax is required. Transaction 103 remains because note is optional.

Count rejected rows

from pyspark.sql import functions as F

invalid_sales = sales.filter(
    F.col("transaction_id").isNull()
    | F.col("net_amount").isNull()
    | F.col("tax_amount").isNull()
)

print("rejected:", invalid_sales.count())

Keep the rejected count in your job logs. A pipeline that quietly drops half its input is still a broken pipeline.