Home / Tutorials / CSV schemas

PYSPARK SCHEMAS

PySpark CSV schema example using StructType

Example tested with PySpark 4.2.0

`inferSchema=True` is convenient while exploring a file. A repeatable pipeline should state which types it expects.

The input

transaction_id,customer_name,net_amount,tax_amount,is_member
101,Alice,250.50,25.05,true
102,Bob,120.00,6.00,false

Define the expected fields

from pyspark.sql import types as T

sales_schema = T.StructType([
    T.StructField("transaction_id", T.IntegerType(), False),
    T.StructField("customer_name", T.StringType(), False),
    T.StructField("net_amount", T.DoubleType(), True),
    T.StructField("tax_amount", T.DoubleType(), True),
    T.StructField("is_member", T.BooleanType(), True),
])

The final value records whether the field may be null. Treat it as schema information, not a replacement for checking required values after loading.

Read the file

sales = (
    spark.read
    .option("header", True)
    .option("mode", "FAILFAST")
    .schema(sales_schema)
    .csv("sales.csv")
)

sales.printSchema()

The output is predictable even if the values in next week's file change:

root
 |-- transaction_id: integer (nullable = true)
 |-- customer_name: string (nullable = true)
 |-- net_amount: double (nullable = true)
 |-- tax_amount: double (nullable = true)
 |-- is_member: boolean (nullable = true)

CSV readers may still report nullable fields because text input cannot provide the same enforcement as a database. Remove invalid business records explicitly:

valid_sales = sales.dropna(
    subset=["transaction_id", "net_amount", "tax_amount"]
)