Home / Tutorials / CSV to Parquet

PYSPARK FILES

Convert CSV to Parquet with PySpark

Example tested with PySpark 4.2.0

CSV stores text. Parquet stores column types with the data, so the next Spark job can read the result without guessing the schema again.

Read CSV with a schema

from pyspark.sql import functions as F, types as T

schema = T.StructType([
    T.StructField("transaction_id", T.IntegerType(), False),
    T.StructField("net_amount", T.DoubleType(), True),
    T.StructField("tax_amount", T.DoubleType(), True),
])

sales = (
    spark.read
    .option("header", True)
    .schema(schema)
    .csv("data/sales.csv")
)

Add the result you want to keep

enriched = sales.withColumn(
    "gross_amount",
    F.round(F.col("net_amount") + F.col("tax_amount"), 2),
)

Write and read Parquet

enriched.write.mode("overwrite").parquet(
    "output/enriched_sales"
)

saved = spark.read.parquet("output/enriched_sales")
saved.printSchema()
saved.orderBy("transaction_id").show()

The saved schema includes `integer` and `double` columns. Spark does not need `inferSchema` when it reads the Parquet dataset.

On native Windows, Hadoop may fail while committing local Parquet output unless its Windows binaries are configured. Run the example in WSL or the official Apache Spark Docker image if you encounter `NativeIO$Windows.access0`.