What is Partitioning?

  • A partition is a chunk of data distributed across Spark executors.

  • Spark processes partitions in parallel → more partitions = more parallelism (but also more overhead).

  • Good partitioning reduces shuffle and improves performance.

df = spark.read.csv("users.csv", header=True, inferSchema=True)
print("Partitions:", df.rdd.getNumPartitions())

Change number of partitions:

df = df.repartition(8)   # Increase partitions
df = df.coalesce(2)      # Decrease partitions

Use repartition() for reshuffling (expensive), coalesce() for reducing partitions without full shuffle.

What is a Shuffle?

A shuffle happens when Spark redistributes data across partitions, usually during:

  • groupBy()

  • join()

  • distinct()

  • repartition()

Shuffles are expensive because they:

  • Move data between executors (network I/O).

  • Write temporary files to disk.

Example (Shuffle Caused by GroupBy)

df.groupBy("country").count().show()

This triggers a shuffle because rows need to be regrouped by country.

Broadcast Joins

  • In a join, Spark normally shuffles both tables.

  • If one table is small enough (fits in memory), Spark can broadcast it to all executors → avoids shuffling the small table.

from pyspark.sql.functions import broadcast

# Big fact table
transactions = spark.read.parquet("transactions.parquet")

# Small dimension table
countries = spark.read.csv("countries.csv", header=True)

# Broadcast join
joined = transactions.join(broadcast(countries), "country_id")

Use broadcast joins when one dataset is small (< 500 MB by default).

Partitioning + Joins

Partitioning wisely can reduce shuffles:

transactions = transactions.repartition("country_id")
joined = transactions.join(countries, "country_id")

This aligns partitions on the join key, reducing shuffle volume.

Best Practices

  • Use broadcast joins for small lookup tables.

  • Monitor number of partitions with .rdd.getNumPartitions().

  • Avoid excessive repartitioning → each shuffle costs time and memory.

  • For large data, aim for ~200–500 MB per partition.

  • Use Spark UI (http://localhost:4040) to inspect shuffles and stages.

PreviousNext