What is Lazy Evaluation?

Lazy evaluation means delaying the execution of transformations until an action is explicitly called.

  • When you apply transformations (e.g., filter, select, map), Spark does not immediately process the data.

  • Instead, it builds a logical execution plan describing the steps needed.

  • Only when an action (e.g., show(), count(), collect(), write()) is called does Spark:

    1. Compile the logical plan into an optimized physical plan.

    2. Distribute the work efficiently across the cluster.

Why Lazy Evaluation?

This design gives Spark several advantages:

  1. Global Optimization

    • Spark can analyze the full sequence of transformations before execution.

    • This allows it to combine, rearrange, or eliminate unnecessary operations.

  2. Efficiency at Scale

    • For example, predicate pushdown ensures filters (WHERE clauses) are applied as early as possible, reducing the amount of data read.

  3. Fault Tolerance

    • Because Spark remembers the transformation plan (not just results), it can recompute missing data after a failure.

Lazy Evaluation in Action

Open in Colab 
# Step 1: Read CSV (transformation, no execution yet)
df = spark.read.csv("users.csv", header=True)

# Step 2: Apply filter (still a transformation, Spark just records it)
filtered = df.filter(df['user_id'] > 1)

# Step 3: Action – triggers execution
filtered.show()

👉 Notes:

  • Steps 1 & 2 don’t actually read or filter the data yet.

  • Only at Step 3 (show() = action), Spark executes the plan:

    1. Reads the CSV.

    2. Applies the filter.

    3. Displays the results.

Summary

  • Spark uses lazy evaluation to defer execution until an action is called.

  • This allows Spark to optimize the full DAG (Directed Acyclic Graph) of transformations.

  • Actions trigger execution, while transformations just define the plan.

  • Lazy evaluation is a key reason why Spark is both efficient and scalable.

PreviousNext