When working with Spark, it’s important to understand the difference between transformations and actions. This distinction explains why some operations appear lazy and why Spark waits until the last possible moment to actually compute results.
Transformations
A transformation in Spark defines how data should be processed, but it does not execute immediately. Instead of modifying data directly, transformations create a new RDD or DataFrame with the specified changes.
This lazy behavior allows Spark to build up a logical execution plan (a DAG of operations) without running each step right away. The actual computation only happens later, when an action is called.
Examples of transformations include:
df2 = df.filter(df["age"] > 30) # Filter rows
df3 = df2.select("name", "age") # Select columns
Here, Spark simply records the operations, but nothing is executed yet.
Narrow Transformations
A narrow transformation means each input partition contributes to only one output partition. Since data remains on the same worker node, these transformations are faster and more efficient. Example: .filter(), .map().
Wide Transformations (Shuffles)
A wide transformation means data from one partition is needed by multiple output partitions, requiring data to be shuffled across the cluster. This makes them more resource-intensive. Example: groupBy(), join().
Warning
Be aware that when using wide transformations, they can significantly slow down your applications.
Actions
Once transformations are defined, an action is what tells Spark to actually compute and return results. Actions trigger execution of the entire logical plan that has been built up to that point.
Examples of actions include:
df3.show() # Display results
df3.count() # Return row count
df3.collect() # Bring data to driverTip
The output of transformations is always an RDD, whereas the output of actions includes everything except an RDD.