Partitions

Partitioning in Spark refers to the division of data into smaller, more manageable chunks known as partitions. Partitions are the basic units of parallelism in Spark, and they allow the framework to process different portions of the data simultaneously on different nodes in a cluster. A partition is a collection of rows that sit on one physical machine in your cluster

In the example below, the DataFrame on the left is divided into three partitions: P1, P2, and P3, with each partition containing two rows. Partition P1 is assigned to Executor 1, while P2 and P3 are assigned to Executor 2.

⚠️ SET THE RIGHT PARTITION NUMBER

Too few partitions lead to large partition sizes, potentially causing disk spills and slowing down queries. Too many partitions result in small partition sizes, leading to inefficient I/O patterns and an increased burden on the Spark task scheduler. Setting the right number of partitions is a key perfoamnce for your Spark Applications.