I have read some articles on OOM error in executor of a Spark application and a number of the mention high concurrency as one of the possible reasons. I am aware that the concurrency is determined by the number of cores which determine maximum number of tasks that can run within an executor.

High concurrency: If we have a disproportionate number of cores for executors, we will be processing too many partitions. Each partition will have its metadata and memory overhead. Since they are running in parallel, they will be needing memory from the executor and eventually cause OOM errors. To address this we can set the following configuration.

from above I understand multiple parallel tasks will fight for memory in order to process partitions. But I am also aware that Spark spill data into disk. So why can’t each core spill data as needed in order to facilitate un-interrupted run of the application?