Lets say that there are 20 partitions and I reduce it to 10 partitions using a df.coalesce(10)
call.
This will obviously result in shuffle where one or more partitions will need to be transferred to one single executor (that is a shuffle).
So why is there a confusion that coalesce operations do not result in shuffles ?
I agree it is not for all partitions ie. a full shuffle but shuffles do take place.