I have job running on spark 2.4.4 on hdfs cluster. It reads and joins two sources and does some processing of this data. Job is quite massive. shuffle-partitions are set up to 10000.
Job ends up on last task of one of the stages