I am using SparkListener and listening on onStageCompleted
event. I want to check the query execution plan and get the table names from that plan if there is any join operation. How can I do this? If I use QueryExecutionListener
, I am unable to relate it to a specific stage. Is it possible to get stage metrics, query plan for that stage at a time?