I have lot of tables to query and extract results but some tables much time to process, I want if any particular table is taking much time, then defined time like 5 or 10 mins, I want to stop the execution for that table and move for the next table, I had tried with multiprocessing & multithreading also with cancelling spark job. It works but, in some condition, if process enters at collect method, then it hangs there working right for some.
stop the execution after timeout and move for next
New contributor
Hitesh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.