I’m trying to integration Spark Connect(spark-connect) into my spark jobs. It throws circular import error for importing Spark Session inside Celery
I’m trying to integration Spark Connect(spark-connect) into my spark jobs. For running the jobs in background I use celery combined with eventlet with concurrency of 5. This works fine for Spark Cluster which is setup in local, which has a Master and one Worker node.