I am trying to explore Spark-Connect.
I could successfully spin up Spark-Connect, Spark Workers and Spark master in a dockerised way.
Using a simple Pyspark code, I am able to push and execute some simple Dataframe queries to Spark via Spark-Connect. Here is a simple code example,