I am trying to read delta files using pyspark in my local machine but i am facing an issue.
When i am using the following code i am not getting any error as idWithoutTopologyInfo is null.

I have a pyspark application that runs fine locally with master “local”. Now i want to spark-submit the application to a simple cluster (standalone, client mode, running in docker).

I have a pyspark application that runs fine locally with master “local”. Now i want to spark-submit the application to a simple cluster (standalone, client mode, running in docker).

With this example:

I have a spark job where I passed all the conf parameter as form of spark-submit where as in other job (same code and same data) where I created spark session and added same config . However the the job execution time is quite different. Does it really matter in peroformance?

I have have a docker container running spark connect. Both the apache and bitmani images work.

I have multiple question about how spark handles data internally.

I am importing pyspark libraries as follows

I have a dataframe and I am trying to do following transformation.

The existing dataframe is
| header | body |
| ——– | ————– |
| xxx | ‘{“name”:”john”,”age”:20,”emails”:[“[email protected]”,”[email protected]”]}’|
| xxx | ‘{“name”:”jerry”,”age”:30,”emails”:[“[email protected]”,”[email protected]”]}’ |