I need to persist certain information into parquet file to be accessed and updated during one batch job or the next (e.g. average values, slopes etc).

I am using PySpark and implemented a some pipelines using batch processing. This pipelines need to save some state between batches so i created my own state manager (is there a better way in general?):

I have a pipeline that does much exploding at the beginning (pyspark):

I am loading the multiple parquet under a directory but the data type for one of the column is not inferring properly. I tried couple of setting below mentioned on the internet and stackoverflow

In a Spark data pipeline, I want to rely on mapPartitions to run some computations. I prepare some data and want to store it in partitions using DataFrameWriter.partitionBy.