Getting stackoverflow error In Azure Synapse Notebook (PySpark) while trying to create a view from dataframe using createOrReplaceTempView(). When I divide the notebook into 3 parts , distributing the transformations and joins, there is no error. But I want the notebook to execute in a single part.
Things I have already tried-
- Enabling Adaptive Query Execution
- Reducing Spark Shuffle Partitions
- Increasing Broadcast Memory