Pyspark Azure Synapse – kryoserializer Buffer overflow
I am facing a problem with the Azure Synapse Notebook. I have a big python script where is used the Pandas Dataframe, I can load a ‘parquet’ file, but I cannot convert into pandas using toPandas(), because is throwing the error: 'org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow'.