How to optimize the below Pyspark code while inserting data into Lake database table in azure synapse
The below code we running the job incrementally every 10 mins in Azure Synapse pipeline(inserting 3k to 4k records for every run) taking more time to insert the records into the table. we are not able to perform partition on the table. how we can insert the records in lessthan 5 mins to target table.