How do i load and transform large postgreSQL database into databricks without databricks timing out when displaying/writing data
I’ve been tasked with loading data from three postgreSQL tables, combining the data in databricks, and then writing the data to Azure blob storage as .csv files. I’m able to load the data into databricks directly from the PostgreSQL database. First, I load data from the first database and combine it accordingly on the load: