There is a REST api that has data I need to use for lookup in my app. To do that, I’m saving that data into the Azure SQL database for the my app, mainly for three reasons: I can use that data on JOINs with my data; if we lose connectivity to the REST API, I still have the data; rate limits on the REST API.

My question is what would be the best way to do that on a regular basis. For most of the data, I’m thinking once per hour would good enough.

Some data I could use being real time, but rate limits on the API won’t allow it. That data will also need to have deletions accounted for too (whereas the hourly data can just be upserted without need to delete anything)

I’m currently using a Go cli utility to save the data using gorm and recently added batching to my cli which has improved performance a lot, but I can’t help think there must be a better more efficient way to do it.

A quick search showed up things like sqlbulkcopy, ADF and Spark.

Any thoughts greatly appreciated. Many thanks.