I am trying to perform a “copy” operation from an external Amazon RDS into my own Azure SQL database. However, when communicating with the Amazon RDS admin team they expect this connection to resolve to a singular IP of which they can easily whitelist.
Realistically, I just want to be able to:
1. Connect to the Amazon RDS from Azure
2. Query some data from this database
3. Store in my Azure database
4. Report on the data in PowerBI
Ideally, I want to perform this at the lowest cost possible as the amount of data I am reporting on is quite small (2000 rows per day to be pulled into my Azure database for reporting)
I thought this would be possible by using a combination of the following Azure elements:
-
VNet
-
NAT Gateway
-
Public IP Address
-
Integration Runtime in Azure Datafactory associated with the linked service (to the Amazon RDS)
I feel like I am very close, but I don’t understand how to make my Azure Integration run time explicitly reference my VNet/Subnet/Nat Gateway/Public IP address.
When creating the Integration Runtime in ADF I would expect this UI element to allow me to directly reference my VNet and other elements listed above, but it does not seem to have any other options. So I don’t actually understand what these options do when set to enabled
I found this medium article which I believe directly references my problem, but is admittedly a few steps outside of my realm of understanding and is seemingly overengineered for the scale of the project I am trying to complete.
Any help is welcome on this. Is what I’m trying to do even possible? Does it make sense to wrap the outbound ADF connection to the RDS for the purposes of whitelisting? I am struggling with the RDS team to get them to budge on any other approach.
Please let me know if I am way off base with the elements I am using or if there is an easier way to do this (connect to an external database to copy data from it and bring into azure).
As my first time working with Azure infrastructure, I am dumbfounded by how this seemingly easy task has become more and more complex. I just want data from an external cloud db into my azure db at a low cost.