I am porting an old SSIS package to AWS Glue. In several steps in this package, I take data from one table on one Microsoft SQL Server and copy all of it to an empty table of identical schema on another Microsoft SQL Server. I wish to replicate these steps in AWS Glue. This would be rather easy if I could write code in C# or PowerShell’s dbatools because, like SSIS, they all use SqlBulkCopy
either directly or indirectly. Unfortunately, I am under the following restrictions:
- If I am using a system where I can pick a language and one of the options is Python, then I must use either Python or SQL embedded in Python. I believe that IronPython is not an option.
- My copying must get the same or better performance as SSIS. In other words, I need
SqlBulkCopy
or an equivalent. - AWS Glue must be involved. It is allowed to call other services, e.g. Lambda and Step Functions.
With these restrictions in mind, what options do I have? Do either Python or AWS Glue provide an alternative to .NET’s SqlBulkCopy
?