We need to copy the contents of a bucket to a new account in a different AWS region.
The bucket contains ±200K objects, most of which are archived (Glacier), and a large portion of them are quite big (>10GB).
At first we tried the following:
- Batch script to restore all the objects.
- Configure a Lambda trigger to handle restore completion per object.
- Execute the copy method with boto3 on the Lambda function.
This works well for smaller objects (up to ~2GB), but larger objects take longer than the maximum allowed Lambda duration (15min).
To complete the rest of them (>10K objects) we must use some VM, however VMs are forced to be in a VPC and for some reason boto3 can’t handle cross-region copy commands from within a VPC:
An error occurred (AccessDenied) when calling the UploadPartCopy operation: VPC endpoints do not support cross-region requests
So we resorted to download and upload each object. This takes too long for the amount of objects we need to copy.
Can anyone offer a better solution?
NOTE #1: AWS has something called Batch Jobs which may be relevant, but it was a bit difficult to configure and for some reason requires both source and destination buckets to enable versioning (which is disable in this case).
NOTE #2: We have a role on the destination account which has access to the source account and S3 requests are called with the requester paying for them.
7
Eventually we did use DataSync. One caveat is that you need to manually take care of restoring the objects before running the DataSync task.
The AWS tutorial on how do that is here.