I have millions of files in an S3 bucket located in one AWS region. I need to copy about 400,000 of these files to another S3 bucket located in a different region, but with different file names and paths. I have a CSV file that maps the source keys to the destination keys, specifying where each file should be copied and what its new name should be.

Currently, I’m using the AWS CLI command aws s3 cp, but it’s very slow and seems like it could take over a week to complete.

I’ve also experimented with S3 Batch Operations using a custom CSV manifest for a small test set of 5 files, but it appears that Batch Operations don’t support renaming files or changing their paths during the copy process.

Given that the buckets are in different regions and the files need to be renamed and relocated during the copy, is there a more efficient way to handle this task? Any advice on tools, scripts, or AWS services that could help speed this up would be greatly appreciated.