list all files in an S3 bucket with 1 million files into a TXT file
I tried using s3cmd ls s3://bucket01/, but it’s extremely slow and even unresponsive. Even if it succeeds, the speed is extremely slow, and listing 1 million files could take a month.
1
AWS CLI version 2 supports multithreading. The parameter max_concurrent_requests has a default value of 10, but you can increase this value to speed up the list operations, provided that your CPU can handle the increased load. The speed improvement is indeed significant!
New contributor
Dyck Anthony is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.