I have a file of s3 paths, sources.lst
. Contents:
[
"s3://bucket-name/path/to/file0.tif",
"s3://bucket-name/path/to/file1.tif",
]
I am attempting to iterate over these s3 paths & download the files. I am trying to using jq
and a bash while loop. My download.sh
:
#!/bin/bash
set -o errexit
while read -r source
do
echo "Attempting to execute:"
echo "aws s3 cp "${source}" ."
aws s3 cp "${source}" .
done < <(cat sources.lst | jq '.[]')
Executing bash download.sh
gives me:
Attempting to execute:
aws s3 cp "s3://bucket-name/path/to/file0.tif" .
usage: aws s3 cp <LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>
Error: Invalid argument type
Which is strange, since if I copy the line causing the error, ie
aws s3 cp "s3://bucket-name/path/to/file0.tif" .
and execute it in the terminal I run bash download.sh
, the download succeeds.
How do I download the files specified in sources.lst
using bash?
I am using Mac, bash --version
: GNU bash, version 5.2.26(1)-release (aarch64-apple-darwin22.6.0)
, aws --version
: aws-cli/2.13.1 Python/3.11.4 Darwin/23.4.0 exe/x86_64 prompt/off