I have a bash script that downloads S3 objects into a TEMP_FILE and then uses jq to determine if that object has malformed JSON or not. This actually works pretty well:

#!/bin/bash

BUCKET_NAME="<REDACTED>" # assume this string is correct
COMMON_PREFIX="<REDACTED>" # assume this string is correct
AWS_PROFILE="<REDACTED>" # assume this string is correct
OUTPUT_FILE="./malformed_json_$(date +%Y%m%dT%H%M).txt"
TEMP_FILE="$(mktemp)"

# Fetch the list of objects and continuation token
args=(--bucket "$BUCKET_NAME" --prefix "$COMMON_PREFIX" --output json --profile "$AWS_PROFILE")
response=$(aws s3api list-objects-v2 "${args[@]}")
echo "Received responses for metadata objects!"

# Parse the response to get object keys
keys=$(echo "$response" | jq -r '.Contents[] | select(.Key | endswith("metadata.json")) | .Key')

# Write object key to file if object is malformed
for key in ${keys}; do
  printf "Processing: %sn" "$key"
  aws s3api get-object --bucket "$BUCKET_NAME" --key "$key" --profile "$AWS_PROFILE" "$TEMP_FILE" >/dev/null 2>&1
  if jq -e . "$TEMP_FILE" >/dev/null 2>&1; then
    :
  else
    echo " --- INVALID JSON FOUND --- "
    echo "$key" >>"$OUTPUT_FILE"
  fi
done
echo "Finished processing metadata objects!"

rm "$TEMP_FILE"
echo "Cleaned up temporary file!"

…but the issue I’m having is that the script crashes once all keys are exhausted with the following errors:

/malformed_json.sh: line 32: unexpected EOF while looking for matching `"'
./malformed_json.sh: line 33: syntax error: unexpected end of file

This is fine since I get the answers I need (it properly identifies the malformed JSON), but I don’t like the bug that causes a crash rather than exiting successfully.

When I comment out the aws s3api command, everything “works” fine (i.e., it doesn’t do any processing since TEMP_FILE stays empty, but it does exit gracefully). I’m wondering why I’m getting this error…there isn’t any obvious syntax error (at least when I check with shellcheck and tired eyes).

(I’d like to get better at writing these sort of scripts, so any advice is well-received!)

Update: When I run the script with the if ... fi block commented out, but leaving in the aws s3api get-object ..., things exit gracefully (no crash, I’m able to make the API calls fine). So, given that and that the script does’t crash when the get-object call is omitted, it seems the issue only arises when both are left in!

Running the above with set -x doesn’t give me back any additional info — same errors as mentioned above.