First things first: I am fairly new to bash scripting, so forgive me any crucial oversights.

On a remote machine, I have the following folder structure:

base_path/

• batch1

  • model1

    • folder_a

    • folder_b

    • …

  • model2

  • …

• batch2
• …

Whenever a batch is finished, I want to copy some folders to my local machine. Within each model folder, there are many subfolders (folder_a, folder_b, etc…). To reduce the workload and time needed for file transfer, I only want to copy some folders (lets say folder_a and folder_b) of every model of the batch in question to local. However, as the remote machine requires two-factor authentication, I am prompted with a authentication request for every scp command that I initiate on local. I can not create ssh keys as one part of the 2 factor authentication of the remote machine changes every few seconds.

I am looking for a solution, where I specify the batch name when executing a shell script that copies folder_a and folder_b of every model in said batch to local. As folder_a and folder_b have identical names, I need to preserve the folder structure of the remote machine. Furthermore, I need to set it up in a way that I am not prompted with an authentication request for every scp command.

My last attempt included a cat <<EOF approach, where I tried to scram all commands that I wanted to execute remotely into a variable and then executing it via one ssh command (I redacted USER and HOST information). Furthermore I need to make directories on local according to the file structure on remote which I only know as soon as I am on the remote machine.

#!/bin/bash

# Remote server details
REMOTE_USER="USER"
REMOTE_HOST="HOST"
REMOTE_DIRECTORY="/smth/smth/smth/$1"
REMOTE_FOLDERS=("folder_a" "folder_b")
LOCAL_USERNAME=$(whoami)
LOCAL_IP=$(hostname -I | awk '{print $1}')
LOCAL_DIRECTORY="$PWD/$1"
mkdir -p "$LOCAL_DIRECTORY"

remote_commands=$(cat <<EOF
cd "$REMOTE_DIRECTORY"
folders=$(ls -d */)
for folder in $folders; do
    for remote_folder in ${REMOTE_FOLDERS[@]}; do
        echo "$folder/$remote_folder"
        scp -r "$folder/$remote_folder" "$LOCAL_USERNAME@$LOCAL_IP:$LOCAL_DIRECTORY/$folder"
    done
done
EOF
)

# Start an SSH session and run commands, capturing the output
output=$(ssh -T "$REMOTE_USER@$REMOTE_HOST" "$remote_commands" 2>/dev/null)

# Display the output
echo "$output"