I have a bash script on Linux to extract NetCDF data for each grid into a .txt file. The script runs correctly without any errors. The code in the script (extract_global.sh) is as follows:

#!/bin/bash

# Define input file and log file
input_file="/path/canesm5_r1i1p1f1_w5e5_ssp126_tas_global_daily_2015_2100.nc"
#log_file="log.txt"

# Remove existing log file if it exists
#rm -f $log_file

# Function to extract data for a specific latitude
extract_latitude() {
    lat=$1
    lon_index=1
    max_jobs=20
    job_count=0

    # Loop through longitudes from -180 to 180 in 0.5 degree increments
    for lon in $(seq -180 0.5 179.5); do
        # Define the output file name based on the current latitude and longitude index
        output_file="${lat}_${lon_index}.txt"
        
        # Extract data for the grid point at the current latitude and longitude, save only values, and remove the header
        (cdo -outputtab,value -remapnn,lon=${lon}_lat=${lat} $input_file | sed '1d' > $output_file) #&>> $log_file &
        
        # Increment the longitude index and job count
        lon_index=$((lon_index + 1))
        job_count=$((job_count + 1))

        # Check if the max number of jobs has been reached
        if [ $job_count -ge $max_jobs ]; then
            wait  # Wait for all background jobs to complete
            job_count=0  # Reset job count
        fi
    done

    # Wait for any remaining background processes to finish
    wait
}

# Loop through latitudes from 90N to -90S in 0.5 degree increments
for lat in $(seq 90 -0.5 -90); do
    echo "Extracting data for latitude $lat"
    extract_latitude $lat
done

echo "Data extraction completed. Check $log_file for details."

By default, all jobs on our Linux server run on the main CPU. I want to change the CPU used for this job by changing to different queue.
The main CPU is used by several members, so running heavy jobs can slow down the server. Therefore, in the code above, I set the maximum number of jobs to 20 and used a loop. However, this makes the process very long and inefficient.

I tried to solve this matter, by created a go.bat file to set up the queue and other configuration. The contents of the go.bat file are as follows:

#!/bin/sh
#$ -S /bin/sh
#$ -q ib.q@node02   # Specify the queue
#$ -N SSP_kzm       # Define job name
#$ -j y             # Standard output and error message will be written in the same file
#$ -o /path/  # Specify the output file
#$ -e /path/log.txt  # Specify the error file
#$ -M [email protected]
#$ -m be            # Send email at the beginning and end of the job

# Load necessary modules
module load cdo

# Or if cdo is in a custom directory, add it to the PATH
export PATH=/usr/local/bin/cdo:$PATH

cd /path/SSP
./extract_global.sh >& log.txt

Finally, I ran the job with the settings in the run.sh file, as shown below:

#!/bin/bash

# Submit the conversion program and wait for it to complete
qsub go.bat
echo "Waiting for conversion computation"
date

# Wait for the job to finish
while qstat | grep -q kzm; do
  sleep 5
done

date
echo "Job completed"

I ran by using

nohup ./run.sh >& log.txt &

The results I got:

• The program successfully generated all of the .txt files according to the grid area of the NetCDF file, but the .txt files were empty.
• In the log.txt file, it says: error: ./extract_global.sh: line 27: cdo: command not found. However, CDO is installed on our server, and the ./extract_global.sh script runs fine when executed directly.
• I’m not sure if the method I’m using is correct. I’ve looked for information and tried similar solutions, but I haven’t been able to fix it yet.

The main point is that I want to run the ./extract_global.sh script on a queue named ib.q@node02. If possible, I would like to use -pe to allocate all 20 CPUs available to speed up the extraction process.

If someone is able to help me, I would be very grateful.

Thank you.

I tried running ./extract_global.sh and go.bat using run.sh, but it produced the errors I described in the problem details.

./extract_global.sh: line 27: cdo: command not found ./extract_global.sh: line 27: cdo: command not found ./extract_global.sh: line 27: cdo: command not found ./extract_global.sh: line 27: cdo: command not found