How to cycle through a small set of options within a for loop? - bash

Question

I have a bunch of jobs I need to submit to a job queue. The queue has 8 different machines I can pick from or I can submit to any available server. Sometimes a server may be faulty so I would like to be able to loop through available servers I send my jobs to. A barebones version is below
# jobscript.sh
dir='some/directory/of/files/to/process'
for fn in $(ls $dir); do
submit_job -q server#machine -x python script.py $fn
done
If I don't care what machine the job is sent to I remove the #machine portion so the command is just submit_job -q server -x python script.py $fn.
If I do want to specify the specific machine then I specify which machine by appending a number after machine as server#machine1 then on the next iteration server#machine2 then server#machine2 etc. The output of the script would then look like the following if I only use the first 3 servers
submit_job -q server#machine1 -x python script.py file1
submit_job -q server#machine2 -x python script.py file2
submit_job -q server#machine3 -x python script.py file3
submit_job -q server#machine1 -x python script.py file4
submit_job -q server#machine2 -x python script.py file5
submit_job -q server#machine3 -x python script.py file6
submit_job -q server#machine1 -x python script.py file7
submit_job -q server#machine2 -x python script.py file8
...
The list of available servers is [1, 2, 3, 4, 5, 6, 7, 8] but I would like to additionally specify from the command line a list of servers to ignore so something like
$bash jobscript.sh -skip 1,4,8
which would only cycle through 2, 3, 5, 6, 7 and product the output
submit_job -q server#machine2 -x python script.py file1
submit_job -q server#machine3 -x python script.py file2
submit_job -q server#machine5 -x python script.py file3
submit_job -q server#machine6 -x python script.py file4
submit_job -q server#machine7 -x python script.py file5
submit_job -q server#machine2 -x python script.py file6
submit_job -q server#machine3 -x python script.py file7
submit_job -q server#machine5 -x python script.py file8
submit_job -q server#machine6 -x python script.py file8
...
if flag -skip is not present, just run the command without #machine which will allow the queue to decide where to place the job and the commands look like
submit_job -q server -x python script.py file1
submit_job -q server -x python script.py file2
submit_job -q server -x python script.py file3
submit_job -q server -x python script.py file4
submit_job -q server -x python script.py file5
submit_job -q server -x python script.py file6
submit_job -q server -x python script.py file7
submit_job -q server -x python script.py file8
submit_job -q server -x python script.py file8
...

Answer 1

Something like this should do most of the work for you:
#!/bin/bash
machines=(1 2 3 4 5 6 7 8)
skip_arr=(1 4 8)
declare -a arr
for i in "${machines[#]}"; do
if [[ ! " ${skip_arr[#]} " =~ " $i " ]]; then
arr+=($i)
fi
done
arr_len="${#arr[#]}"
declare -i i=0
for f in $(ls); do
i="i % arr_len"
echo "file is $f, machine is $i"
let i++
done
Right now, I've set it up to go through the current directory, and just echo the values of the machine and filename. Obviously you'll want to change this to actually execute the commands from the right directory.
The last thing you need to do is build up skip_arr from the command line input, and then check if it's empty when you're executing your command.
Hopefully this gets you most of the way there. Let me know if you have any questions about anything I've done here.

Answer 2

Cycle through array of machines
#!/bin/bash
rotate() {
if [[ "$1" = "all" ]]; then
machines=(1 2 3 4 5 6 7 8)
else
machines=($*)
fi
idx=0
max=${#machines[#]}
for ((fn=0; fn<20; fn++)); do
if (( $max > 0 )); then
servernr=${machines[idx]}
((idx=(idx+1) % ${max}))
else
servernr=""
fi
echo "submit -q server${servernr} file${fn}"
done
}
# test
echo "Rotate 0 machines"
rotate
echo "Rotate all machines"
rotate all
echo "Rotate some machines"
rotate 2 5 6