bash: manager processes run - bash

Question

I have multiple files in $tmpdir/$i.dirlist with entries command rsync.
Each file have (depending on the amount) 10 sometimes 50 and even 150 entries of rsync.
I'm wondering now how to manage it by FOR or WHILE loop with IF sequence to run from each files ($tmpdir/$i.dirlist - if we have have 100 files) only 2 entries and wait for complet some processes and if total of all running process of rsync are less than 200 processes - launched new entries, maintaining a fixed number of processes defined in the parameter. In this case 200
Any idea? how to do it?
Edit:
about rsync entry.
In each file $tmpdir/*.dirlist is (in this example 200) entries with directory
path like:
==> /tmp/rsync.23611/0.dirlist <==
system/root/etc/ssl
system/root/etc/dbus-1
system/root/etc/lirc
system/root/etc/sysctl.d
==> /tmp/rsync.23611/1.dirlist <==
system/root/etc/binfmt.d
system/root/etc/cit
system/root/etc/gdb
==> /tmp/rsync.23611/2.dirlist <==
system/root/usr/local
system/root/usr/bin
system/root/usr/lib
now to run it i use simply for
for i in $(seq 1 $rsyncs); do
while read r; do
rsync $rsyncopts backup#$host:$remotepath/$ri $r 2>&1 |
tee $tmpdir/$i.dirlist.log ;
done < $tmpdir/$i.dirlist &
done

Answer 1

with an example of use
for ARG in $*; do
command $ARG &
NPROC=$(($NPROC+1))
if [ "$NPROC" -ge 4 ]; then
wait
NPROC=0
fi
done

Answer 2

Assuming the maximum value of $i is 100, with your code above you are still below the maximum you want to allow of 200 processes.
So a solution would be to run twice as much processes. I suggest you to divide your main loop for i in $(seq 1 $rsyncs); do ... in two loops running concurrently, introduced by resp. for i in $(seq 1 2 $rsyncs); do ... for the odd values of $i, and for i in $(seq 2 2 $rsyncs); do ... for the even values of $i.
for i in $(seq 1 2 $rsyncs); do # i = 1 3 5 ...
while read r; do
rsync $rsyncopts backup#$host:$remotepath/$ri $r 2>&1 |
tee $tmpdir/$i.dirlist.log ;
done < $tmpdir/$i.dirlist &
done & # added an ampersand here
for i in $(seq 2 2 $rsyncs); do # i = 2 4 6 ...
while read r; do
rsync $rsyncopts backup#$host:$remotepath/$ri $r 2>&1 |
tee $tmpdir/$i.dirlist.log ;
done < $tmpdir/$i.dirlist &
done
Edit: Since my approach above doesn't convince you, let us try something completely different. First, create a list of all the processes you want to run and store these in an array:
processes=() # create an empty bash array
for i in $(sed 1 $rsyncs); do
while read r; do
# add the full rsync command line to the array
processes+=("rsync $rsyncopts backup#$host:$remotepath/$ri $r 2>&1 | tee $tmpdir/$i.dirlist.log");
done < $tmpdir/$i.dirlist
done
Once you have that array, launch say 200 processes, and then enter a loop to wait a process to finish and launch the next one:
for ((j=0;j<200;j++)); do
$processes[$j]& # launch processes in background
done
while [ ! -z "$processes[$j]" ] ; do
wait # wait one process finishes
$processes[((j++))]& # launch one more process
done
Please try this and tell us.