Batch files processing in bash with full processor occupancy - bash

Question

Maybe really simple question, but I don't know where to dig.
I have a list of files (random names), and I want to process them using some command
processing_command $i ${i%.*}.txt
I want to speed up by using all processors. How to make such the script occupy the 10 processors simultaneously (by processing 10 files)? processing_command is not parallel by default. Thank you!

Answer 1

the trivial approach would be to use:
for i in $items
do
processing_command $i ${i%.*}.txt &
done
which will start a new (parallel instance of) processing_command for each $i (the trick is the trailing & which will background the process)
the drawback is, that if you have e.g. 1000 items, then this will start 1000 parallel processes, which (while occupying all 10 cores) will be busy doing context switching rather than doing the actual processing.
if you have as many (or less) items as cores, than this is a good and simple solution.
usually you don't want to start more processes than cores.
a simplistic approach (assuming that all items take about the same time when processing), is to split the the original "items" list into number_of_cores equally long lists. the following is slightly modified version of an example taken from an article in the german linux-magazin:
#!/bin/bash
## number of processors
PMAX=$(ls -1d /sys/devices/system/cpu/cpu[0-9]* | wc -l)
## call processing_command on each argument:
doSequential() {
local i
for i in "$#"; do
processing_command $i ${i%.*}.txt
done
}
## run PMAX parallel processes
doParallel() {
# split the arguments into PMAX equally sized lists
local items item currentProcess=0
for item in "$#"; do
items[$currentProcess]="${items[$currentProcess]} "$item""
shift
let currentProcess=$(( (currentProcess+1)%PMAX ))
done
# run PMAX processes, each with the shorter list of items
currentProcess=0
while [ $currentProcess -lt $PMAX ]; do
[ -n "${items[$currentProcess]}" ] &&
eval doSequential ${items[$currentProcess]} &
currentProcess=$((currentProcess+1))
done
wait
}
doParallel $ITEMS