How to add grouping mechanism inside for loop in bash - bash

Question

I have a for loop that loops through a list of files, and inside the for loop a script is called, that takes this file name as input.
Something like
for file in $(cat list_of_files) ; do
script $file
done
the file list_of_files has files like
file1
file2
file3
...
so with each iteration, one file is processed.
I have to design something like, loop through all the files, group them into groups of 3 , so that in one loop, script will be called 3 times, and not one by one,and then again the other 3 will be called in second loop iteration and so on
something like,
for file in $(cat list_of_files) ; do
# do somekind of grouping here
call one more loop to run the sript.sh 3 times, so something like
for i=1 to 3 and then next iteration from 4 to 6 and so on..
script.sh $file1
script.sh $file2
script.sh $file3
done
I am struggling currently on how to get this looping done and i am stuck here and could not think of efficient way here.

Answer 1

Change for ... in to while read
for file in $(cat list_of_files)
This style of loop is subtly dangerous and/or incorrect. It won't work right on file names with spaces, asterisks, or other special characters. As a general rule, avoid for x in $(...) loops. For more details, see:
Bash Pitfalls: for f in $(ls *.mp3).
A safer alternative is to use while read along with process substitution, like so:
while IFS= read -r file; do
...
done < <(cat list_of_files)
It's ugly, I'll admit, but it will handle special characters safely. It split apart file names with spaces and it won't expand * globs. For more details on what this is doing, see:
Unix.SE: Understanding “IFS= read -r line”.
You can then remove the Useless Use of Cat and use a simple redirection instead:
while IFS= read -r file; do
...
done < list_of_files
Read 3 at a time
So far these changes haven't answered your core question, how to group files 3 at a time. The switch to read has actually served a second purpose. It makes grouping easy. The trick is to call read multiple times per iteration. This is an easy change with while read; it's not so easy with for ... in.
Here's what that looks like:
while IFS= read -r file1 &&
IFS= read -r file2 &&
IFS= read -r file3
do
script.sh "$file1"
script.sh "$file2"
script.sh "$file3"
done < list_of_files
This calls read three times, and once all three succeed it proceeds to the loop body.
It will work great if you always have a multiple of 3 items to process. If not, it will mess up at the end and skip the last file or two. If that's an issue we can update it to try to handle that case.
while IFS= read -r file1; do
IFS= read -r file2
IFS= read -r file3
script.sh "$file1"
[[ -n $file2 ]] && script.sh "$file2"
[[ -n $file3 ]] && script.sh "$file3"
done < list_of_files
Run the scripts in parallel
If I understand your question right, you also want to run the scripts at the same time rather than sequentially, one after the other. If so, the way to do that is to append &, which will cause them to run in the background. Then call wait to block until they have all finished before proceeding.
while IFS= read -r file1; do
IFS= read -r file2
IFS= read -r file3
script.sh "$file1" &
[[ -n $file2 ]] && script.sh "$file2" &
[[ -n $file3 ]] && script.sh "$file3" &
wait
done < list_of_files

Answer 2

How about
xargs -d $'\n' -L 1 -P 3 script.sh <list_of_files
-P 3 runs 3 processes in parallel. Each of those gets the input of one line (due to -L 1), and the -d options ensures that spaces in an input line are not considered separate arguments.

Answer 3

You can use bash arrays to store the filenames until you get 3 of them:
#!/bin/bash
files=()
while IFS= read -r f; do
files+=( "$f" )
(( ${#files[#]} < 3 )) && continue
script.sh "${files[0]}"
script.sh "${files[1]}"
script.sh "${files[2]}"
files=()
done < list_of_files
However, I think that John Kugelman's answer is simpler, then better: it uses less bash-specific features, then it can be more easily converted to a POSIX version.

Answer 4

you should not mix script languages if you don't absolutely have to
you can start with that
from os import listdir
from os.path import isfile, join
PATH_FILES = "/yourfolder"
def yourFunction(file_name):
file_path = PATH_FILES + "/" + file_name
print(file_path) #or do something else
print(file_path) #or do something else
print(file_path) #or do something else
file_names = [f for f in listdir(PATH_FILES) if isfile(join(PATH_FILES, f))]
for file_name in file_names:
yourFunction(file_name)

Answer 5

If mapfile aka readarray is available/acceptable. (bash4+ is required)
Assuming script.sh can accept multiple input.
#!/usr/bin/env bash
while mapfile -tn3 files && (( ${#files[*]} == 3 )); do
script.sh "${files[#]}"
done < list_of_files
otherwise loop through the array named files
#!/usr/bin/env bash
while mapfile -tn3 files && (( ${#files[*]} == 3 )); do
for file in "${files[#]}"; do
script.sh "$file"
done
done < list_of_files
The body after the do will run/execute if there are always 3 lines if there is not enough lines to satisfy 3 lines until the end of the file, just remove the
&& (( ${#files[*]} == 3 ))
from the script.
or do it manually one-by-one, but it should have 3 lines to be processed (the file) until the end.
#!/usr/bin/env bash
while mapfile -tn3 files && (( ${#files[*]} == 3 )); do
script.sh "${file[0]}"
script.sh "${file[1]}"
script.sh "${file[2]}"
done < list_of_files