How to iterate over files in many folders - bash

Question

I have 15 folders and each folder contained a *.gz file. I would like to use that file for one of the package to do some filtering.
For this I would like to write something that can open that folder and read the that specific file and do the actions as mentioned and than save the results in the same folder with different extension.
What I did is(PBS Script):
#!/bin/bash
#PBS -N Trimmomatics_filtering
#PBS -l nodes=1:ppn=8
#PBS -l walltime=04:00:00
#PBS -l vmem=23gb
#PBS -q ext_chem_guest
# Go to the Trimmomatics directory
cd /home/tb44227/bioinfo_packages/Trimmomatic/Trimmomatic-0.36
# Java module load
module load java/1.8.0-162
# Input File (I have a list of 15 folders and each contained fastq.gz file)
**inputFile= for f in /home/tb44227/nobackup/small_RNAseq_260917/support.igatech.it/sequences-export/536-RNA-seq_Disco_TuDO/delivery_25092017/754_{1..15}/*fastq.gz; $f**
# Start the code to filter the file and save the results in the same folder where the input file is
java -jar trimmomatic-0.36.jar SE -threads ${PBS_NUM_PPN} -phred33 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:17 $inputFile $outputFile
# Output File
outputFile=$inputFile{.TRIMMIMG}
My question is How could I define $inputFile and $outputfile so that it can read for all the 15 files.
Thanks

Answer 1

If your application does only process a single input file at a time, you have two options:
Process all files in one single job
Process each file in a different job
From the user's perspective you are usually more interested in the second option, as multiple jobs may run simultaneously if there are resources available. However, this depends on the number of files you need to process and your system usage policy, as sending too many jobs in a short amount of time can cause problems in the job scheudler.
The first option is, more or less, what you already got. You can use find program and a simple bash loop. You basically store find output into a variable, and then iterate over it, like in this example:
#!/bin/bash
# PBS job parameters
module load java
root_dir=/home/tb44227/nobackup/small_RNAseq_260917/support.igatech.it/sequences-export/536-RNA-seq_Disco_TuDO/delivery_25092017
# Get all files to be processed
files=$(find $root_dir -type f -name "*fastq.gz")
for inputfile in $files; do
outputfile="$inputFile{.TRIMMIMG}"
# Process one file at a time
java -jar ... $inputfile $outputfile
done
Then, you just submit your job script, which will generate a single job.
$ qsub myjobscript.sh
The second option is more powerful, but requires you to change the jobscript for each file. Most job managers let you pass the job script by standard input. This is really helpful because it avoids us to generate intermediate files, which pollute your directories.
#!/bin/bash
function submit_job() {
# Submit job. Jobscript passed through standard input using a HEREDOC.
# Must define $inputfile and $outputfile before calling the function.
qsub - <<- EOF
# PBS job parameters
module load java
# Process a single file only
java -jar ... $inputfile $outputfile
EOF
}
root_dir=/home/tb44227/nobackup/small_RNAseq_260917/support.igatech.it/sequences-export/536-RNA-seq_Disco_TuDO/delivery_25092017
# Get all files to be processed
files=$(find $root_dir -type f -name "*fastq.gz")
for inputfile in $files; do
outputfile="$inputFile{.TRIMMIMG}"
submit_job
done
Since you are calling qsub inside the script, you just need to call the script itself, like any regular shell script file.
$ bash multijobscript.sh