How to batch files in shell script? - bash

I would like to build a shell script to automatically archive object files into a static library, and copy all the headers and .a file to desired directory.
However, as the number of obj files grows, the following mess gets worse:
8 CWD=$(pwd)
9
10 FILE1="SDL_Logger.o"
11 HEADER1="SDL_Logger.h"
12 FILE2="SDL_Initializer.o"
13 HEADER2="SDL_Logger.h"
14 ARC="libsdlhelper.a"
15
16 INCLUDE=~/include
17 LIB=~/lib
18
19 if [ ! -f $FILE1 ];
20 then
21 echo " error: file $FILE1 does not exist. Abort."
22 else
23 if [ ! -f $FILE2 ];
24 then
25 echo " error: file $FILE2 does not exist. Abort."
26 else
27 echo " building archive... "
28 ar rs $ARC $FILE1 $FILE2
29
31 # lib
32 cp $ARC $LIB
34
35 # include
36 cp $HEADER1 $INCLUDE
37 cp $HEADER2 $INCLUDE
39
41 fi
42 fi
So, if I were to group all files into ONE variable like:
FILE="obj1.o obj2.o ... "
How would I check the existence of each file, and copy them(headers)? I can only do this one by one, which will be soon unacceptable.

Here is what I came up with to help you.
The best way to do this is using an array for FILE
short example to implement what you are trying to do:
# !/bin/bash
#define an array of files to look for
FILES=("SDL_Logger.h" "SDL_Initializer.h" "libsdlhelpere.o")
#define your directories
INCLUDE=~/include
LIB=~/lib
#set the amount of entries in our array
SIZE=${#FILES[#]}
# use for loop read all FILES
for (( i=0; i<${SIZE}; i++ ));
do
echo "checking for file: "${FILES[$i]}
if [ ! -f ${FILES[$i]} ];
then
echo " error: file"${FILES[$i]}" does not exist. Abort."
else
echo " building archive... "
fi
done
Here we define FILES as our array
FILES=("SDL_Logger.h" "SDL_Initializer.h" "libsdlhelpere.a")
then we Find the size of files
SIZE=${#FILES[#]}
Then we go through each file and execute a command which is echo and then our if statement:
for (( i=0; i<${SIZE}; i++ ));
do
Here is how we call our item in the array...
${FILES[$i]}
so in what we are doing in plain english is:
echo FILES[1]....
IF FILES[1] doesn't exist then error....
else execute building archive...
echo FILES[2]....
IF FILES[2] doesn't exist then error...
etc...
This will repeat until the SIZE of FILES in met.
Using this method you can also just define the beginning part of the file name such as
"SDL_Initializer" and have your loop add in the .o .h. .a etc... Maybe another array FILETYPES ;)
hope this helped...
Sorry I didn't complete the code for you :)

Here is an answer to the question as asked:
#!/bin/bash
FILES="SDL_Logger.o SDL_Initializer.o"
HEADERS="${FILES//.o/.h}"
ARC="libsdlhelper.a"
INCLUDE=~/include
LIB=~/lib
for f in ${FILES}; do
if [ ! -f "${f}" ]; then
echo "error: file ${f} does not exist. Abort."
exit 1
fi
done
ar rs ${ARC} ${FILES}
cp ${ARC} ${LIB}
cp ${HEADERS} ${INCLUDE}
However, as the comments point out, the right tool for this job is make. For example GNU make. If you plan on developing software on a unix platform, investing time learning make is time well spent.

Related

For loop in script with arrays and script/terminal inconsistency

I want to loop through a list of directories (a subset of the directories in a folder) and do operations with them. However, for some reason it is not working. Here is the code where I am just echoing all of them:
#!/bin/bash
cd images
array=$(ls -d *)
selection=(${array[#]:1:12})
cd ..
for sub in ${selection[#]}
do
echo $sub
mkdir $HOME/Projects/PhD/0_project/fMRI/scans/temp/temp_$sub
done
The ouptut I get for the echo command is:
04
306
307
3
And the folders: temp_3, temp_04, temp_306, temp_307
HOWEVER, if I run each single line in bash in the termiinal (interactive mode, no script) I do get the correct output for the echo command:
306
307
308
309
310
311
312
314
317
318
323
324
And for the mkdir command: temp_306, temp_307... temp_324
Any idea about this strange and inconsistent behaviour? What am I missing?
Thanks for your help.
result of $(ls -d *) is a string.
you are slicing a string not an array.
remove :1:12.
#!/bin/bash
cd images || exit
array=(*/) # result is array of subdirectories with / on end of each
selection=("${array[#]%/}") # remove trailing slashes from that array.
selection_sliced=("${selection[#]:1:12}") # if you want to slice.
cd .. || exit
for sub in "${selection_sliced[#]}"
do
echo "$sub"
mkdir "$HOME/Projects/PhD/0_project/fMRI/scans/temp/temp_$sub"
done

Cannot understand why our function calls return twice?

We have a 15 year (or so) old script we are trying to figure out and document. We have found some errors in it but one specific log file gives us much headache. and I would love some help figuring it out.
First the function that are run with the question:
#=========================================================#
# Define function removeOldBackupFile. #
#=========================================================#
removeOldBackupFile()
{
#set -x
echo "Removing old backups if they exists." >> "${report}"
local RCLOC=0
spaceBefore=$(getAvailableSpace ${backupDirectory})
timesToWait=60 # Wait a maximum of 10 minutes before bailing
cat ${oldDbContainer} | while read fileName
do
echo "Old file exists. Removing ${fileName}." >> "${report}"
removeFileIfExist "${fileName}"
RC=$?
echo "Resultcode for removing old backup is: RC=$RC." >> "${report}"
RCLOC=$(($RC+$RCLOC))
spaceAfter=$(getAvailableSpace ${backupDirectory})
# Wait for the OS to register that the file is removed
cnt=0
while [ $spaceAfter -le $spaceBefore ]; do
cnt=$((cnt+1))
if [ $cnt -gt $timesToWait ]; then
echo "Waited too long for space in ${backupDirectory}" | tee -a "${report}"
RCLOC=$(($RCLOC+1))
return $RCLOC
fi
sleep 10
spaceAfter=$(getAvailableSpace ${backupDirectory})
done
done
return $RCLOC
}
The place where this function is ran looks as follows:
#=========================================================#
# Remove old backupfiles if any exist. #
#=========================================================#
removeOldBackupFile
RC=$?
RCSUM=$(($RC+$RCSUM))
We have identified that the if condition is a bit wrong and the while loops would not work as intended if there are multiple files.
But what bothers us is output from a log file:
...
+ cnt=61
+ '[' 61 -gt 60 ']'
+ echo 'Waited too long for space in /<redacted>/backup'
+ tee -a /tmp/maintenanceBackupMessage.70927
Waited too long for space in /<redacted>/backup
+ RCLOC=1
+ return 1
+ return 0
+ RC=0
+ RCSUM=0
...
As seen in the log output after the inner loop have ran 60 times and ending it returns 1 as expected.. BUT! it also have return 0 after!? Why is it also returning 0?
We are unable to figure out the double returns... Any help appriciated
The first return executes in the subshell started by the pipe cat ${oldDbContainer} | while .... The second return is from return $RCLOC at the end of the function. Get rid of the useless use of cat:
removeOldBackupFile()
{
#set -x
echo "Removing old backups if they exists." >> "${report}"
local RCLOC=0
spaceBefore=$(getAvailableSpace ${backupDirectory})
timesToWait=60 # Wait a maximum of 10 minutes before bailing
while read fileName
do
...
done < ${oldDbContainer}
return $RCLOC
}

A bash script to split a data file into many sub-files as per an index file using dd

I have a large data file that contains many joint files.
It has an separate index file has that file name, start + end byte of each file within the data file.
I'm needing help in creating a bash script to split the large file into it's 1000's of sub files.
Data File : fileafilebfilec etc
Index File:
filename.png<0>3049
folder\filename2.png<3049>6136.
I guess this needs to loop through each line of the index file, then using dd to extract the relevant bytes into a file. Maybe a fiddly part might be the folder structure bracket being windows style rather than linux style.
Any help much appreciated.
while read p; do
q=${p#*<}
startbyte=${q%>*}
endbyte=${q#*>}
filename=${p%<*}
count=$(($endbyte - $startbyte))
toprint="processing $filename startbyte: $startbyte endbyte: $endbyte count: $c$
echo $toprint
done <indexfile
Worked it out :-) FYI:
while read p; do
#sort out variables
q=${p#*<}
startbyte=${q%>*}
endbyte=${q#*>}
filename=${p%<*}
count=$(($endbyte - $startbyte))
#let it know we're working
toprint="processing $filename startbyte: $startbyte endbyte: $endbyte count: $c$
echo $toprint
if [[ $filename == *"/"* ]]; then
echo "have found /"
directory=${filename%/*}
#if no directory exists, create it
if [ ! -d "$directory" ]; then
# Control will enter here if $directory doesn't exist.
echo "directory not found - creating one"
mkdir ~/etg/$directory
fi
fi
dd skip=$startbyte count=$count if=~/etg/largefile of=~/etg/$filename bs=1
done <indexfile

linux for loop two variables each time

I have several files in a directory and I want to run some linux packages on these files by every two of them, like ERR1045141_1 with ERR1045141_2 and ERR1045144_1 with ERR1045144_2 and so on. So I write a for loop for this but it is not working.
files:
ERR1045141_1.fastq.gz
ERR1045141_2.fastq.gz
ERR1045144_1.fastq.gz
ERR1045144_2.fastq.gz
ERR1045145_1.fastq.gz
ERR1045145_2.fastq.gz
ERR1045146_1.fastq.gz
ERR1045146_2.fastq.gz
ERR1045148_1.fastq.gz
ERR1045148_2.fastq.gz
ERR1045149_1.fastq.gz
ERR1045149_2.fastq.gz
ERR1045151_1.fastq.gz
ERR1045151_2.fastq.gz
ERR1045152_1.fastq.gz
ERR1045152_2.fastq.gz
ERR1045154_1.fastq.gz
ERR1045154_2.fastq.gz
codes:
files=ls
for (( i=0; i<${#files[#]} ; i+=2 )) ; do
echo "${files[i]}" "${files[i+1]}"
done
It did not work and I am not sure is the files=ls has something wrong.Or any better way to do it.please advise.
Try the following if you are sure about the existence of the second file:
for file1 in ERR*_1*
do
file2=`echo $file1 | sed 's/_1/_2/g'`
echo $file1 $file2
done
No, what you really want to do is to process all the 1 files, performing some action on it and its associated 2 file.
You can do that with something as simple as the for loop in this complete test program:
#!/usr/bin/env bash
doSomethingWith() {
echo "[$1] [$2]"
}
touch 'xERR1045141_1.fastq.gz' 'xERR1045141_2.fastq.gz'
touch 'xERR1045144_1.fastq.gz' 'xERR1045144_2.fastq.gz'
touch 'xERR1045145_1.fastq.gz' 'xERR1045145_2.fastq.gz'
touch 'xERR1045146_1.fastq.gz' 'xERR1045146_2.fastq.gz'
touch 'xERR1045148_1.fastq.gz' 'xERR1045148_2.fastq.gz'
touch 'xERR1045149_1.fastq.gz' 'xERR1045149_2.fastq.gz'
touch 'xERR1045151_1.fastq.gz' 'xERR1045151_2.fastq.gz'
touch 'xERR1045152_1.fastq.gz' 'xERR1045152_2.fastq.gz'
touch 'xERR1045154_1.fastq.gz' 'xERR1045154_2.fastq.gz'
touch 'xERR 45154_1.fastq.gz' 'xERR 45154_2.fastq.gz'
for file1 in xERR*_1.fastq.gz ; do
file2="${file1/_1/_2}"
doSomethingWith "${file1}" "${file2}"
done
rm -rf xERR*.fastq.gz
This program outputs:
[xERR1045141_1.fastq.gz] [xERR1045141_2.fastq.gz]
[xERR1045144_1.fastq.gz] [xERR1045144_2.fastq.gz]
[xERR1045145_1.fastq.gz] [xERR1045145_2.fastq.gz]
[xERR1045146_1.fastq.gz] [xERR1045146_2.fastq.gz]
[xERR1045148_1.fastq.gz] [xERR1045148_2.fastq.gz]
[xERR1045149_1.fastq.gz] [xERR1045149_2.fastq.gz]
[xERR1045151_1.fastq.gz] [xERR1045151_2.fastq.gz]
[xERR1045152_1.fastq.gz] [xERR1045152_2.fastq.gz]
[xERR1045154_1.fastq.gz] [xERR1045154_2.fastq.gz]
[xERR 45154_1.fastq.gz] [xERR 45154_2.fastq.gz]
to show that the names are being handled correctly.
Note that I've named the files xERR* so as not to clash with your own files. You should adjust the loop to handle your own files once you're satisfied it will work okay.
And, just as an aside, if you don't want to do anything except for those cases where both files exist, you can simply replace the "action" line with something like:
[[ -f "${file2}" ]] && doSomethingWith "${file1}" "${file2}"
This will bypass those where the 2 file is not a regular file.

batch processing : File name comparison error

I have written a program (Cifti_subject_fmri) which compares whether file name matches in two folders and essentially executes a set of instructions
#!/bin/bash -- fix_mni_paths
source activate ciftify_v1.0.0
export SUBJECTS_DIR=/scratch/m/mchakrav/dev/functional_data
export HCP_DATA=/scratch/m/mchakrav/dev/tCDS_ciftify
## make the $SUBJECTS_DIR if it does not already exist
mkdir -p ${HCP_DATA}
SUBJECTS=`cd $SUBJECTS_DIR; ls -1d *` ## list of my subjects
HCP=`cd $HCP_DATA; ls -1d *` ## List of HCP Subjects
cd $HCP_DATA
## submit the files to the queue
for i in $SUBJECTS;do
for j in $HCP ; do
if [[ $i == $j ]];then
parallel "echo ciftify_subject_fmri $i/filtered_func_data.nii.gz $j fMRI " ::: $SUBJECTS |qbatch --walltime '05:00:00' --ppj 8 -c 4 -j 4 -N ciftify_subject_fmri -
fi
done
done
When i run this code in the cluster i am getting an error which says
./Cifti_subject_fmri: [[AS1: command not found
The query ciftify_subject_fmri is part of toolbox ciftify, for it to execute it requires following instructions
ciftify_subject_fmri <func.nii.gz> <Subject> <NameOffMRI>
I have 33 subjects [AS1 -AS33] each with its own func.nii.gz files located SUBJECTS directory,the results need to be populated in HCP directory, fMRI is name of file format .
Could some one kindly let me know why i am getting an error in loop

Resources