How to find the combination of files in unix directory? - shell

I have a set of files at a directory. I need to exit out of my script if i don't find the pairs of files at a given time.
Let's say i have these 3 files at directory $SRC_DIR
file 1: apple_iphone_file.zip
file 2: apple_ipad_file.zip
file 3: apple_mac_file.zip
If these 3 set of files are present i am doing some post processing.
There can be multiple pairs like 2,3, OR N set of these 3 files (file1,file2,file3).
I should exit the script if the same set are not present for all 3 files.
I am planing to count file 1 and if it gives me 2 , i will check if the other two files (file 2 and file 3) also gives me same count , else i will exit.
Do you think , we can do in any other way too?
Any input is highly appreciated.
Code Tried
#!/usr/bin/ksh
file1_count=$(ls ${SRC_DIR}/apple_iphone_file.zip | wc -l)
file2_count=$(ls ${SRC_DIR}/apple_ipad_file.zip | wc -l)
file3_count=$(ls ${SRC_DIR}/apple_mac_file.zip | wc -l)
if [ "$file1_count" == "$file2_count" -a "$file2_count" == "$file3_count" ]; then
echo "Files count match"
else
echo "Files count don't match"
exit 1
fi
This is giving me the results. However, if the files aren't present (none of them) it still shows me "Count Match".

Two scripts, check and pre-process. In check, if the variables corresponding with all three files equal 1, run pre-process. The default action (which runs next if that fails) is to exit.
Pre-process finds all files in the directory, puts their names in an input stack, and then uses each name as input for the main function. The code between the two ed lines is an example; replace it with your own. After that, it exits.
check.sh:-
#!/bin/sh
find apple_iphone_file.zip && iphone=1
find apple_ipad_file.zip && ipad=1
find apple_mac_file.zip && mac=1
[ "${iphone}" -eq 1 ] && [ "${ipad}" -eq 1 ] && \
[ "${mac}" -eq 1 ] && ./pre-process.sh
exit 0
pre-process.sh:-
#!/bin/sh
next() {
[ -s stack ] && main
end
}
main() {
line=$(ed -s stack < edprint+.txt)
echo "${line}" | tr '[a-z]' '[A-Z]'
ed -s stack < edpop+.txt
next
}
end() {
rm -v ./stack
rm -v ./edprint+.txt
rm -v ./edpop+.txt
exit 0
}
find . -type -f > stack
cat >> edprint+.txt << EOF
1
q
EOF
cat >> edpop+.txt << EOF
1d
wq
EOF

There are so many possibilities.
One option:
my_counter=""
[ -f "file_name_1" ] && my_counter="x$my_counter"
[ -f "file_name_2" ] && my_counter="x$my_counter"
[ -f "file_name_3" ] && my_counter="x$my_counter"
if [ "${#my_counter}" -lt 2 ]; then
echo "Error"
else
echo "doing stuffs"
fi
You can easily add check for file, change lower and upper limit, and ev. doing more tests (in similar way) in series. Note: for shell scripts, I tend to copy paste, because the command are easy (low probability of refactoring), and often I find the need to add extra tests on specific cases.

Related

File count in a folder not showing accurate

I am writing a shell script to check two things at one time. The first condition is to check for the existence of a specific file and the second condition is to confirm that there is only one file in that directory.
I am using the following code:
conf_file=ls -1 /opt/files/conf.json 2>/dev/null | wc -l
total_file=ls -1 /opt/files/* 2>/dev/null| wc -l
if [ $conf_file -eq 1 ] && [ $total_file -eq 1 ]
then
echo "done"
else
echo "Not Done"
fi
It is returning the following error
0
0
./ifexist.sh: 4: [: -eq: unexpected operator
Not Done
I am probably doing a very silly mistake. Can anyone help me a little bit?
One of the reasons you should normally not parse ls is that you can get strange results when you have files with newlines. In your case that won't be an issue, because any file different from json.conf should make the test fail. However you should make the code counting the files be future-proof. You can use find for this.
Your code can be changed into
jsonfile="/opt/files/conf.json"
countfiles=$(find /opt/files -maxdepth 1 -type f -exec printf '.\n' \; | wc -l)
if [[ -f "${jsonfile}" ]] && (( "${countfiles}" == 1)); then
echo "Done"
else
echo "Not Done"
fi
When you say this:
conf_file=ls -1 /opt/files/conf.json 2>/dev/null | wc -l
That assigns the value "ls" to the variable conf_file, and then tries to run a command called "-1" and pipe the result to wc If you want to run a pipe sequence, you have to enclose it in $( ):
conf_file=$(ls -1 /opt/files/conf.json 2./dev/null | wc -l)
Next, when combining clauses in the test command ([), do it inside the command:
if [ $conf_file -eq 1 -a $total_file -eq 1 ]
However, there are better ways to do this. You can check if a file exists with "-f", and you can just check whether the output of ls matches what you expect, without creating variables or running other commands:
if [ -f /opt/files/conf.json -a "$(ls /opt/files/conf.*)" -eq "/opt/files/conf.json" ]
However, it is not a friendly practice to prohibit other files. In many cases, people might want to leave backup or test copies (conf.json.bak or conf.json.test), and there's no reason for you to block that.

Check if multiple files are empty in single if statement

awkOut1="awkOut1.csv"
awkOut2="awkOut2.csv"
if [[ "$(-s $awkOut1)" || "$(-s $awkOut2)" ]]
The above 'if' check in shell script gives me below error:
-bash: -s: command not found
Suggestions anyone?
If you just have 2 files I would do
if [[ -e "$awkOut1" && ! -s "$awkOut1" ]] &&
[[ -e "$awkOut2" && ! -s "$awkOut2" ]]
then
echo both files exist and are empty
fi
Since [[ is a command, you can chain the exit statuses together with && to ensure they are all true. Also, within [[ (but not [), you can use && to chain tests together.
Note that -s tests for True if file exists and is not empty. so I'm explicitly adding the -e tests so that -s only checks if the file is not empty.
If you have more than 2:
files=( awkOut1.csv awkOut2.csv ... )
sum=$( stat -c '%s' "${files[#]}" | awk '{sum += $1} END {print sum}' )
if (( sum == 0 )); then
echo all the files are empty
fi
This one does not test for existence of the files.
You can use basic Bourne shell syntax and the test command (a single left bracket) to find out if either file is non-empty:
if [ -s "$awkOut1" -o -s "$awkOut2" ]; then
echo "One of the files is non-empty."
fi
When using single brackets, the -o means "or", so this expression is checking to see if awkOut1 or awkOut2 is non-empty.
If you have a whole directory full of files and you want to find out if any of them is empty, you could do something like this (again with basic Bourne syntax and standard utilities):
find . -empty | grep -q . && echo "some are empty" || echo "no file is empty"
In this line, find will print any files in the current directory (and recursively in any subdirectories) that are empty; grep will turn that into an exit status; and then you can take action based on success or failure to find empties. In an if statement, it would look like this:
if find . -empty | grep -q .; then
echo "some are empty"
else
echo "no file is empty"
fi
Here is one for GNU awk and filefuncs extension. It checks all parameter given files and exits once the first one is empty:
$ touch foo
$ awk '
#load "filefuncs" # enable
END {
for(i=1;i<ARGC;i++) { # all given files
if(stat(ARGV[i], fdata)<0) { # use stat
printf("could not stat %s: %s\n", # nonexists n exits
ARGV[i], ERRNO) > "/dev/stderr"
exit 1
}
if(fdata["size"]==0) { # file size check
printf("%s is empty\n",
ARGV[i]) > "/dev/stderr"
exit 2
}
}
exit
}' foo
Output:
foo is empty

Multiple conditions with if in shell scripting

I want to check if a file exists or not along with the no of lines = 4 in a single if condition . Can anyone help here.
if [[ -f report]] && [[`wc -l report` -eq 4 ]]; then
echo " Proceed further"
else
exit 1
fi
This is simpler:
{ [ `wc -l < report` -eq 4 ] || exit; } 2>/dev/null
echo " Proceed further"
Notes:
If report exists and is 4 lines long, then wc -l report
returns:
4 report
...which -eq can't understand. Instead do wc -l < report which
outputs an -eq-friendly:
4
There's no need to check if report exists, since the < redirection
will do that anyway, and returns the same error code.
More specific exit codes. If report does not exist, the exit code is 2. If report is 5 lines long, the exit code is 1.

Nested for loop to enter and exit multiple directories Bash script

As an example, I have 7 directories each containing 4 files. The 4 files follow the following naming convention name_S#_L001_R1_001.fastq.gz. The sed command is to partially keep the unique file name.
I have a nested for loop in order to enter a directory and perform a command, exit the directory and proceed to the next directory. Everything seems to be working beautifully, however the code gets stuck on the last directory looping 4 times.
for f in /completepath/*
do
[ -d $f ] && cd "$f" && echo Entering into $f
for y in `ls *.fastq.gz | sed 's/_L00[1234]_R1_001.fastq.gz//g' | sort -u`
do
echo ${y}
done
done
Example output-
Entering into /completepath/m_i_cast_avpv_1
iavpvcast1_S6
Entering into /completepath/m_i_cast_avpv_2
iavpvcast2_S6
Entering into /completepath/m_i_int_avpv_1
iavpvint1_S5
Entering into /completepath/m_i_int_avpv_2
iavpvint2_S5
Entering into /completepath/m_p_cast_avpv_1
pavpvcast1_S8
Entering into /completepathd/m_p_int_avpv_1
pavpvint1_S7
Entering into /completepath/m_p_int_avpv_2
pavpvint2_S7
pavpvint2_S7
pavpvint2_S7
pavpvint2_S7
Any recommendations of how to correctly exit the inner loop?
It looks like /completepath/ contains some entries that are not directories. When the loop over /completepath/* sees something that's not a directory, it doesn't enter it, thanks to the [ -d $f ] check.
But it still continues to run the next for y in ... loop.
At that point the script is still in the previous directory it has seen.
One way to solve that is to skip the rest of the loop when $f is not a directory:
if [ -d $f ]; then
cd "$f" && echo Entering into $f
else
continue
fi
There's an even better way. By writing /completepath/*/ only directory entries will be matched, so you can simplify your loop to this:
for f in /completepath/*/
do
cd "$f" && echo "Entering into $f" || { echo "Error: could not enter into $f"; continue; }
for y in $(ls *.fastq.gz | sed 's/_L00[1234]_R1_001.fastq.gz//g' | sort -u)
do
echo ${y}
done
done

Shell if else fi syntax error while actual syntax seems correct

So I have a shell script that contains a big if/fi block, and it was working fine until I decided to place an else case for this big if/fi block. Now I am getting this error:
/root/VVPN/Scripts/scriptPrincipal.sh: line 201: syntax error near unexpected token `else'
/root/VVPN/Scripts/scriptPrincipal.sh: line 201: `else'
I went through 8~10 stackoverflow posts where people had exactly the same error, except that all of them were simple syntax errors like a missing space after the [ of the if statement, or a : instead of a ; before the then keyword, or an else intended for an if that was already closed with a fi, etc... (you get the idea :p).
However I've checked my code for all these errors over and over and everything seems to be correct when it comes to if/else/fi syntax. I even showed the code to some colleagues and they too couldn't find the reason for this error.
Here's the code:
if [ ${CP} != "continue" ]
then
echo 'Downloading the necessary files from the server...'
# If folder F*** doesn't exist in /root/VVPN/Numbers
if [ ! -d ${N} ]
then
# Create folders F***, and F***/Results
mkdir ${N}
mkdir ${N}/Results
cd ${N}
# Get real number to factorize from server (e.g: wget server.com:8000/route/to/F1067/F1067)
ROUTE="VVPN/Numbers/${N}/${N}"
REAL_NUM_FILENAME=${N}
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${REAL_NUM_FILENAME} --continue
if [ $? -eq 0 ]; then
break
fi
sleep 1
done
# Get ECM-Program tuned for this number from server (e.g: wget server.com:8000/route/to/F1067/ecm)
ROUTE="VVPN/Numbers/${N}/ecm"
PROGRAM_NAME='ecm'
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${PROGRAM_NAME} --continue
if [ $? = 0 ]; then
break
fi
sleep 1
done
else
# The folder already exists, now we have to check whether the number has ",c" label or not
cd ${N}
fi
# Give the permission to execute program
chmod +x ecm
# Make 6 directories, one for each SPE
for i in {1..6}
do
mkdir "SPE${i}"
cp ecm "SPE${i}/"
done
# if there is no checkpoints:
if [ $CP = $N ]
then
# Get currentSigma for this number from server (e.g : wget server.com:8000/rout/to/F1067/currentSigma.txt)
ROUTE="VVPN/Numbers/${N}/currentSigma"
REAL_FILE_NAME='sigma'
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${REAL_FILE_NAME} --continue
if [ $? = 0 ]; then
break
fi
sleep 1
done
else
#The number has a ",c" label (= w/ checkpoint)
# Get i (server sigma) and C (current job counter) from the server (e.g: wget server.com:8000/route/to/F1067/icy)
ROUTE="icy?number=${N}"
SIGMA_C_Y='icy'
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${SIGMA_C_Y} --continue
if [ $? = 0 ]; then
break
fi
sleep 1
done
i=$(cat ${SIGMA_C_Y} | cut -d "," -f1)
C=$(cat ${SIGMA_C_Y} | cut -d "," -f2)
Y=$(cat ${SIGMA_C_Y} | cut -d "," -f3)
echo $i > sigma
echo $C > /root/VVPN/Scripts/C
# Get the checkpoints from the server
ROUTE="VVPN/Numbers/${N}/CP/${i},${C},${Y}"
for speNum in {1..6}
do
SPE="SPE${speNum}"
touch $SPE/again
for bigX in {0..3}
do
CHECKPOINT="pointsCurve${bigX}.${Y}"
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE}/${SPE}/${CHECKPOINT} -O ${CHECKPOINT} --continue
if [ $? = 0 ]; then
mv $CHECKPOINT $SPE
break
fi
sleep 1
done
done
done
fi
cd ..
else
echo "Found ${N}'s folder on this PS3"
fi
So the else mentioned in the error (at line 201) is actually the last else in the code. The script works fine without this else and the echo that comes right after.
Any help would be much appreciated :)
Could you change if [ $CP = $N ] as below;
if [ "$CP" == "$N" ]
This is not if else problem; For example; if you run the following code, output is same. So You should focus other commands inside if statement.
CP="continue1"
if [ ${CP} != "continue" ]
then
while #this is wrong
echo ok
echo ok
echo ok
else
fi
./test.sh: line 8: syntax error near unexpected token `else'
./test.sh: line 8: `else'

Resources