How to use if statements in bash scripting - bash

I am trying to filter lines in a file using a for loop followed by an if statement with the following code:
#!/bin/bash
> oldFiles.txt
files=$(grep ' jane ' ~/data/list.txt | cut -d ' ' -f 3)
for file in $files; do
file="~${file}";
if [ -e "$file" ]; then
echo $file;
fi
done
Where list.txt is a list of names like this:
001 jane /data/jane_profile_07272018.doc
002 kwood /data/kwood_profile_04022017.doc
003 pchow /data/pchow_profile_05152019.doc
004 janez /data/janez_profile_11042019.doc
005 jane /data/jane_pic_07282018.jpg
006 kwood /data/kwood_pic_04032017.jpg
007 pchow /data/pchow_pic_05162019.jpg
008 jane /data/jane_contact_07292018.csv
009 kwood /data/kwood_contact_04042017.csv
010 pchow /data/pchow_contact_05172019.csv
I need to append the result in the file oldFiles.txt but i'm echo-ing it for now, yet it doesn't show anything on screen.
Everything looks perfect, for example when i replace "$file" in the if statement with an actual value of a file that exists (like this one /data/jane_profile_07272018.doc) then it works. Here's a list of files that exists in the file system:
jane_contact_07292018.csv
janez_profile_11042019.doc
kwood_profile_04022017.doc
pchow_pic_05162019.jpg
jane_profile_07272018.doc
kwood_pic_04032017.jpg
list.txt
test.txt
My script should echo only files that exist in the filesystem, hence the if statement.
What is there missing !?

You were almost there!!!
I would change a little bit your code to
#!/bin/bash
OLDFILES="odlfiles.txt"
FILES=$(egrep " jane " /data/list.txt | cut -d ' ' -f 3 )
#Alternative using awk with space as default delimiter to get the 3rd column ;)
#FILES=$(egrep " jane " /data/list.txt | awk '{print $3}' )
for file in $FILES; do
if [ -f $file ]; then
echo -e "[EXISTS]\tAppending to ${OLDFILES} path file: ${file}"
#Appending now to target file
echo $file >> $OLDFILES
fi
done
I think you are aware of the use of grep in order of filtering the content of the list.txt. I would recommend you to use in such cases egrep and then the pattern/regex which suits your needs the most.
In this case:
#!/bin/bash
OLDFILES="odlfiles.txt"
WORD2MATCH="jane"
FILES=$(egrep "\s${WORD2MATCH}\s" list.txt | awk '{print $3}')
for file in $FILES; do
if [ -f $file ]; then
echo -e "[EXISTS]\tAppending to ${OLDFILES} path file: ${file}"
echo $file >> $OLDFILES
fi
done
Or maybe in the compact form :)
#!/bin/bash
OLDFILES="odlfiles.txt"
WORD2MATCH="jane"
FILES=$(egrep "\s${WORD2MATCH}\s" list.txt | awk '{print $3}')
for file in $FILES; do
[ -f $file ] && echo $file >> $OLDFILES && echo -e "[APPENDED]\t${file}"
done

Related

in shell script how to print a line if the previous and the next line has a blank and the

I have a file like
abc
1234567890
0987654321
cde
fgh
ijk
1234567890
0987654321
I need to write a script that extract the lines with a blank before and after, in the example should be like this:
cde
fgh
I guess that awk or sed could do the work but I wasn't able to make them work. Any help?
Here is the solution.
#!/bin/bash
amt=$(sed -n '$=' path-to-your-file)
i=0
while :
do
((i++))
if [ $i == $amt ]; then
break
fi
if ! [ $i == 1 ]; then
j=$(expr $i - 1)
emp=$(sed $j'!d' path-to-your-file)
if [ h$emp == h ]; then
j=$(expr $i + 1)
emp=$(sed $j'!d' path-to-your-file)
if [ h$emp == h ]; then
emp=$(sed $i'!d' path-to-your-file)
echo >> extracted $emp
fi
fi
fi
done
With awk:
awk '
BEGIN{
RS=""
FS="\n"
}
NF==1' file
Prints:
cde
fgh
very simple solution
cat "myfile.txt" | grep -A 1 '^$' | grep -B 1 '^$' | grep -v -e '^--$' | grep -v '^$'
assuming "--" is the default group separator
you may get ride of group separator by other means like
--group-separator="" or --no-group-separator options
but depends of grep program variant (BSD, Gnu, OSX... )

Creating file with Bash scripting for loop and if test

Problem Statement
Find files using bash script
In this section, you are going to write a script named findJane.sh within the scripts directory.
This script should catch all "jane" lines and store them in another text file called oldFiles.txt. You will complete the script using the command we practiced in earlier sections. Don't worry, we'll guide you throughout the whole process.
Navigate to /scripts directory and create a new file named findJane.sh.
Mycode
#!/bin/bash
>oldFiles.txt
files=$(grep " jane " ../data/list.txt | cut -d ' ' -f 3)
for i in $files:do
do if test -e ~/data/"$i"; then
echo "$i" >> OldFiles.txt;
else
echo "File doesn't exist"; fi
done
output now
file does not exist
file does not exist
file does not exist
it should not print nothing
and cat oldFiles.txt should return all those files with name 'jane
Where i am coding wrong
Guide
Create the text file oldFiles.txt and make sure it's empty. This oldFiles.txt file should save files with username "jane".
Now, search for all lines that contain the name "jane" and save the file names into a variable. Let's call this variable files, we will refer to it with that name later in the lab.
Since none of the files present in the file list.txt are available in the file system, check if file names present in files variable are actually present in the file system. To do this, we'll use the test command that we practiced in the previous section.
Now, iterate over the files variable and add a test expression within the loop. If the item within the files variable passes the test, add/append it to the file oldFiles.txt.
I have the same problem, just don't call the files by $in grep
#!/bin/bash
files= grep ' jane ' ../data/list.txt | cut -d ' ' -f 3
for file in files; do
if test -e ~/data/$file; then
echo $file >> oldFiles.txt;
else
echo "File dosen't exsist"; fi
done
And make the new fie .oldFiles.txt in the command line
like
user#linux: ~/program$ ./findJane.sh > oldFiles.txt
The code shall work
If you're trying to do a Qwicklabs assignment, try this
#!/bin/bash
> oldFiles.txt
files=$(grep " jane " /home/<Student-id>/data/list.txt | cut -d ' ' -f 3)
for i in $files;do
if test -e ~/$i;then
echo $i>>oldFiles.txt;
else echo "not working"; fi
done
#!/bin/bash
>oldFiles.txt
files=$(grep " jane " ../data/list.txt | cut -d ' ' -f 3)
for i in $files; do
if test -e ~/$i; then
echo $i >> OldFiles.txt;
else
echo "File doesn't exist"; fi
done
you have just some typos. this is "working" code for your example
#!/bin/bash
>oldFiles.txt
files=$(grep " jane " ../data/list.txt | cut -d ' ' -f 3)
for i in $files; do
if test -e ~/data/"$i"; then
echo "$i" >> OldFiles.txt;
else
echo "File doesn't exist"; fi
done
however, this works only if list.txt doesn't contain any spaces except as delimiters between (at least) three columns
it won't work for Jane because it doesn't match jane
it won't work when there are more spaces like jane example.doc
it won't work for file names containing spaces like best movies.txt
it makes less sense to show how to do it right because the concept of list.txt is bad choice
#!/bin/bash
>oldFiles.txt
files=$(grep " jane " ../data/list.txt | cut -d ' ' -f 3)
for file in $files; do
if [ -e $HOME$file ] ; then
echo $HOME$file >> oldFiles.txt;
fi
done
#!/bin/bash
> oldFiles.txt
files=$(grep ' jane ' ../data/list.txt | cut -d ' ' -f 3)
#echo $files
for file in $files; do
if test -e ~/$file; then
echo "File path added !!"
echo $file >> oldFiles.txt;
fi
done
You need to check the path and then append it to the oldFile
#1/bin/bash
>oldFiles.txt
files=$(grep "jane " ../data/list.txt | cut -d' ' -f3)
for f in $files; do
if [ -e $HOME$f ];then
echo $HOME$f >> oldFiles.txt;
fi
done
I made a few fixes to your code and now works 😉
#!/bin/bash
> oldFiles.txt
files=$(grep " jane " ../data/list.txt | cut -d " " -f 3)
for file in $files; do
if test -e ~/$file; then echo $file >> oldFiles.txt; else echo "File doesn't exist"; fi done

How to pass a variable string to a file txt at the biginig of test?

I have a problem
I Have a program general like this gene.sh
that for all file (es file: geneX.csv) make a directory with the name of gene (example: Genex/geneX.csv) next this program compile an other program inside gene.sh but this progrm need a varieble and I dont know how do it.
this is the program gene.sh
#!/bin/bash
# Create a dictory for each file *.xls and *.csv
for fname in *.xlsx *csv
do
dname=${fname%.*}
[[ -d $dname ]] || mkdir "$dname"
mv "$fname" "$dname"
done
# For each gene go inside the directory and compile the programs getChromosomicPositions.sh to have the positions, and getHapolotipeStings.sh to have the variants
for geni in */; do
cd $geni
z=$(tail -n 1 *.csv | tr ';' "\n" | wc -l)
cd ..
cp getChromosomicPositions.sh $geni --->
cp getHaplotypeStrings.sh $geni
cd $geni
export z
./getChromosomicPositions.sh *.csv
export z
./getHaplotypeStrings.sh *.csv
cd ..
done
This is the program getChromosomichPositions.sh:
rm chrPosRs.txt
grep '^Haplotype\ ID' $1 | cut -d ";" -f 4-61 | tr ";" "\n" | awk '{print "select chrom,chromStart,chromEnd,name from snp147 where name=\""$1"\";"}' > listOfQuery.txt
while read l; do
echo $l > query.txt
mysql -h genome-mysql.cse.ucsc.edu -u genome -A -D hg38 --skip-column-names < query.txt > queryResult.txt
if [[ "$(cat queryResult.txt)" == "" ]];
then
cat query.txt |
while read line; do
echo $line | awk '$6 ~/rs/ {print $6}' > temp.txt;
if [[ "$(cat temp.txt)" != "" ]];
then cat temp.txt | awk -F'name="' '{print $2}' | sed -e 's/";//g' > temp.txt;
./getHGSVposHG19.sh temp.txt ---> Hear the problem--->
else
echo $line | awk '{num=sub(/.*:g\./,"");num+=sub(/\".*/,"");if(num==2){print};num=""}' > temp2.txt
fi
done
cat query.txt >> varianti.txt
echo "Missing Data" >> chrPosRs.txt
else
cat queryResult.txt >> chrPosRs.txt
fi
done < listOfQuery.txt
rm query*
hear the problem:
I need to enter in the file temp.txt and put automatically at the beginning of the file the variable $geni of the program gene.sh
How can I do that?
Why not pass "$geni" as say the first argument when invoking your script, and treating the rest of the arguments as your expected .csv files.
./getChromosomicPositions.sh "$geni" *.csv
Alternatively, you can set it as environment variable for the script, so that it can be used there (or just export it).
geni="$geni" ./getChromosomicPositions.sh *.csv
In any case, once you have it available in the second script, you can do
if passed as the first argument:
echo "${1}:$(cat temp.txt | awk -F'name="' '{print $2}' | sed -e 's/";//g')
or if passed as environment variable:
echo "${geni}:$(cat temp.txt | awk -F'name="' '{print $2}' | sed -e 's/";//g')

How to copy only that data to a file which is not present in that file in shell script / bash?

I have to put some data in a file which should be unique.
suppose in
file1 I have following data.
ABC
XYZ
PQR
and now I want to add MNO DES ABC then it should only copy "MNO" and "DES" as "ABC" is already present.
file1 should look like
ABC
XYZ
PQR
MNO
DES
(ABC should be there for only once.)
Easiest way: this sholud add non-matching line in f1
diff -c f1 f2|grep ^+|awk -F '+ ' '{print $NF}' >> f1
or if '+ ' is going to be a part of actual text:
diff -c f1 f2|grep ^+|awk -F '+ ' '{ for(i=2;i<=NF;i++)print $i}' >> f1
shell script way:
I have compare script that compares line counts/lenght etc.. but for your requirement I think below part should do the job....
input:
$ cat f1
ABC
XYZ
PQR
$ cat f2
MNO
DES
ABC
output after script*
$ ./compareCopy f1 f2
-----------------------------------------------------
comparing f1 f2
-----------------------------------------------------
Lines check - DONE
$ cat f1
ABC
XYZ
PQR
DES
MNO
#!/bin/sh
if [ $# != "2" ]; then
echo
echo "Requires arguments from command prompt"
echo "Usage: compare <file1> <file2>"
echo
exit
fi
proc="compareCopy"
#sort files for line by line compare
cat $1|sort > file1.tmp
cat $2|sort > file2.tmp
echo "-----------------------------------------------------"
echo " comparing $1 $2" |tee ${proc}_compare.result
echo "-----------------------------------------------------"
file1_lines=`wc -l $1|cut -d " " -f1`
file2_lines=`wc -l $2|cut -d " " -f1`
#Check each line
x=1
while [ "${x}" -le "${file1_lines}" ]
do
f1_line=`sed -n ${x}p file1.tmp`
f2_line=`sed -n ${x}p file2.tmp`
if [ "${f1_line}" != "${f2_line}" ]; then
echo "line number ${x} don't match in both $1 and $2 files" >> ${proc}_compare.result
echo "$1 line: "${f1_line}"" >> ${proc}_compare.result
echo "$2 line: "${f2_line}"" >> ${proc}_compare.result
# so add this line in file 1
echo $f2_line >> $1
fi
x=$[${x} +1]
done
rm -f file1.tmp file2.tmp
echo "Lines check - DONE" |tee -a ${proc}_compare.result
Use fgrep:
fgrep -vf file1 file2 > file2.tmp && cat file2.tmp >> file1 && rm file2.tmp
which fetches all lines of file2 that are not in file1 and appends the result to file1.
You may want to take a look at this post: grep -f maximum number of patterns?
Perl one liner
file one:
1
2
3
file two:
1
4
3
Print Only Unique Line
perl -lne 'print if ++$n{ $_ } == 1 ' file_one.txt file_two.txt
Or
perl -lne 'print unless ++$n{ $_ } ' file_one.txt file_two.txt
output
1
4
3
2
The natural way:
sort -u File1 File2 >Temp && mv Temp File1
The tricky way if the files are already sorted:
comm File1 File2 | awk '{$1=$1};1' >Temp && mv Temp File1

Shell script: count the copies of each line from a txt

I would like to count the copies of each line in a txt file and I have tried so many things until know, but none worked well. In my case the text has just a word in each line.
This was my last try
echo -n 'enter file for edit: '
read file
for line in $file ; do
echo 'grep -w $line $file'
done; <$file
For example:
input file
a
a
a
c
c
Output file
a 3
c 2
Thanks in advance.
$ sort < $file | uniq -c | awk '{print $2 " " $1}'

Resources