How to list files and match first line in bash script? - bash

I would like to check for (only) python files for those which do not have the #!/usr/bin/env python in the first line. So, I write a bash script that does the following:
#!/bin/bash
#list all of python files
for file in `find . -name "*.py"`
do
if [ `head -1 $file` != "#!/usr/bin/env python"] then;
echo "no match in file $file"
else then;
echo "match!"
fi
done
However, for some reason I cannot get the if statement correct! I've looked at many questions, but I cannot find one that succinctly describes the issue. Here is the error message:
./run_test.sh: line 9: syntax error near unexpected token `else'
./run_test.sh: line 9: ` else then;'
where am I going awry? Thank you.

You can do something like
find . -type f -name '*.py' -exec \
awk 'NR==1 && /#!\/usr\/bin\/env python/ \
{ print "Match in file " FILENAME; exit } \
{ print "No match in file " FILENAME; exit }' \
{} \;
If you are going to loop over it, don't use a for loop
#!/bin/bash
find . -type f -name '*.py' -print0 | while IFS= read -r -d $'\0' file; do
if [[ $(head -n1 "$file") == "#!/usr/bin/env python" ]]; then
echo "Match in file [$file]"
else
echo "No match in file [$file]"
fi
done

Things to notice:
The [] after your if statement needs correct spacing
The ';' (if you enter a new line is not necessary) goes after the if and not after the then
You added an extra then after the else.
#!/bin/bash
#list all of python files
for file in `find . -name "*.py"`
do
if [ `head -1 $file` != "#!/usr/bin/env python" ];
then
echo "no match in file $file"
else
echo "match!"
fi
done

can you use -exec option by any chance? I find it easier.
find . -name "*.py" -exec head -1 {} | grep -H '#!/usr/bin/env python' \;
You can control the output using grep options.
edit
Thanks to #chepner - To avoid the pipe being swallowed too early:
-exec sh -c "head -1 {} | grep -H '#!/usr/bin/env python'" \;

Related

iterate over lines in file then find in directory

I am having trouble looping and searching. It seems that the loop is not waiting for the find to finish. What am I doing wrong?
I made a loop the reads a file line by line. I then want to use that "name" to search a directory looking to see if a folder has that name. If it exists copy it to a drive.
#!/bin/bash
DIRFIND="$2"
DIRCOPY="$3"
if [ -d $DIRFIND ]; then
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "$line"
FILE=`find "$DIRFIND" -type d -name "$line"`
if [ -n "$FILE" ]; then
echo "Found $FILE"
cp -a "$FILE" "$DIRCOPY"
else
echo "$line not found."
fi
done < "$1"
else
echo "No such file or directory"
fi
Have you tried xargs...
Proposed Solution
cat filenamelist | xargs -n1 -I {} find . -type d -name {} -print | xargs -n1 -I {} mv {} .
what the above does is pipe a list of filenames into find (one at a time), when found find prints the name and passes to xarg which moves the file...
Expansion
file = yogo
yogo -> | xargs -n1 -I yogo find . -type d -name yogo -print | xargs -n1 -I {} mv ./<path>/yogo .
I hope the above helps, note that xargs has the advantage that you do not run out of command line buffer.

Differential backup issue using comparison with md5sum in shell script

Hello I have a quick question. I am trying to implement a differential backup but where I am having trouble is comparing the hash from md5sum.txt to diffmd5.txt
I am getting the following error:
Currently the command runs and gives no errors but files are not being replaced and not backing up any files
#!/bin/bash
bkdest="/home/user/backup/differential/backup_diff"
bksource="/home/user/Documents"
destgen=`find $bkdest/* -exec md5sum {} + > diffmd5.txt`
sourcegen=`find $bksource/* -exec md5sum {} + > md5sum.txt`
$sourcegen
$destgen
$(cat diffmd5.txt) | while read f;
do
if [ $(grep f md5sum.txt | wc -l) -lt 1 ]
then
# Code to backup the file that has changed or is not on record
cp $(cut -d ' ' -f2-- <<< $f) $bkdest
fi
done
# Afterwards, update md5hashes to list newly backed up files
$sourcegen
Please help me out on figuring out where I went wrong. Thank you!
Guess this is my error when running in debug mode
Try 'cp --help' for more information.
grep: md5sum.txt: No such file or directory
cut: invalid byte, character or field list
Try 'cut --help' for more information.
cp: missing destination file operand after ‘/home/dmitriy/backup/differential/backup_diff’
Lets try with:
#!/bin/bash
bkdest="/home/user/backup/differential/backup_diff"
bksource="/home/user/Documents"
cd $bkdest
find . -type f -exec md5sum {} \; > /tmp/md5dest.txt
cd $bksource
find . -type f -exec md5sum {} \; | while read c f; do
if fgrep "$c" /tmp/md5dest.txt | fgrep -q "$f"; then
echo "$f" ignored
else
cp $f $bkdest
fi
done

find statement in cygwin bash script

for i in `find . -type f -name "VF-Outlet*.edi" -exec basename \{} \;` ; do
if [ -n "${i}" ];
then
echo file "VF-Outlet found";
sed -e 's/\*UK\*00/\*UP\*/g;s/XQ.*$/XQ\*H\*20150104\*20150110/g' $i > ${i}_fix
else
echo file "VF-Outlet" not found;
fi
done
The above code works if the file is found. The 'echo' statement prints file found.
If the file is not found however, nothing prints. I tried all the various tests for empty string, and unset variables, nothing works.
Also if I try:
i=`find . -type f -name "VF-Outlet*.edi" -exec basename \{} \;`;
Then do the test:
if [ -n "${i}" ];
then
echo file ${i} found;
else
echo file "VF-Outlet" not found;
fi
done
It works correctly if the file is found or not.
Need help in figuring this out. I need the for loop to test multiple files.
The reason it is not working is due to the fact that "for" does not take null value as input for the variable "i"
For ex:
for i in echo > /dev/null; do echo hi; done
The above command wont give any result, because no value has been assigned to value $i for running the loop.
In the case mentioned here if we check the script in debug mode, we can see that the script dies at initial variable assignment.
# sh -x script.sh
+ find . -type f -name VF-Outlet*.edi -exec basename {} ;
here, script.sh file contains the script you have provided.
If there is a file present in the directory, the script will successfully execute.
# sh -x tet
+ find . -type f -name VF-Outlet*.edi -exec basename {} ;
+ [ -n VF-Outlet1.edi ]
+ echo file VF-Outlet found
file VF-Outlet found
As #shellter mentioned, this not how I would have done. You can use -f instead of -n to check if a file exists.
Hope this helps!

Is there a way to pipe from a variable?

I'm trying to find all files in a file structure above a certain file size, list them, then delete them. What I currently have looks like this:
filesToDelete=$(find $find $1 -type f -size +$2k -ls)
if [ -n "$filesToDelete" ];then
echo "Deleting files..."
echo $filesToDelete
$filesToDelete | xargs rm
else
echo "no files to delete"
fi
Everything works, except the $filesToDelete | xargs rm, obviously. Is there a way to use pipe on a variable? Or is there another way I could do this? My google-fu didn't really find anything, so any help would be appreciated.
Edit: Thanks for the information everyone. I will post the working code here now for anyone else stumbling upon this question later:
if [ $(find $1 -type f -size +$2k | wc -l) -ge 1 ]; then
find $1 -type f -size +$2k -exec sh -c 'f={}; echo "deleting file $f"; rm $f' {} \;
else
echo "no files above" $2 "kb found"
fi
As already pointed out, you don't need piping a var in this case. But just in case you needed it in some other situation, you can use
xargs rm <<< $filesToDelete
or, more portably
echo $filesToDelete | xargs rm
Beware of spaces in file names.
To also output the value together with piping it, use tee with process substitution:
echo "$x" | tee >( xargs rm )
You can directly use -exec to perform an action on the files that were found in find:
find $1 -type f -size +$2k -exec rm {} \;
The -exec trick makes find execute the command given for each one of the matches found. To refer the match itself we have to use {} \;.
If you want to perform more than one action, -exec sh -c "..." makes it. For example, here you can both print the name of the files are about to be removed... and remove them. Note the f={} thingy to store the name of the file, so that it can be used later on in echo and rm:
find $1 -type f -size +$2k -exec sh -c 'f={}; echo "removing $f"; rm $f' {} \;
In case you want to print a message if no matches were found, you can use wc -l to count the number of matches (if any) and do an if / else condition with it:
if [ $(find $1 -type f -size +$2k | wc -l) -ge 1 ]; then
find $1 -type f -size +$2k -exec rm {} \;
else
echo "no matches found"
fi
wc is a command that does word count (see man wc for more info). Doing wc -l counts the number of lines. So command | wc -l counts the number of lines returned by command.
Then we use the if [ $(command | wc -l) -ge 1 ] check, which does an integer comparison: if the value is greater or equal to 1, then do what follows; otherwise, do what is in else.
Buuuut the previous approach was using find twice, which is a bit inefficient. As -exec sh -c is opening a sub-shell, we cannot rely on a variable to keep track of the number of files opened. Why? Because a sub-shell cannot assign values to its parent shell.
Instead, let's store the files that were deleted into a file, and then count it:
find . -name "*.txt" -exec sh -c 'f={}; echo "$f" >> /tmp/findtest; rm $f' {} \;
if [ -s /tmp/findtest ]; then #check if the file is empty
echo "file has $(wc -l < /tmp/findtest) lines"
# you can also `cat /tmp/findtest` here to show the deleted files
else
echo "no matches"
fi
Note that you can cat /tmp/findtest to see the deleted files, or also use echo "$f" alone (without redirection) to indicate while removing. rm /tmp/findtest is also an option, to do once the process is finished.
You don't need to do all this. You can directly use find command to get the files over a particular size limit and delete it using xargs.
This should work:
#!/bin/bash
if [ $(find $1 -type f -size +$2k | wc -l) -eq 0 ]; then
echo "No Files to delete"
else
echo "Deleting the following files"
find $1 -size +$2 -exec ls {} \+
find $1 -size +$2 -exec ls {} \+ | xargs rm -f
echo "Done"
fi

Bash script to list files not found

I have been looking for a way to list file that do not exist from a list of files that are required to exist. The files can exist in more than one location. What I have now:
#!/bin/bash
fileslist="$1"
while read fn
do
if [ ! -f `find . -type f -name $fn ` ];
then
echo $fn
fi
done < $fileslist
If a file does not exist the find command will not print anything and the test does not work. Removing the not and creating an if then else condition does not resolve the problem.
How can i print the filenames that are not found from a list of file names?
New script:
#!/bin/bash
fileslist="$1"
foundfiles="~/tmp/tmp`date +%Y%m%d%H%M%S`.txt"
touch $foundfiles
while read fn
do
`find . -type f -name $fn | sed 's:./.*/::' >> $foundfiles`
done < $fileslist
cat $fileslist $foundfiles | sort | uniq -u
rm $foundfiles
#!/bin/bash
fileslist="$1"
while read fn
do
FPATH=`find . -type f -name $fn`
if [ "$FPATH." = "." ]
then
echo $fn
fi
done < $fileslist
You were close!
Here is test.bash:
#!/bin/bash
fn=test.bash
exists=`find . -type f -name $fn`
if [ -n "$exists" ]
then
echo Found it
fi
It sets $exists = to the result of the find. the if -n checks if the result is not null.
Try replacing body with [[ -z "$(find . -type f -name $fn)" ]] && echo $fn. (note that this code is bound to have problems with filenames containing spaces).
More efficient bashism:
diff <(sort $fileslist|uniq) <(find . -type f -printf %f\\n|sort|uniq)
I think you can handle diff output.
Give this a try:
find -type f -print0 | grep -Fzxvf - requiredfiles.txt
The -print0 and -z protect against filenames which contain newlines. If your utilities don't have these options and your filenames don't contain newlines, you should be OK.
The repeated find to filter one file at a time is very expensive. If your file list is directly compatible with the output from find, run a single find and remove any matches from your list:
find . -type f |
fgrep -vxf - "$1"
If not, maybe you can massage the output from find in the pipeline before the fgrep so that it matches the format in your file; or, conversely, massage the data in your file into find-compatible.
I use this script and it works for me
#!/bin/bash
fileslist="$1"
found="Found:"
notfound="Not found:"
len=`cat $1 | wc -l`
n=0;
while read fn
do
# don't worry about this, i use it to display the file list progress
n=$((n + 1))
echo -en "\rLooking $(echo "scale=0; $n * 100 / $len" | bc)% "
if [ $(find / -name $fn | wc -l) -gt 0 ]
then
found=$(printf "$found\n\t$fn")
else
notfound=$(printf "$notfound\n\t$fn")
fi
done < $fileslist
printf "\n$found\n$notfound\n"
The line counts the number of lines and if its greater than 0 the find was a success. This searches everything on the hdd. You could replace / with . for just the current directory.
$(find / -name $fn | wc -l) -gt 0
Then i simply run it with the files in the files list being separated by newline
./search.sh files.list

Resources