I have a script with the following command to upload a bunch of zip files to a site:
find . -name "*.zip" -exec echo {} \; -exec sh -c 'err=$(curl -s --data-binary "#{}" http://mystorage.com | jq -r ".error"); if [ -z $err ] || [ $err = "file already exists" ]; then exit 0; else exit 1; fi' \;
The intention is that if any file fail to upload with the reason other than "file already exists" then the script must fail. However, if i run this command alone, it never exit with 1. My guess is that the subshell opened in the 2nd -exec returns 1 but the -exec ignore the return status and return 0 for the whole find command. Is there a way to make my command fail when the subshell fail?
I wouldn't bother with find for this. Just use an ordinary loop (with the globstar option to search recursively, if necessary).
shopt -s globstar nullglob
for f in **/*.zip; do
err=$(curl -s --data-binary "#$f" http://mystorage.com | jq -r ".error")
if [ -n "$err" ] && [ "$err" = "file already exists" ]; then
exit 1
fi
done
Note that you don't want to exit 0 when the first job succeeds; just do nothing and let the next file be uploaded.
You need to look at the four exec forms find can use:
https://www.man7.org/linux/man-pages/man1/find.1.html
-exec command {} \;
-exec command {} \+
-execdir command {} \;
-execdir command {} \+
They all have different behaviors regarding their boolean value in the search boolean expression.
My brief interpretation (I have only read the manpages, I have not actually tried this):
Commands that end with a semicolon result in the -exec term having a true or false value, and do not effect the exit value of find even when the wrapped command has a nonzero exit value.
Commands that end with a plus cause find to exit with a nonzero value when a wrapped command has a nonzero result value.
So I think you want to switch to the -exec form where the command ends with a '+'.
But really I think you need a better multithreaded script that can handle each failure separately, and remember which ones it has successfully uploaded.
Using GNU find:
find . -name '*.zip' -print -exec sh -c '
err=$(curl -s --data-binary "#$1" http://mystorage.com | jq -r ".error")
[ -z "$err" ] || [ "$err" = 'file already exists' ] || exit 1
' _ {} \; \
-o -quit
Note that this exits if .error is empty. That may be what you want, but maybe you want this (or similar):
find . -name '*.zip' -print -exec sh -c '
response=$(curl -s --data-binary "#$1" http://mystorage.com) || exit 1
[ "$response" ] || exit 1
[ "$(echo "$response" | jq -r .error)" = 'file already exists' ] && exit 1
' _ {} \; \
-o -quit
Now it exits prematurely if curl fails, if the response is empty, or if .error matches.
Related
I want to 'git rm' a bunch of files that are found by a 'find' command. The files should have a certain suffix. I got this:
TEST_PATH='/usr/src'
function main() {
for i in "$#"
do
echo "current i = ${i}"
COMMAND='find $TEST_PATH -maxdepth 20 -name '*_${i}.txt' -exec git rm {} \;'
# COMMAND="$(find $TEST_PATH -maxdepth 20 name '*_${i}.txt' -print0 | xargs -0 -I{} cp {} .)"
# COMMAND="find $TEST_PATH -maxdepth 20 -name '*_${i}.txt' -exec cp {} . \;"
# COMMAND="find . '*.BUILD' | while read file; do echo "$file"; done \;"
done
echo "Running Command: $COMMAND"
$COMMAND
}
gbash::main "$#"
Running it will throw an error like this:
$ sh abc.sh 123
current i = 123
Running Command: find ../../src/python/servers/innertube/tests/ -maxdepth 20 -name "*_9421870.txt" -exec rm {}\;
find: missing argument to `-exec'
I've read and tried all the solutions on stackoverflow (see the commented out code) but none works...
Update
The problem is that you should eval contents of the variable containing command:
eval $COMMAND
From man eval:
The eval utility shall construct a command by concatenating arguments together, separating each with a <space> character. The constructed command shall be read and executed by the shell.
Original answer
Replace {}\; with {} \; or {} +.
Read the man page for find. The action used in your command is documented as:
-exec command ;
Execute command; true if 0 status is returned. All following arguments > to find are taken to be arguments to the command until an argument consisting of ; is encountered. The string {} is replaced by the current file name being processed everywhere it occurs in the arguments to the command...
So the command failed because the {}\; sequence is interpreted as command.
I would like to check for (only) python files for those which do not have the #!/usr/bin/env python in the first line. So, I write a bash script that does the following:
#!/bin/bash
#list all of python files
for file in `find . -name "*.py"`
do
if [ `head -1 $file` != "#!/usr/bin/env python"] then;
echo "no match in file $file"
else then;
echo "match!"
fi
done
However, for some reason I cannot get the if statement correct! I've looked at many questions, but I cannot find one that succinctly describes the issue. Here is the error message:
./run_test.sh: line 9: syntax error near unexpected token `else'
./run_test.sh: line 9: ` else then;'
where am I going awry? Thank you.
You can do something like
find . -type f -name '*.py' -exec \
awk 'NR==1 && /#!\/usr\/bin\/env python/ \
{ print "Match in file " FILENAME; exit } \
{ print "No match in file " FILENAME; exit }' \
{} \;
If you are going to loop over it, don't use a for loop
#!/bin/bash
find . -type f -name '*.py' -print0 | while IFS= read -r -d $'\0' file; do
if [[ $(head -n1 "$file") == "#!/usr/bin/env python" ]]; then
echo "Match in file [$file]"
else
echo "No match in file [$file]"
fi
done
Things to notice:
The [] after your if statement needs correct spacing
The ';' (if you enter a new line is not necessary) goes after the if and not after the then
You added an extra then after the else.
#!/bin/bash
#list all of python files
for file in `find . -name "*.py"`
do
if [ `head -1 $file` != "#!/usr/bin/env python" ];
then
echo "no match in file $file"
else
echo "match!"
fi
done
can you use -exec option by any chance? I find it easier.
find . -name "*.py" -exec head -1 {} | grep -H '#!/usr/bin/env python' \;
You can control the output using grep options.
edit
Thanks to #chepner - To avoid the pipe being swallowed too early:
-exec sh -c "head -1 {} | grep -H '#!/usr/bin/env python'" \;
Ok so this code works
if grep -lq something file.txt ; then
So why something like this doesnt? what am i doing wrong?
if find . -name file.txt -exec grep -lq something {} \;
its always true as long as the directory exist.
From the find man page:
Exit Status
find exits with status 0 if all files are processed successfully, greater than 0 if errors occur. This is deliberately a very broad description, but if the return value is non-zero, you should not rely on the correctness of the results of find.
What you're getting back from your command is the exit value of the find and not the grep. Find almost always returns an exit value of zero as long as the query is good.
I was thinking this might work:
find . -name file.txt -print0 | xargs --0 grep -lq something
But that will return only the exit status of the last execution of grep. If grep was executed multiple times, you won't get the intermediate values. However, this probably won't be an issue with your command.
A very simple way is to check if find's output is empty:
output=$( find . -name file.txt -exec grep -lq something {} \; )
if [ -n "$output" ]
then
# found
else
# not found
fi
One approach which will short-circuit as soon as a file containing the desired contents is found (presuming that your intent is to look for whether any file matches, as opposed to whether every file matches):
check_for_content() {
target=$1; shift
while IFS= read -r -d '' filename; do
if grep -lq -e "$target" "$filename"; then
return 0
fi
done < <(find "$#" -print0)
return 1
}
Usage:
check_for_content thing-to-look-for -type f -name file.txt
for i in `find . -type f -name "VF-Outlet*.edi" -exec basename \{} \;` ; do
if [ -n "${i}" ];
then
echo file "VF-Outlet found";
sed -e 's/\*UK\*00/\*UP\*/g;s/XQ.*$/XQ\*H\*20150104\*20150110/g' $i > ${i}_fix
else
echo file "VF-Outlet" not found;
fi
done
The above code works if the file is found. The 'echo' statement prints file found.
If the file is not found however, nothing prints. I tried all the various tests for empty string, and unset variables, nothing works.
Also if I try:
i=`find . -type f -name "VF-Outlet*.edi" -exec basename \{} \;`;
Then do the test:
if [ -n "${i}" ];
then
echo file ${i} found;
else
echo file "VF-Outlet" not found;
fi
done
It works correctly if the file is found or not.
Need help in figuring this out. I need the for loop to test multiple files.
The reason it is not working is due to the fact that "for" does not take null value as input for the variable "i"
For ex:
for i in echo > /dev/null; do echo hi; done
The above command wont give any result, because no value has been assigned to value $i for running the loop.
In the case mentioned here if we check the script in debug mode, we can see that the script dies at initial variable assignment.
# sh -x script.sh
+ find . -type f -name VF-Outlet*.edi -exec basename {} ;
here, script.sh file contains the script you have provided.
If there is a file present in the directory, the script will successfully execute.
# sh -x tet
+ find . -type f -name VF-Outlet*.edi -exec basename {} ;
+ [ -n VF-Outlet1.edi ]
+ echo file VF-Outlet found
file VF-Outlet found
As #shellter mentioned, this not how I would have done. You can use -f instead of -n to check if a file exists.
Hope this helps!
I'm trying to find all files in a file structure above a certain file size, list them, then delete them. What I currently have looks like this:
filesToDelete=$(find $find $1 -type f -size +$2k -ls)
if [ -n "$filesToDelete" ];then
echo "Deleting files..."
echo $filesToDelete
$filesToDelete | xargs rm
else
echo "no files to delete"
fi
Everything works, except the $filesToDelete | xargs rm, obviously. Is there a way to use pipe on a variable? Or is there another way I could do this? My google-fu didn't really find anything, so any help would be appreciated.
Edit: Thanks for the information everyone. I will post the working code here now for anyone else stumbling upon this question later:
if [ $(find $1 -type f -size +$2k | wc -l) -ge 1 ]; then
find $1 -type f -size +$2k -exec sh -c 'f={}; echo "deleting file $f"; rm $f' {} \;
else
echo "no files above" $2 "kb found"
fi
As already pointed out, you don't need piping a var in this case. But just in case you needed it in some other situation, you can use
xargs rm <<< $filesToDelete
or, more portably
echo $filesToDelete | xargs rm
Beware of spaces in file names.
To also output the value together with piping it, use tee with process substitution:
echo "$x" | tee >( xargs rm )
You can directly use -exec to perform an action on the files that were found in find:
find $1 -type f -size +$2k -exec rm {} \;
The -exec trick makes find execute the command given for each one of the matches found. To refer the match itself we have to use {} \;.
If you want to perform more than one action, -exec sh -c "..." makes it. For example, here you can both print the name of the files are about to be removed... and remove them. Note the f={} thingy to store the name of the file, so that it can be used later on in echo and rm:
find $1 -type f -size +$2k -exec sh -c 'f={}; echo "removing $f"; rm $f' {} \;
In case you want to print a message if no matches were found, you can use wc -l to count the number of matches (if any) and do an if / else condition with it:
if [ $(find $1 -type f -size +$2k | wc -l) -ge 1 ]; then
find $1 -type f -size +$2k -exec rm {} \;
else
echo "no matches found"
fi
wc is a command that does word count (see man wc for more info). Doing wc -l counts the number of lines. So command | wc -l counts the number of lines returned by command.
Then we use the if [ $(command | wc -l) -ge 1 ] check, which does an integer comparison: if the value is greater or equal to 1, then do what follows; otherwise, do what is in else.
Buuuut the previous approach was using find twice, which is a bit inefficient. As -exec sh -c is opening a sub-shell, we cannot rely on a variable to keep track of the number of files opened. Why? Because a sub-shell cannot assign values to its parent shell.
Instead, let's store the files that were deleted into a file, and then count it:
find . -name "*.txt" -exec sh -c 'f={}; echo "$f" >> /tmp/findtest; rm $f' {} \;
if [ -s /tmp/findtest ]; then #check if the file is empty
echo "file has $(wc -l < /tmp/findtest) lines"
# you can also `cat /tmp/findtest` here to show the deleted files
else
echo "no matches"
fi
Note that you can cat /tmp/findtest to see the deleted files, or also use echo "$f" alone (without redirection) to indicate while removing. rm /tmp/findtest is also an option, to do once the process is finished.
You don't need to do all this. You can directly use find command to get the files over a particular size limit and delete it using xargs.
This should work:
#!/bin/bash
if [ $(find $1 -type f -size +$2k | wc -l) -eq 0 ]; then
echo "No Files to delete"
else
echo "Deleting the following files"
find $1 -size +$2 -exec ls {} \+
find $1 -size +$2 -exec ls {} \+ | xargs rm -f
echo "Done"
fi