I have this find command:
$ find . -name '*.jar' -exec grep -Hls BuildConfig {} \;
I need to execute this command for each result of command find above:
$ zip -d RESULTPATH "*/BuildConfig.class"
So can I do this with for loop or not? Can I create .sh file that will be doing what I need?
Thanks!
A clean way to run a command for each result of a grep command with xargs, is using the -Z or --null flag with grep to make the results terminated by null, and the -0 flag with xargs so that it expects values terminated by null, like this:
find . -name '*.jar' -exec grep -Z BuildConfig {} \; | xargs -0 zip -d RESULTPATH "*/BuildConfig.class
I removed the Hls flags because they all seem pointless (even harmful) in your use case.
But I'm afraid this will not actually work for your case,
because a .jar file is usually a binary file (zipped archive of Java classes),
so I don't think the grep will ever match anything. You can give zgrep a try to search inside the jars.
The grep command has two channels for information out of it. The first and most obvious one is of course stdout, where it sends matches it finds. But if it finds no matches, it also uses an exit value > 0 to inform you. Combined with the -q (quiet) option, you can use this as a more intelligent option for find:
$ find . -name '*.jar' -exec zgrep -sq BuildConfig {} \; -exec zip -d {} "*/BuildConfig.class" \;
This assumes that you want to search through the compressed file using grep -Z, of course. :)
Or for easier reading:
find . -name '*.jar' \
-exec zgrep -sq BuildConfig {} \; \
-exec zip -d {} "*/BuildConfig.class" \;
Find operates by running each test in order. You can think of the options as a series of filters; the -name option is the first filters, and a file only gets passed to the second -exec if the preceding -exec exited without errors.
Related
I have to find files & execute a python command on them. There are several thousand files that find command finds. So, instead of running a for loop to iterate over the output of find command, I'm trying to do an xargs over find output.
I am unable to work out how to pass the output of find command as parameter to the script.
find "$DIRLOC" -iname "*.html" |
xargs python3 src/shell/updates/python/updateshere.py <filename from find command's output to go here> "$HASH" {} \;
Please could someone help me with this?
You can do it like this:
find "$DIRLOC" -iname "*.html" -print0 |
xargs -0 -I {} python3 src/shell/updates/python/updateshere.py {} "$HASH"
Used -print0 option in find and -0 in xargs to handle filename/directory names with space or other special characters.
You can handle all in find also using -exec option:
find "$DIRLOC" -iname "*.html" -exec \
python3 src/shell/updates/python/updateshere.py {} "$HASH" \;
I am using find to list files within multiple directories with a specific extension. I tried
find /path/to/encompassing/directory/ -d -name "*modified.tif" | xargs cp Destination_Directory/
but it didn't work. Using
find /path/ -d -name "*modified.tif" -type f -exec cp {} Destination_Directory \;
works but I don't understand why xargs isn't working.
If you write
find -name '*modified.tif' | xargs cp directory
then that's the same as writing
cp directory file1modified.tif file2modified.tif
(or whatever filenames matched), which is the wrong way around, because xargs by default appends arguments.
find -name '*modified.tif' -exec cp {} directory \;
is the same as
cp file1modified.tif directory
cp file2modified.tif directory
which is what you want.
You can achieve the same with xargs by using
xargs -I{} cp {} directory
to specify where in the command you want to use the argument, but that implies that only one file at a time will be copied (because -I implies -L1).
To avoid calling cp once per file, you can use the -t option for cp so the files to be copied can be appended to the end of the command (requires GNU cp):
find -name '*modified.tif' | xargs cp -t directory
which is equivalent to
cp -t directory file1modified.tif file2modified.tif
or better, taking care of blanks in filenames,
find -name '*modified.tif' -print0 | xargs -0 cp -t directory
Alternatively, without xargs:
find -name '*modified.tif' -exec cp -t directory {} +
where -exec {} + makes sure to invoke cp as few times as possible.
xargs passes each word from its standard input as the last argument to cp, not the first. As a result, you are trying to run the series of commands
cp Destination_Directory/ foo
cp Destination_Directory/ bar
# etc
If you are using GNU cp, you can fix this simply by using the -t option to specify that Destination_Directory is the target, rather than a source.
... | xargs cp -t Destination_Directory
# cp -t Destination_Directory foo
# cp -t Destination_Directory bar
# etc
You might be able to use the -I option in xargs to make it use the incoming file name as the first argument:
... | xargs -I '{}' cp '{}' Destination_Directory
however, this makes a lot of assumptions about the names find will produce. (No leading or trailing whitespace, and no newlines in the file names.) (For that matter, xargs without -I is treating each whitespace-delimited word from its input as a separate argument for a call to cp.) In general, you should not try to use the output of find programmatically. Stick with its -exec primary instead.
Your code
find /path/ -d -name "*modified.tif" -type f -exec cp {} Destination_Directory \;
is the right way to go. No shell is involved, so each file name is passed as is as the first argument to cp.
I don't use xargs, but I think it should work like this :
cp `find /path/to/encompassing/directory/ -d -name "*modified.tif"` Destination_Directory/
No need for a pipe then.
I have the following function defined in my .bashrc, but for some reason the --exclude-dir option is not excluding the .git directory. Can anyone see what I've done wrong? I'm using Ubuntu 13.10 if that helps.
function fif # find in files
{
pattern=${1?" Usage: fif <word_pattern> [files pattern]"};
files=${2:+"-iname \"$2\""};
grep "$pattern" --color -n -H -s $(find . $files -type f) --exclude-dir=.git --exclude="*.min.*"
return 0;
}
Make sure not to include a trailing slash when you specify the directory to exclude. For example:
Do this:
$ grep -r --exclude-dir=node_modules firebase .
NOT this:
$ grep -r --exclude-dir=node_modules/ firebase .
(This answer not applicable to OP, but may be helpful for others who find --exclude-dir not to be working -- it worked for me.)
Do a man grep on your system, and see what version you have. Your version of grep may not be able to use --exclude-dirs.
You're really better off using find to find the files you want, then use grep to parse them:
$ find . -name '.git' -type d -prune \
-o -name "*.min.*" -prune \
-o -type f -exec grep --color -n -H {} "$pattern" \;
I'm not a fan of the recursive grep. Its syntax has become bloated, and it's really unnecessary. We have a perfectly good tool for finding files that match a particular criteria, thank you.
In the find program, the -o separate out the various clauses. If a file has not been filtered out by a previous -prune clause, it is passed to the next one. Once you've pruned out all of the .git directories and all of the *.min.* files, you pass the results to the -exec clause that executes your grep command on that one file.
Some people prefer it this way:
$ find . -name '.git' -type d -prune \
-o -name "*.min.*" -prune \
-o -type f -print0 | xargs -0 grep --color -n -H "$pattern"
The -print0 prints out all of the found files separated by the NULL character. The xargs -0 will read in that list of files and pass them to the grep command. The -0 tells xargs that the file names are NULL separated and not whitespace separated. Some xargs will take --null instead of the -0 parameter.
This is the command I've been using for finding matches (queryString) in php files, in the current directory, with grep, case insensitive, and showing matching results in line:
find . -iname "*php" -exec grep -iH queryString {} \;
Is there a way to also pipe just the file name of the matches to another script?
I could probably run the -exec command twice, but that seems inefficient.
What I'd love to do on Mac OS X is then actually to "reveal" that file in the finder. I think I can handle that part. If I had to give up the inline matches and just let grep show the files names, and then pipe that to a third script, that would be fine, too - I would settle.
But I'm actually not even sure how to pipe the output (the matched file names) to somewhere else...
Help! :)
Clarification
I'd like to reveal each of the files in a finder window - so I'm probably not going to using the -q flag and stop at the first one.
I'm going to run this in the console, ideally I'd like to see the inline matches printed out there, as well as being able to pipe them to another script, like oascript (applescript, to reveal them in the finder). That's why I have been using -H - because I like to see both the file name and the match.
If I had to settle for just using -l so that the file name could more easily be piped to another script, that would be OK, too. But I think after looking at the reply below from #Charlie Martin, that xargs could be helpful here in doing both at the same time with a single find, and single grep command.
I did say bash but I don't really mind if this needs to be ran as /bin/sh instead - I don't know too much about the differences yet, but I do know there are some important ones.
Thank you all for the responses, I'm going to try some of them at the command line and see if I can get any of them to work and then I think I can choose the best answer. Leave a comment if you want me to clarify anything more.
Thanks again!
You bet. The usual thing is something like
$ find /path -name pattern -print | xargs command
So you might for example do
$ find . -name '*.[ch]' -print | xargs grep -H 'main'
(Quiz: why -H?)
You can carry on with this farther; for example. you might use
$ find . -name '*.[ch]' -print | xargs grep -H 'main' | cut -d ':' -f 1
to get the vector of file names for files that contain 'main', or
$ find . -name '*.[ch]' -print | xargs grep -H 'main' | cut -d ':' -f 1 |
xargs growlnotify -
to have each name become a Growl notification.
You could also do
$ grep pattern `find /path -name pattern`
or
$ grep pattern $(find /path -name pattern)
(in bash(1) at least these are equivalent) but you can run into limits on the length of a command line that way.
Update
To answer your questions:
(1) You can do anything in bash you can do in sh. The one thing I've mentioned that would be any different is the use of $(command) in place of using backticks around command, and that works in the version of sh on Macs. The csh, zsh, ash, and fish are different.
(2) I think merely doing $ open $(dirname arg) will opena finder window on the containing directory.
It sounds like you want to open all *.php files that contain querystring from within a Terminal.app session.
You could do it this way:
find . -name '*.php' -exec grep -li 'querystring' {} \; | xargs open
With my setup, this opens MacVim with each file on a separate tab. YMMV.
Replace -H with -l and you will get a list of those filenames that matched the pattern.
if you have bash4, simply do
grep pattern /path/**/*.php
the ** operator is like
grep pattern `find -name \*.php -print`
find /home/aaronmcdaid/Code/ -name '*.cpp' -exec grep -q -iH boost {} \; -exec echo {} \;
The first change I made is to add -q to your grep command. This is "Exit immediately with zero status if any match is found".
The good news is that this speeds up grep when a file has many matching lines. You don't care how many matches there are. But that means we need another exec on the end to actually print the filenames when grep has been successful
The grep result will be sent to stdout, so another -exec predicate is probably the best solution here.
Pipe to another script:
find . -iname "*.php" | myScript
File names will come into the stdin of myScript 1 line at a time.
You can also use xargs to form/execute commands to act on each file:
find . -iname "*.php" | xargs ls -l
act on files you find that match:
find . -iname "*.php" | xargs grep -l pattern | myScript
act that don't match pattern
find . -iname "*.php" | xargs grep -L pattern | myScript
In general using multiple -exec's and grep -q will be FAR faster than piping, since find has implied short circuits -a's separating each juxtaposed pair of expressions that's not separated with an explicit operator. The main problem here, is that you want something to happen if grep matches something AND for matches to be printed. If the files are reasonably sized then this should be faster (because grep -q exits after finding a single match)
find . -iname "*php" -exec grep -iq queryString {} \; -exec grep -iH queryString {} \; -exec otherprogram {} \;
If the files are particularly big, encapsulating it in a shell script may be faster then running multiple grep commands
find . -iname "*php" -exec bash -c \
'out=$(grep -iH queryString "$1"); [[ -n $out ]] && echo "$out" && exit 0 || exit 1' \
bash {} \; -print
Also note, if the matches are not particularly needed, then
find . -iname "*php" -exec grep -iq queryString {} \; -exec otherprogram {} \;
Will virtually always be faster than then a piped solution like
find . -iname "*php" -print0 | xargs -0 grep -iH | ...
Additionally, you should really have -type f in all cases, unless you want to catch *php directories
Regarding the question of which is faster, and you actually care about the minuscule time difference, which maybe you might if you are trying to see which will save your processor some time... perhaps testing using the command as a suffix to the "time" command, and see which one performs better.
I'm trying to construct a find command to process a bunch of files in a directory using two different executables. Unfortunately, -exec on find doesn't allow to use pipe or even \| because the shell interprets that character first.
Here is specifically what I'm trying to do (which doesn't work because pipe ends the find command):
find /path/to/jpgs -type f -exec jhead -v {} | grep 123 \; -print
Try this
find /path/to/jpgs -type f -exec sh -c 'jhead -v {} | grep 123' \; -print
Alternatively you could try to embed your exec statement inside a sh script and then do:
find -exec some_script {} \;
A slightly different approach would be to use xargs:
find /path/to/jpgs -type f -print0 | xargs -0 jhead -v | grep 123
which I always found a bit easier to understand and to adapt (the -print0 and -0 arguments are necessary to cope with filenames containing blanks)
This might (not tested) be more effective than using -exec because it will pipe the list of files to xargs and xargs makes sure that the jhead commandline does not get too long.
With -exec you can only run a single executable with some arguments, not arbitrary shell commands. To circumvent this, you can use sh -c '<shell command>'.
Do note that the use of -exec is quite inefficient. For each file that is found, the command has to be executed again. It would be more efficient if you can avoid this. (For example, by moving the grep outside the -exec or piping the results of find to xargs as suggested by Palmin.)
Using find command for this type of a task is maybe not the best alternative. I use the following command frequently to find files that contain the requested information:
for i in dist/*.jar; do echo ">> $i"; jar -tf "$i" | grep BeanException; done
As this outputs a list would you not :
find /path/to/jpgs -type f -exec jhead -v {} \; | grep 123
or
find /path/to/jpgs -type f -print -exec jhead -v {} \; | grep 123
Put your grep on the results of the find -exec.
There is kind of another way you can do it but it is also pretty ghetto.
Using the shell option extquote you can do something similar to this in order to make find exec stuff and then pipe it to sh.
root#ifrit findtest # find -type f -exec echo ls $"|" cat \;|sh
filename
root#ifrit findtest # find -type f -exec echo ls $"|" cat $"|" xargs cat\;|sh
h
I just figured I'd add that because at least the way i visualized it, it was closer to the OP's original question of using pipes within exec.