With GNU find, it is easy to pipe to xargs. A typical (useless) example:
find /var/log -name "*.log" | xargs dirname
This returns all the directory names containing some log file.
The same command with BSD find does not work, ending with:
usage: dirname path
That is xargs is unable to pass file list entries to dirname.
BSD find's manpage mentions the -exec and -execdir options, stating "This behaviour is similar to that of xargs(1)."
-exec utility [argument ...] {} +
Same as -exec, except that ``{}'' is replaced with as many pathnames as possible for
each invocation of utility. This behaviour is similar to that of xargs(1).
-execdir utility [argument ...] {} +
Same as -execdir, except that ``{}'' is replaced with as many pathnames as possible
for each invocation of utility. This behaviour is similar to that of xargs(1).
Each time I fall back on these two flags, I have to read the documentation again. I seem unable to remember their usage! Also, I am concerned with script portability across GNU/BSD systems, basically Linux, Open/FreeBSD, and MacOS.
Any way to pipe BSD find to xargs, or -exec is really the only option?
Both GNU and FreeBSD version of xargs support a way to pass the strings from stdin to the command as part of the -I flag. All you need to is
find /var/log -name "*.log" | xargs -I {} dirname -- "{}"
The GNU xargs page says about the flag as
-I replace-str
Replace occurrences of replace-str in the initial-arguments with names read from standard input.
This provides an alternate way than using -exec or -execdir. However, having said, that using -exec is not too complex for your case.
find /var/log -name "*.log" -type f -exec dirname "{}" \;
Related
I am trying to execute this command:
find ./ -type f -readable -writable -exec sed -i "s/A/B/g" {} \;
and I'm getting this error:
find: -readable: unknown primary or operator
I am trying to find and replace extensively, anything with the name 'A' to 'B', this will apply to file names and text inside of files.
What's the best way for me to execute this on Mac terminal?
The BSD version of find doesn't have the -readable or -writable primaries, but you can fake them using the test command:
find ./ -type f -exec test -r {} -a -w {} \; -exec sed -i "" "s/A/B/g" {} \;
There's a possible problem here in that test's syntax can be ambiguous with more than three arguments, so a compound test expression like this might be misparsed. I don't think this can be a problem with paths that begin with ./, but if you're worried about it you can use two separate tests:
find ./ -type f -exec test -r {} \; -exec -w {} \; -exec sed -i "" "s/A/B/g" {} \;
Also, note that I added a null argument to sed after -i. This is another BSD-vs-GNU thing. The BSD version of sed requires an argument (the extension to use for a backup) to the -i option, so in order to avoid making a backup you have to explicitly supply a blank. The GNU version, on the other hand, would be confused by sed -i .bak because it requires the argument to -i be directly attached to it (sed -i.bak). Wheee.
When using scp or rsync I often fail to deal with 'Argument list too long' error. When having to mv or rm, I have no problem to use find and xargs but I fail to understand how to use find and -exec despite all the SE posts on the subject. Consider the following issue...
I tried
$scp /Path/to/B* Me#137.92.4.152:/Path/to/
-bash: /usr/bin/scp: Argument list too long
So I tried
$find . -name "/Path/to/B*" -exec scp "{}" Me#137.92.4.152:/Path/to/ '\;'
find: -exec: no terminating ";" or "+"
so I tried
$find . -name "/Path/to/B*" -exec scp "{}" Me#137.92.4.152:/Path/to/ ';'
find: ./.gnupg: Permission denied
find: ./.subversion/auth: Permission denied
So I tried
$sudo find . -name "/Path/to/B*" -exec scp "{}" Me#137.92.4.152:/Path/to/ ';'
and nothing happen onces I enter my password
I am on Mac OSX version 10.11.3, Terminal version 2.6.1
R. Saban's helpful answer solves your primary problem:
-name only accepts a filename pattern, not a path pattern.
Alternatively, you could simply use the -path primary instead of the -name primary.
As for using as few invocations of scp as possible - each of which requires specifying a password by default:
As an alternative, consider bypassing the use of scp altogether, as suggested in Eric Renouf's helpful answer.
While find's -exec primary allows using terminator + in lieu of ; (which must be passed as ';' or \; to prevent the shell from interpreting ; as a command terminator) for passing as many filenames as will fit on a single command line (a built-in xargs, in a manner of speaking), this is NOT an option here, because use of + requires that placeholder {} come last on the command line, immediately before +.
However, since you're on macOS, you can use BSD xarg's nonstandard -J option for placing the placeholder anywhere on the command line, while still passing as many arguments as possible at once (using BSD find's nonstandard -print0 option in combination with xargs's nonstandard -0 option ensures that all filenames are passed as-is, even if they have embedded spaces, for instance):
find . -path "/Path/to/B*" -print0 | xargs -0 -J {} scp {} Me#137.92.4.152:/Path/to/
Now you will at most be prompted a few times: one prompt for every batch of arguments, as required to accommodate all arguments while observing the max. command-line length with the fewest number of calls possible.
EDIT after your update:
find "/Path/to" -maxdepth 1 -name "B*" -exec scp {} Me#137.92.4.152:/Path/to/ \;
A solution that wouldn't require multiple scp connections (and therefore password entries) would be to tar on one side and untar on the other like:
find /Path/to -maxdepth 1 -name 'B*' -print0 | tar -c --null -T - | ssh ME#137.92.4.152 tar -x -C /Path/to
assuming your version of find supports -print0 and the like. It works by printing out null terminated list of files from find and telling tar to read its list of files from stdin (-T -) treating the list as null terminated (--null) and create a new archive (-c). By default, tar will write to stdout.
So then we'll pipe that archive to an ssh command to the target host. That will read the output of the previous command on its stdin, so we'll use tar there to extract (-x) the archive into the given directory (-C /Path/to)
I need to recursively find and replace a string in my .cpp and .hpp files.
Looking at an answer to this question I've found the following command:
find /home/www -type f -print0 | xargs -0 sed -i 's/subdomainA.example.com/subdomainB.example.com/g'
Changing it to include my file type did not work - did not changed any single word:
find /myprojects -type f -name *.cpp -print0 | xargs -0 sed -i 's/previousword/newword/g'
Help appreciated.
Don't bother with xargs; use the -exec primary. (Split across two lines for readability.)
find /home/www -type f -name '*.cpp' \
-exec sed -i 's/previousword/newword/g' '{}' \;
chepner's helpful answer proposes the simpler and more efficient use of find's -exec action instead of piping to xargs.
Unless special xargs features are needed, this change is always worth making, and maps to xargs features as follows:
find ... -exec ... {} \; is equivalent to find ... -print0 | xargs -0 -n 1 ...
find ... -exec ... {} + is equivalent to find ... -print0 | xargs -0 ...
In other words:
the \; terminator invokes the target command once for each matching file/folder.
the + terminator invokes the target command once overall, supplying all matching file/folder paths as a single list of arguments.
Multiple calls happen only if the resulting command line becomes too long, which is rare, especially on Linux, where getconf ARG_MAX, the max. command-line length, is large.
Troubleshooting the OP's command:
Since the OP's xargs command passes all matching file paths at once - and per xargs defaults at the end of the command line, the resulting command will effectively look something like this:
sed -i 's/previousword/newword/g' /myprojects/file1.cpp /myprojects/file2.cpp ...
This can easily be verified by prepending echo to sed - though (conceptual) quoting of arguments that need it (paths with, e.g., embedded spaces) will not show (note the echo):
find /myprojects -type f -name '*.cpp' -print0 |
xargs -0 echo sed -i 's/previousword/newword/g'
Next, after running the actual command, check whether the last-modified date of the files has changed using stat:
If they have, yet the contents haven't changed, the implication is that sed has processed the files, but the regex in the s function call didn't match anything.
It is conceivable that older GNU sed versions don't work properly when combining -i (in-place editing) with multiple file operands (though I couldn't find anything in the GNU sed release notes).
To rule that out, invoke sed once for each file:
If you still want to use xargs, add -n 1:
find /myprojects -type f -name '*.cpp' -print0 |
xargs -0 -n 1 sed -i 's/previousword/newword/g'
To use find's -exec action, see chepner's answer.
With a GNU sed version that does support updating of multiple files with the -i option - which is the case as of at least v4.2.2 - the best formulation of your command is (note the quoted *.cpp argument to prevent premature expansion by the shell, and the use of terminator + to only invoke sed once):
find /myprojects -type f -name '*.cpp' -exec sed -i 's/previousword/newword/g' '{}' +
Can someone show me to use xargs properly? Or if not xargs, what unix command should I use?
I basically want to input more than (1) file name for input <localfile>, third input parameter.
For example:
1. use `find` to get list of files
2. use each filename as input to shell script
Usage of shell script:
test.sh <localdir> <localfile> <projectname>
My attempt, but not working:
find /share1/test -name '*.dat' | xargs ./test.sh /staging/data/project/ '{}' projectZ \;
Edit:
After some input from everybody and trying -exec, I am finding that my <localfile> filename input with find is also giving me the full path. /path/filename.dat instead of filename.dat. Is there a way to get the basename from find? I think this will have to be a separate question.
I'd just use find -exec here:
% find /share1/test -name '*.dat' -exec ./test.sh /staging/data/project/ {} projectZ \;
This will invoke ./test.sh with your three arguments once for each .dat file under /share1/test.
xargs would pack up all of these filenames and pass them into one invocation of ./test.sh, which doesn't look like your desired behaviour.
If you want to execute the shell script for each file (as opposed to execute in only once on the whole list of files), you may want to use find -exec:
find /share1/test -name '*.dat' -exec ./test.sh /staging/data/project/ '{}' projectZ \;
Remember:
find -exec is for when you want to run a command on one file, for each file.
xargs instead runs a command only once, using all the files as arguments.
xargs stuffs as many files as it can onto the end of the command line.
Do you want to execute the script on one file at a time or all files? For one at a time, use file's exec, which it looks like you're already using the syntax for, and which xargs doesn't use:
find /share1/test -name '*.dat' -exec ./test.sh /staging/data/project/ '{}' projectZ \;
xargs does not have to combine arguments, it's just the default behavior. this properly uses xargs, to execute the commands, as intended.
find /share1/test -name '*.dat' -print0 | xargs -0 -I'{}' ./test.sh /staging/data/project/ '{}' projectZ
When piping find to xargs, NULL termination is usually preferred, I recommend appending the -print0 option to find. After which you must add -0 to xargs, so it will expect NULL terminated arguments. This ensures proper handling of filenames. It's not POSIX proper, but considered well supported. You can always drop the NULL terminating options, if your commands lack support.
Remeber while find's purpose is finding files, xargs is much more generic. I often use xargs to process non-filename arguments.
I'm trying to construct a find command to process a bunch of files in a directory using two different executables. Unfortunately, -exec on find doesn't allow to use pipe or even \| because the shell interprets that character first.
Here is specifically what I'm trying to do (which doesn't work because pipe ends the find command):
find /path/to/jpgs -type f -exec jhead -v {} | grep 123 \; -print
Try this
find /path/to/jpgs -type f -exec sh -c 'jhead -v {} | grep 123' \; -print
Alternatively you could try to embed your exec statement inside a sh script and then do:
find -exec some_script {} \;
A slightly different approach would be to use xargs:
find /path/to/jpgs -type f -print0 | xargs -0 jhead -v | grep 123
which I always found a bit easier to understand and to adapt (the -print0 and -0 arguments are necessary to cope with filenames containing blanks)
This might (not tested) be more effective than using -exec because it will pipe the list of files to xargs and xargs makes sure that the jhead commandline does not get too long.
With -exec you can only run a single executable with some arguments, not arbitrary shell commands. To circumvent this, you can use sh -c '<shell command>'.
Do note that the use of -exec is quite inefficient. For each file that is found, the command has to be executed again. It would be more efficient if you can avoid this. (For example, by moving the grep outside the -exec or piping the results of find to xargs as suggested by Palmin.)
Using find command for this type of a task is maybe not the best alternative. I use the following command frequently to find files that contain the requested information:
for i in dist/*.jar; do echo ">> $i"; jar -tf "$i" | grep BeanException; done
As this outputs a list would you not :
find /path/to/jpgs -type f -exec jhead -v {} \; | grep 123
or
find /path/to/jpgs -type f -print -exec jhead -v {} \; | grep 123
Put your grep on the results of the find -exec.
There is kind of another way you can do it but it is also pretty ghetto.
Using the shell option extquote you can do something similar to this in order to make find exec stuff and then pipe it to sh.
root#ifrit findtest # find -type f -exec echo ls $"|" cat \;|sh
filename
root#ifrit findtest # find -type f -exec echo ls $"|" cat $"|" xargs cat\;|sh
h
I just figured I'd add that because at least the way i visualized it, it was closer to the OP's original question of using pipes within exec.