I have never been able to fully understand the -prune action of the find command. But in actuality at least some of my misunderstanding stems from the effect of omitting the '-print' expression.
From the 'find' man page..
"If the expression contains no actions other than -prune, -print is performed on all files for which the expression is true."
.. which I have always (for many years) taken to mean I can leave out '-print'.
However, as the following example illustrates, there is a difference between using '-print' and omitting '-print', at least when a '-prune' expression appears.
First of all, I have the following 8 directories under my working directory..
aqua/
aqua/blue/
blue/
blue/orange/
blue/red/
cyan/blue/
green/
green/yellow/
There are a total of 10 files in those 8 directories..
aqua/blue/config.txt
aqua/config.txt
blue/config.txt
blue/orange/config.txt
blue/red/config.txt
cyan/blue/config.txt
green/config.txt
green/test.log
green/yellow/config.txt
green/yellow/test.log
My goal is to use 'find' to display all regular files not having 'blue' as part of the file's path. There are five files matching this requirement.
This works as expected..
% find . -path '*blue*' -prune -o -type f -print
./green/test.log
./green/yellow/config.txt
./green/yellow/test.log
./green/config.txt
./aqua/config.txt
But when I leave out '-print' it returns not only the five desired files, but also any directory whose path name contains 'blue'..
% find . -path '*blue*' -prune -o -type f
./green/test.log
./green/yellow/config.txt
./green/yellow/test.log
./green/config.txt
./cyan/blue
./blue
./aqua/blue
./aqua/config.txt
So why are the three 'blue' directories displayed?
This can be significant because often I'm trying to prune out a directory structure that contains more than 50,000 files. When that path is processed my find command, especially if I'm doing an '-exec grep' to each file, can take a huge amount of time processing files for which I have absolutely no interest. I need to have confidence that find is not going into the pruned structure.
The implicit -print applies to the entire expression, not just the last part of it.
% find . \( -path '*blue*' -prune -o -type f \) -print
./green/test.log
./green/yellow/config.txt
./green/yellow/test.log
./green/config.txt
./cyan/blue
./blue
./aqua/blue
./aqua/config.txt
It's not decending into the pruned directories, but it is printing out the top level.
A slight modification:
$ find . ! \( -path '*blue*' -prune \) -type f
./green/test.log
./green/yellow/config.txt
./green/yellow/test.log
./green/config.txt
./aqua/config.txt
(with implicit -a) would lead to having the same behavior with and without -print.
Related
I have this find command:
find . -type f -not -path '**/.git/**' -not -path '**/node_modules/**' | xargs sed -i '' s/typescript-library-skeleton/xxx/g;
for some reason it's giving me these warnings/errors:
find: ./.git/objects/3c: No such file or directory
find: ./.git/objects/3f: No such file or directory
find: ./.git/objects/41: No such file or directory
I even tried using:
-not -path '**/.git/objects/**'
and got the same thing. Anybody know why the find is searching in the .git directory? Seems weird.
why is the find searching in the .git directory?
GNU find is clever and supports several optimizations over a naive implementation:
It can flip the order of -size +512b -name '*.txt' and check the name first, because querying the size will require a second syscall.
It can count the hard links of a directory to determine the number of subdirectories, and when it's seen all it no longers needs to check them for -type d or for recursing.
It can even rewrite (-B -or -C) -and -A so that if the checks are equally costly and free of side effects, the -A will be evaluated first, hoping to reject the file after 1 test instead of 2.
However, it is not yet clever enough to realize that -not -path '*/.git/*' means that if you find a directory .git then you don't even need to recurse into it because all files inside will fail to match.
Instead, it dutifully recurses, finds each file and matches it against the pattern as if it was a black box.
To explicitly tell it to skip a directory entirely, you can instead use -prune. See How to exclude a directory in find . command
Both more efficient and more correct would be to avoid the default -print action, change -not -path ... to -prune, and ensure that xargs is only used with NUL-delimited input:
find . -name .git -prune -o \
-name node_modules -prune -o \
-type f -print0 | xargs -0 sed -i '' s/typescript-library-skeleton/xxx/g '{}' +
Note the following points:
We use -prune to tell find to not even recurse down the undesired directories, rather than -not -path ... to tell it to discard names in those directories after they were found.
We put the -prunes before the -type f, so we're able to match directories for pruning.
We have an explicit action, not depending on the default -print. This is important because the default -print effectively has a set of parenthesis: find ... behaves like find '(' ... ')' -print, not like find ... -print, no if explicit action is given.
We use xargs only with the -0 argument enabling NUL-delimited input, and the -print0 action on the find side to generate a NUL-delimited list of names. NUL is the only character which cannot be present in an arbitrary file path (yes, newlines can be present) -- and thus the only character which is safe to use to separate paths. (If the -0 extension to xargs and the -print0 extension to find are not guaranteed to be available, use -exec sed -i '' ... {} + instead).
This question already has answers here:
How do I exclude a directory when using `find`?
(46 answers)
Closed 7 years ago.
I'm using the find command to get a list of folders where certain files are located. But because of a permission denied error for certain subdirectories, I want to exclude a certain subdirectory name.
I already tried these solutions I found here:
find /path/to/folders -path "*/noDuplicates" -prune -type f -name "fileName.txt"
find /path/to/folders ! -path "*/noDuplicates" -type f -name "fileName.txt"
And some variations for these commands (variations on the path name for example).
In the first case it won't find a folder at all, in the second case I get the error again, so I guess it still tries to access this directory. Does anyone know what I'm doing wrong or does anyone have a different solution for this?
To complement olivm's helpful answer and address the OP's puzzlement at the need for -o:
-prune, as every find primary (action or test, in GNU speak), returns a Boolean, and that Boolean is always true in the case of -prune.
Without explicit operators, primaries are implicitly connected with -a (-and), which, like its brethren -o (-or) performs short-circuiting Boolean logic.
-a has higher precedence than -o.
For a summary of all find concepts, see https://stackoverflow.com/a/29592349/45375
Thus, the accepted answer,
find . -path ./ignored_directory -prune -o -name fileName.txt -print
is equivalent to (parentheses are used to make the evaluation precedence explicit):
find . \( -path ./ignored_directory -a -prune \) \
-o \
\( -name fileName.txt -a -print \)
Since short-circuiting applies, this is evaluated as follows:
an input path matching ./ignored_directory causes -prune to be evaluated; since -prune always returns true, short-circuiting prevents the right side of the -o operator from being evaluated; in effect, nothing happens (the input path is ignored)
an input path NOT matching ./ignored_directory, instantly - again due to short-circuiting - continues evaluation on the right side of -o:
only if the filename part of the input path matches fileName.txt is the -print primary evaluated; in effect, only input paths whose filename matches fileName.txt are printed.
Edit: In spite of what I originally claimed here, -print IS needed on the right-hand side of -o here; without it, the implied -print would apply to the entire expression and thus also print for left-hand side matches; see below for background information.
By contrast, let's consider what mistakenly NOT using -o does:
find . -path ./ignored_directory -prune -name fileName.txt -print
This is equivalent to:
find . -path ./ignored_directory -a -prune -a -name fileName.txt -a -print
This will only print pruned paths (that also match the -name filter), because the -name and -print primaries are (implicitly) connected with logical ANDs;
in this specific case, since ./ignored_directory cannot also match fileName.txt, nothing is printed, but if -path's argument is a glob, it is possible to get output.
A word on find's implicit use of -print:
POSIX mandates that if a find command's expression as a WHOLE does NOT contain either
output-producing primaries, such as -print itself
primaries that execute something, such as -exec and -ok
(the example primaries given are exhaustive for the POSIX spec. of find, but real-world implementations such as GNU find and BSD find add others, such as the output-producing -print0 primary, and the executing -execdir primary)
that -print be applied implicitly, as if the expression had been specified as:
\( expression \) -print
This is convenient, because it allows you to write commands such as find ., without needing to append -print.
However, in certain situations an explicit -print is needed, as is the case here:
Let's say we didn't specify -print at the end of the accepted answer:
find . -path ./ignored_directory -prune -o -name fileName.txt
Since there's now no output-producing or executing primary in the expression, it is evaluated as:
find . \( -path ./ignored_directory -prune -o -name fileName.txt \) -print
This will NOT work as intended, as it will print paths if the entire parenthesized expression evaluates to true, which in this case mistakenly includes the pruned directory.
By contrast, by explicitly appending -print to the -o branch, paths are only printed if the right-hand side of the -o expression evaluates to true; using parentheses to make the logic clearer:
find . -path ./ignored_directory -prune -o \( -name fileName.txt -print \)
If, by contrast, the left-hand side is true, only -prune is executed, which produces no output (and since the overall expression contains a -print, -print is NOT implicitly applied).
Following my previous comment, this works on my Debian :
find . -path ./ignored_directory -prune -o -name fileName.txt -print
or
find /path/to/folder -path "*/ignored_directory" -prune -o -name fileName.txt -print
or
find /path/to/folder -name fileName.txt -not -path "*/ignored_directory/*"
The differences are nicely debated here
Edit (added behavior specification details)
Pruning all permission denied directories in find
Using gnufind.
Specification behavior details - in this solutions we want to:
exclude unreadable directories contents (prune them),
avoid "permission denied" errors coming from unreadable dierctory,
keep the other errors and return states, but
process all files (even unreadable files, if we can read their names)
The basic design pattern is:
find ... \( -readable -o -prune \) ...
Example
find /var/log/ \( -readable -o -prune \) -name "*.1"
\thanks{mklement0}
The problem is in the way find evaluates the expression you are passing to the -path option.
Instead, you should try something like:
find /path/to/folders ! -path "*noDuplicates*" -type f -name "fileName.txt"
For convenience I've created links to a rather large "static" folder inside all of my www\site1, www\site2, etc. folders.
From www\ I would like to find all files starting with test_ in all subdirectories, without recursing into static for all sites.
I have gnuwin32 installed, which includes GNU find version 4.2.20, but its symlink options don't seem aware of Windows junctions ("symlinks" created with mklink /j source target). The closest I've gotten is:
find . -path "*static*" -prune -o -type f -name "test_*"
which sort of works. It's a little unsatisfying since it's not very general, and it also returns all the static folders (but not their contents).
I thought
dir /s /b /a:-L test_*
would work, but that seems to only omit the actual junctions and not their subdirectories.
Is there a way to do this?
Why the pruned directory still gets printed
If you don't include an explicit action to your find it will imply a -print gets applied to the entire expression, so
find . -path "*static*" -prune -o -type f -name "test_*"
actually gets executed as
find . `\( \( -path "*static*" -a -prune \) -o \( -type f -name "test_*" \) \) -print
So if either the left side or the right side of the exp OR exp1 is true, find still prints the file. If you want left side to be omitted, you can just add an explicit action to the right side
find . -path "*static*" -prune -o -type f -name "test_*" -print
Following symlinks
find does not follow symlinks by default. Try adding -L, though I'm not sure if this will work with windows links
-L Follow symbolic links. When find examines or prints information about files, the information used
shall be taken from the properties of the file to which the link points, not from the link itself
(unless it is a broken symbolic link or find is unable to examine the file to which the link
points). Use of this option implies -noleaf. If you later use the -P option, -noleaf will still
be in effect. If -L is in effect and find discovers a symbolic link to a subdirectory during its
search, the subdirectory pointed to by the symbolic link will be searched.
When the -L option is in effect, the -type predicate will always match against the type of the
file that a symbolic link points to rather than the link itself (unless the symbolic link is
broken). Using -L causes the -lname and -ilname predicates always to return false.
I was reading the manual for the find command. As I was going down the list of options I was reading the following..
PRIMARIES
All primaries which take a numeric argument allow the number to be preceded
by a plus sign (``+'') or a minus sign (``-''). A preceding plus
sign means ``more than n'', a preceding minus sign means ``less than n''
and neither means ``exactly n''.
I was having a hard time understanding what that means. I was also trying to find out what are "Primaries" in Google and couldn't get a good answer.
Can anyone help me understand what this means?
From the man page, this is the list of primaries in OS X find:
-Bmin
-Bnewer
-Btime
-amin
-anewer
-atime
-cmin
-cnewer
-ctime
-d
-delete
-depth
-empty
-exec
-execdir
-flags
-fstype
-gid
-group
-ignore
-ilname
-iname
-inum
-ipath
-iregex
-iwholename
-links
-lname
-ls
-maxdepth
-mindepth
-mmin
-mnewer
-mount
-mtime
-name
-newer
-newerXY
-nogroup
-noignore_readdir_race
-noleaf
-nouser
-ok
-okdir
-path
-perm
-print
-print0
-prune
-regex
-samefile
-size
-type
-uid
-user
-wholename
From the beginning of the same man page (emphasis mine):
DESCRIPTION
The find utility recursively descends the directory tree for each path listed, evaluating an expression (composed
of the ``primaries'' and ``operands'' listed below) in terms of each file in the tree.
"Primary" is the term used by the find documentation for one of the building blocks of an expression used by find to filter its output.
The find command accepts two kinds of parameters, they have been named 'primaries' and 'operators' by the authors of find. Primaries are parameters that allow filtering which files you want find to find, while Operators are the parameters that allow combining the primaries.
In mathematics, a primary is the basic component in an arithmetic or logic expression.
There also is a third class of parameters, that have no name and that modify the directory hierarchy traversal behavior of find, and a forth class that define what action to take upon the found files (print, delete, etc.)
The GNU man page uses the word 'Test' instead of 'Primary'
When using the find command, why is it that the following will successfully ignore hidden directories (those starting with a period) while matching everything else:
find . -not \( -type d -name ".?*" -prune \)
but this will not match anything at all:
find . -not \( -type d -name ".*" -prune \)
The only difference is the question mark. Shouldn't the latter command likewise detect and exclude directories beginning with a period?
The latter command prunes everything because it prunes . - try these to see the difference:
$ ls -lad .*
.
..
.dotdir
$ ls -lad .?*
..
.dotdir
You see that in the second one, . isn't included because it is only one character long. The glob ".?*" includes only filenames that are at least two characters long (dot, plus any single character, non-optionally, plus any sequence of zero or more characters).
By the way, find is not a Bash command.
The latter command prunes . itself -- the directory you're running find against -- which is why it generates no results.