The following command works as expected interactively, in a terminal.
$ find . -name '*.foo' -o -name '*.bar'
./a.foo
./b.bar
$
However, if I do this, I get no results!
$ ftypes="-name '*.foo' -o -name '*.bar'"
$ echo $ftypes
-name '*.foo' -o -name '*.bar'
$ find . $ftypes
$
My understanding was/is that $ftypes would get expanded by bash before find got a chance to run. In which case, the ftypes approach should also have worked.
What is going on here?
Many thanks in advance.
PS: I have a need to dynamically build a list of file types (the ftypes variable above) to be given to find later in a script.
Both answers so far have recommended using eval, but that has a well-deserved reputation for causing bugs. Here's an example of the sort of bizarre behavior you can get with this:
$ touch a.foo b.bar "'wibble.foo'"
$ ftypes="-name '*.foo' -o -name '*.bar'"
$ eval find . $ftypes
./b.bar
Why didn't it find the file ./a.foo? It's because of exactly how that eval command got parsed. bash's parsing goes something like this (with some irrelevant steps left out):
bash looks for quotes first (none found -- yet).
bash substitutes variables (but doesn't go back and look for quotes in the substituted values -- this is what lead to the problem in the first place).
bash does wildcard matching (in this case it looks for files matching '*.foo' and '*.bar' -- note that it hasn't parsed the quotes, so it just treats them as part of the filename to match -- and finds 'wibble.foo' and substitutes it for '*.foo'). After this the command is roughly eval find . -name "'wibble.foo'" -o "'*.bar'". BTW, if it had found multiple matches things would've gotten even sillier by the end.
bash sees that the command on the line is eval, and runs the whole parsing process over on the rest of the line.
bash does quote matching again, this time finding two single-quoted strings (so it'll skip most parsing on those parts of the command).
bash looks for variables to substitute and wildcards to matching, etc, but there aren't any in the unquoted sections of the command.
Finally, bash runs find, passing it the arguments ".", "-name", "wibble.foo", "-o", "-name", and "*.bar".
find finds one match for "*.bar", but no match for "wibble.foo". It never even knows you wanted it to look for "*.foo".
So what can you do about this? Well, in this particular case adding strategic double-quotes (eval "find . $ftypes") would prevent the spurious wildcard substitution, but in general it's best to avoid eval entirely. When you need to build commands, an array is a much better way to go (see BashFAQ #050 for more discussion):
$ ftypes=(-name '*.foo' -o -name '*.bar')
$ find . "${ftypes[#]}"
./a.foo
./b.bar
Note that you can also build the options bit by bit:
$ ftypes=(-name '*.foo')
$ ftypes+=(-o -name '*.bar')
$ ftypes+=(-o -name '*.baz')
Simply prefix the line with eval to force the shell to expand and parse the command:
eval find . $ftypes
Without the eval, the '*.foo' is passed on literally instead of just *.foo (that is, the ' are suddenly considered to be part of the filename, so find is looking for files that start with a single quote and have an extension of foo').
The problem is that since $ftypes a single quoted value, find does see it as a single argument.
One way around it is:
$ eval find . $ftypes
Related
I have a ton of screenshots on my desktop (it's the default location where they're saved) with titles of the form "Screen Shot 2020-10-11 at 7.08.12 PM.png" and I'd like to use a bash script with regex on the "Screen Shot" bit to move any such file from my desktop into a folder I created called "screenshots".
Right now, I'm playing around with find . -regex Screen*.*?.png but it's not quite working (it gives find: Screen Shot 2020-10-11 at 7.11.09 PM.png: unknown primary or operator).
I'm also not sure how I'd even use the output once it does find all the correct files to move them to a folder. Could you somehow iterate over all files in a folder using for i in seq 1 100 or something of the like?
You don't actually even need -regex here:
find . -type f -name 'Screen Shot*png' -maxdepth 1 -exec echo mv "{}" screenshots \;
You can run this command safely as it will not do anything but print
what it would do. Remove echo to actually run mv.
All options used are documented in man find but in short:
-type f will make find look only for files, not directories. This
is useful in case you have a directory that matches -name - we don't
want to touch it.
-maxdepth 1 will only look fire files in the same directory level -
it's very useful here because you might already have some files that
match the -name present in screenshots directory - we want to leave
them alone.
-name accepts shell pattern, not regex. We could of course use -regex here but I prefer -name because shell patterns are shorter and easier to use here.
{} is a placeholder that will be replaced will the name of found
file.
\; is a literal semicolon, escaped to prevent it from being
interpreted by shell that ends command specified with -exec.
Taking the regex at face value (probably a mistake), you should use single quotes around the regex:
find . -regex 'Screen*.*?.png'
This prevents the shell from expanding it, leaving that to find. Then to move the files to the ~/screenshots directory (change the name to match the directory you want to use), if you have GNU mv, you can use:
find . -regex 'Screen*.*?.png' -exec mv -t ~/screenshots {} +
This executes a single mv command to move many files to the target directory, reducing the number of times the mv is executed. It might still be executed multiple times, but it will be many fewer times than the alternative.
If you don't have GNU mv with the (very useful) -t option, then you should use:
find . -regex 'Screen*.*?.png' -exec mv {} ~/screenshots ';'
This executes one mv command for each file found, but is more portable.
The primary problem you ran into was that the shell was expanding what you wrote into a list of file names, and then find didn't understand what you meant. Using the quotes prevents the shell from expanding the 'regex'. You can add an echo to the other commands before the mv to see what would be executed.
However, I'm not sure whether you know what your regex matches. It isn't clear that the regex given is a valid regex for find — though it mostly works as a PCRE (Perl-compatible) regular expression. By default, GNU find uses GNU Emacs regular expressions, but you can control the dialect of regular expression it uses. The options available include Emacs, POSIX Awk, POSIX Basic Regular Expressions (BRE), POSIX egrep, and POSIX Extended Regular Expressions (ERE). It doesn't include PCRE. What you supply is more like a shell glob, and the -name operator handles globbing names.
It's quite probable that you should be using the -name operator, using a command along the lines of:
find . -name 'Screen Shot *.png' -exec mv -t ~/screenshots {} +
I want to make script that searches,shows and delete for names ending in "*". My command:
echo rm `find -name "*[*]"`
Command works ,but I create file: something and something(ended star) Now after write command it, shows me : rm something(ended star) and similar file "something"
Why?
As Stefan Hamcke states in comments, this is because the wildcard (*) from find's result ("something*") is passed as argument to echo and ends up being expanded again, resulting in the final output having both something and something*.
Do this instead:
find . -name "*[*]" -exec echo rm {} +
Output:
rm ./something*
You can also achieve the same with the expression "*\*" in find.
The bug is that you do not quote the argument to echo. Here is a trivial fix:
echo "rm $(find -name "*[*]")"
This is not entirely a minimal fix because I also replaced the obsolescent `backtick` syntax with the modern, recommended $(command substitution) syntax.
Without quotes, the string returned from the command substition gets evaluated by the shell for token splitting and wildcard expansion. For details, see When to wrap quotes around a shell variable?
My apologies if this is already answered, but if it is I can't find the proper search terms.
I'm trying to dynamically define search terms (based on user defined settings) for the find command:
# reading user settings results in array of filename patterns to delete:
patterns=("*.url" "*.html")
for i in ${patterns[#]}; do
find . -iname $i -delete
done
If I echo the command, the resulting string looks correct, e.g.
find . -iname "*.url" -delete
find . -iname "*.html" -delete
I know I'm missing something obvious but nothing I've tried works.
I'm using Bash 4.4.5 if that helps.
----------------EDIT-----------------
My thanks to Charles Duffy and l'L'l for the correct solution(s). I had a hard time wrapping my head around the quotes in the array strings vs the quoted variables and failed to quote both variables at the same time.
Lesson learned: always quote shell variables.
The answer by l'L'l is a good one, but let's make it a little more efficient, by invoking find only once with all your patterns passed in a single command line:
patterns=("*.url" "*.html")
find_pat=( )
for i in "${patterns[#]}"; do
find_pat+=( -o -name "$i" )
done
find . -iname '(' "${find_pat[#]:1}" ')' -delete
When run, this will invoke:
find . -iname '(' -name '*.url' -o -name '*.html' ')' -delete
...thus deleting all files which match either *.url or *.html in a single pass.
Note:
We're quoting our globs at all times: They're quoted on assignment (so we assign the globs themselves to the patterns array), on array expansion (so we iterate over the globs themselves and not their results), and on expansion into the find command (so we pass find the literal pattern syntax, not the result of expanding that syntax).
We're prepending -o to the find_pat array, but then expanding from the second element (arrays being 0-indexed), thus skipping that initial -o. The syntax used here is parameter expansion.
You need to double-quote your variables:
for i in "${patterns[#]}"; do
find . -iname "$i" -delete
...
This will prevent globbing and word splitting.
You can always check your script at https://www.shellcheck.net/ for errors as well...
I'm running macOS and looking for a way to quickly sort thousands of jpg files. I need to create folders based on part of filenames and then move those files into it.
Simply, I want to put these files:
x_not_relevant_part_of_name.jpg
x_not_relevant_part_of_name.jpg
y_not_relevant_part_of_name.jpg
y_not_relevant_part_of_name.jpg
Into these folders:
x
y
Keep in mind that length of "x" and "y" part of name may be different.
Is there an automatic solution for that in maxOS?
I've tried using Automator and Terminal but i'm not a programmer so I haven't done well.
I would back up the files first to somewhere safe in case it all goes wrong. Then I would install homebrew and then install rename with:
brew install rename
Then you can do what you want with this:
rename --dry-run -p 's|(^[^_]*)|$1/$1|' *.jpg
If that looks correct, remove the --dry-run and run it again.
Let's look at that command.
--dry-run means just say what the command would do without actually doing anything
-p means create any intermediate paths (i.e. directories) as necessary
's|...|' I will explain in a moment
*.jpg means to run the command on all JPG files.
The funny bit in single quotes is actually a substitution, in its simplest form it is s|a|b| which means substitute thing a with b. In this particular case, the a is caret (^) which means start of filename and then [^_]* means any number of things that are not underscores. As I have surrounded that with parentheses, I can refer back to it in the b part as $1 since it is the first thing in parentheses in a. The b part means "whatever was before the underscore" followed by a slash and "whatever was before the underscore again".
Using find with bash Parameter Substitution in Terminal would likely work:
find . -type f -name "*jpg" -maxdepth 1 -exec bash -c 'mkdir -p "${0%%_*}"' {} \; \
-exec bash -c 'mv "$0" "${0%%_*}"' {} \;
This uses bash Parameter Substitution with find to recursively create directories (if they don't already exist) using the prefix of any filenames matching jpg. It takes the characters before the first underscore (_), then moves the matching files into the appropriate directory. To use the command simply cd into the directory you would like to organize. Keep in mind that without using the maxdepth option running the command multiple times can produce more folders; limit the "depth" at which the command can operate using the maxdepth option.
${parameter%word}
${parameter%%word}
The word is expanded to produce a pattern just as in filename expansion. If the pattern matches a trailing portion of the expanded
value of parameter, then the result of the expansion is the value of
parameter with the shortest matching pattern (the ‘%’ case) or the
longest matching pattern (the ‘%%’ case) deleted.
↳ GNU Bash : Shell Parameter Expansion
I want to do something like the following:
#!/bin/bash
cmd="find . -name '*.sh'"
echo $($cmd)
What I expect is that it will show all the shell script files in the current directory, but nothing happened.
I know that I can solve the problem with eval according to this post
#!/bin/bash
cmd="find . -name '*.sh'"
eval $cmd
So my question is why command substitution doesn't work here and what's the difference between $(...) and eval in terms of the question?
Command substitution works here. Just you have wrong quoting. Your script find only one file name! This one with single quotes and asteriks in it:
'*.sh'
You can create such not usual file by this command and test it:
touch "'*.sh'"
Quoting in bash is different than in other programming languages. Check out details in this answer.
What you need is this quoting:
cmd="find . -name *.sh"
echo $($cmd)
Since you are already including the patter *.sh inside double quotes, there's no need for the single quotes to protect the pattern, and as a result the single quotes are part of the pattern.
You can try using an array to keep *.sh quoted until it is passed to the command substitution:
cmd=(find . -name '*.sh')
echo $("${cmd[#]}")
I don't know if there is a simple way to convert your original string to an array without the pattern being expanded.
Update: This isn't too bad, but it's probably better to just create the array directly if you can.
cmd="find . -name *.sh"
set -f
cmd=($cmd)
set +f
echo $("${cmd[#]}")
When you use the echo $($cmd) syntax, it's basically equivalent to just putting $cmd on it's own line. The problem is the way bash wants to interpolate the wildcard before the command runs. The way to protect against that is to put the variable containing the * char in quotes AGAIN when you dereference them in the script.
But if you put the whole command find . -name "*.sh" in a variable, then quote it with `echo $("$cmd"), the shell will interpret that to mean that the entire line is a file to execute, and you get a file not found error.
So it really depends on what you really need in the variable and what can be pulled out of it. If you need the program in the variable, this will work:
#!/bin/bash
cmd='/usr/bin/find'
$cmd . -name "*.sh" -maxdepth 1
This will find all the files in the current working directory that end in .sh without having the shell interpolate the wildcard.
If you need the pattern to be in a variable, you can use:
#!/bin/bash
pattern="*.sh"
/usr/bin/find . -name "$pattern" -maxdepth 1
But if you put the whole thing in a variable, you won't get what you expect. Hope this helps. If a bash guru knows something I'm missing I'd love to hear it.