Is there a simple function in GNU make to concatenate text and put an "operator" between single parts? I mean, the operator token must occurr n-1 times, only between two tokens.
Example: I have a variable with a list of paths to exclude from compilation:
EXCLUDE_PATH = $(BASE_DIR)/aaa* $(BASE_DIR)/bbb*
Later I want to use that variable to generate an exclusion rule for find command, like:
-not \( -path '$(BASE_DIR)/aaa*' -o -path '$(BASE_DIR)/bbb*' \)
Notice the -o between them.
A first try:
FIND_EXCL_LIST = $(EXCLUDE_PATH:%=-path '%' -o)
FIND_EXCL_RULE = -not \( $(FIND_EXCL_LIST) \)
But this leaves an extra -o at the end, resulting in an error.
Currently I resorted to add a -false at the end, as a stub OR parameter.
Provided that I prefer to decouple variable definitions from their specific uses, is there possibly a simpler way to do that with predefined functions?
Louis is correct: there's no magical way of doing this. But if, instead of trying to remove the last item or treat it specially, you instead treat the first item specially, the work is much simpler to understand (I think); you can create a function combine that takes two arguments: the first is the text to put between the words and the second is a space-separated list of words to combine:
combine = $(word 1,$2) $(foreach W,$(wordlist 2,$(words $2),$2),$1 $W)
Now you can use it like this:
$(call combine,-o,$(EXCLUDE_LIST))
to get output like:
$(BASE_DIR)/aaa* -o $(BASE_DIR)/bbb*
Related
My apologies if this is already answered, but if it is I can't find the proper search terms.
I'm trying to dynamically define search terms (based on user defined settings) for the find command:
# reading user settings results in array of filename patterns to delete:
patterns=("*.url" "*.html")
for i in ${patterns[#]}; do
find . -iname $i -delete
done
If I echo the command, the resulting string looks correct, e.g.
find . -iname "*.url" -delete
find . -iname "*.html" -delete
I know I'm missing something obvious but nothing I've tried works.
I'm using Bash 4.4.5 if that helps.
----------------EDIT-----------------
My thanks to Charles Duffy and l'L'l for the correct solution(s). I had a hard time wrapping my head around the quotes in the array strings vs the quoted variables and failed to quote both variables at the same time.
Lesson learned: always quote shell variables.
The answer by l'L'l is a good one, but let's make it a little more efficient, by invoking find only once with all your patterns passed in a single command line:
patterns=("*.url" "*.html")
find_pat=( )
for i in "${patterns[#]}"; do
find_pat+=( -o -name "$i" )
done
find . -iname '(' "${find_pat[#]:1}" ')' -delete
When run, this will invoke:
find . -iname '(' -name '*.url' -o -name '*.html' ')' -delete
...thus deleting all files which match either *.url or *.html in a single pass.
Note:
We're quoting our globs at all times: They're quoted on assignment (so we assign the globs themselves to the patterns array), on array expansion (so we iterate over the globs themselves and not their results), and on expansion into the find command (so we pass find the literal pattern syntax, not the result of expanding that syntax).
We're prepending -o to the find_pat array, but then expanding from the second element (arrays being 0-indexed), thus skipping that initial -o. The syntax used here is parameter expansion.
You need to double-quote your variables:
for i in "${patterns[#]}"; do
find . -iname "$i" -delete
...
This will prevent globbing and word splitting.
You can always check your script at https://www.shellcheck.net/ for errors as well...
I'm running macOS and looking for a way to quickly sort thousands of jpg files. I need to create folders based on part of filenames and then move those files into it.
Simply, I want to put these files:
x_not_relevant_part_of_name.jpg
x_not_relevant_part_of_name.jpg
y_not_relevant_part_of_name.jpg
y_not_relevant_part_of_name.jpg
Into these folders:
x
y
Keep in mind that length of "x" and "y" part of name may be different.
Is there an automatic solution for that in maxOS?
I've tried using Automator and Terminal but i'm not a programmer so I haven't done well.
I would back up the files first to somewhere safe in case it all goes wrong. Then I would install homebrew and then install rename with:
brew install rename
Then you can do what you want with this:
rename --dry-run -p 's|(^[^_]*)|$1/$1|' *.jpg
If that looks correct, remove the --dry-run and run it again.
Let's look at that command.
--dry-run means just say what the command would do without actually doing anything
-p means create any intermediate paths (i.e. directories) as necessary
's|...|' I will explain in a moment
*.jpg means to run the command on all JPG files.
The funny bit in single quotes is actually a substitution, in its simplest form it is s|a|b| which means substitute thing a with b. In this particular case, the a is caret (^) which means start of filename and then [^_]* means any number of things that are not underscores. As I have surrounded that with parentheses, I can refer back to it in the b part as $1 since it is the first thing in parentheses in a. The b part means "whatever was before the underscore" followed by a slash and "whatever was before the underscore again".
Using find with bash Parameter Substitution in Terminal would likely work:
find . -type f -name "*jpg" -maxdepth 1 -exec bash -c 'mkdir -p "${0%%_*}"' {} \; \
-exec bash -c 'mv "$0" "${0%%_*}"' {} \;
This uses bash Parameter Substitution with find to recursively create directories (if they don't already exist) using the prefix of any filenames matching jpg. It takes the characters before the first underscore (_), then moves the matching files into the appropriate directory. To use the command simply cd into the directory you would like to organize. Keep in mind that without using the maxdepth option running the command multiple times can produce more folders; limit the "depth" at which the command can operate using the maxdepth option.
${parameter%word}
${parameter%%word}
The word is expanded to produce a pattern just as in filename expansion. If the pattern matches a trailing portion of the expanded
value of parameter, then the result of the expansion is the value of
parameter with the shortest matching pattern (the ‘%’ case) or the
longest matching pattern (the ‘%%’ case) deleted.
↳ GNU Bash : Shell Parameter Expansion
Background
What I'm trying to do is exclude all submodules of a git repository in a find command. I know that I can exclude a single directory like this:
find . -not -path './$submodule/*'
So I built a command that generates a lot of these statements and stores them:
EXCLUDES=$(for submodule in $(git submodule status | awk '{print $2}'); do
echo -n "-not -path './$submodule/*' ";
done)
Problem
But when I run find . $EXCLUDES, this does not work. I suspect this is because of a bash quoting strategy that I do not understand. For example, lets assume (# marks output):
tree .
# .
# ├── bar
# │ └── baz.scala
# └── foo.scala
set -x
EXCLUDES="-not -path './bar/*'"
find . -type f $EXCLUDES
# + find . -not -path ''\''./bar/*'\''' <---- weird stuff
# foo.scala
# baz.scala
find . -type f -not -path './bar/*'
# + find . -type f -not -path './bar/*'
# foo.scala
How do I tell bash not to to the weird quoting stuff its doing (see marked line above)?
Edit: #eddiem suggested using git ls-files, which I will do in this concrete case. But I'm still interested in how I'd do this in the general case where I have a variable with quotes and would like to use it as arguments to a command.
The "weird stuff" you note is because bash only expands $EXCLUDES once, by substituting in the value you stored in EXCLUDES. It does not recursively process the contents of EXCLUDES to remove single-quotes like it does when you specify the quoted string on the command line. Instead, bash escapes special characters in $EXCLUDES, assuming that you want them there:
-not -path './bar/*'
becomes
-not -path ''\''./bar/*'\'''
^^ ^^ escaped single quotes
^^ ^^ random empty strings I'm actually not sure about
^ ^ single quotes around the rest of your text.
So, as #Jean-FrançoisFabre said, if you leave off the single quotes in EXCLUDES=..., you won't get the weird stuff.
So why isn't the first find working as expected? Because bash expands $EXCLUDES into a single word, i.e., a single element of argv that gets passed to find.* However, find expects its arguments to be separate words. As a result, find does not do what you expect.
The most reliable way I know of to do this sort of thing is to use an array:
declare -a EXCLUDES #make a new array
EXCLUDES+=("-not" "-path" './bar/*')
# single-quotes ^ ^ so we don't glob when creating the array
and you can repeat the += line any number of times for exclusions that you want. Then, to use these:
find . -type f "${EXCLUDES[#]}"
The "${name[#]}" form, with all that punctuation, expands each element of the array to a separate word, but does not further expand those words. So ./bar/* will stay as that and not be globbed. (If you do want globbing, find . -type f ${EXCLUDES[#]} (without the "") will expand each element of the array.)
Edit By the way, to see what's in your array, do set|grep EXCLUDES. You will each each element listed separately. You can also do echo "${EXCLUDES[#]}", but I find that less useful for debugging since it doesn't show the indices.
* see the "expansion" section of the man page. "Parameter expansion," expanding things that start with $, cannot change the number of words on the command line — except for "$#" and "${name[#]}".
The GNU make manual does not excel at explaining this part, I could not find the explanation or I could not infer the information elsewhere.
I realize % is a kind of wildcard, but what is the difference between % and * in the context of targets, dependencies and commands? Where can I use it and does it have the same meaning everywhere?
target: dependencies ...
commands
The wildcard character * is used to simply generate a list of matching files in the current directory. The pattern substitution character % is a placeholder for a file which may or may not exist at the moment.
To expand on the Wildcard pitfall example from the manual which you had already discovered,
objects = *.o
The proper way to phrase that if there no '.o' files is something like
objects := $(patsubst %.c,%.o,$(wildcard *.c))
make itself performs no wildcard expansion in this context, but of course, if you pass the literal value *.o to the shell, that's when expansion happens (if there are matches) and so this can be slightly hard to debug. make will perform wildcard expansion in the target of a rule, so you can say
foo: *.o
and have it work exactly like you intended (provided the required files are guaranteed to exist at the time this dependency is evaluated).
By contrast, you can have a rule with a pattern placeholder, which gets filled in with any matching name as make tries to find a recipe which can be used to generate a required dependency. There are built-in rules like
%.o: %.c
$(CC) $(CCFLAGS) $^ -o $#
(approximating the real thing here) which say "given a file matching %.c, the corresponding file %.o can be generated as follows." Here, the % is a placeholder which can be replaced by anything; so if it is applied against an existing file foo.c it says how foo.o can be generated.
You could rephrase it to say * matches every matching file while % matches any matching file.
Both % and * are ordinary characters in Make recipe lines; they are just passed to the shell.
% denotes a file "stem" in pattern substitutions, as in $(patsubst %.o,%.c,$(OBJS)). The pattern %.o is applied to each element in $(OBJS), and % captures the matching part. Then in the replacement pattern %.c, the captured part is substituted for the %, and a list of the substitutions emerges out of patsubst as the return value.
* is useful in the argument of the $(wildcard ...) operator, where it resembles the action of the shell * glob in matching some paths in the filesystem.
On the left hand side of a patsubst, where % denotes a match, it resembles * in that it matches some characters. However, % carries some restrictions, such as that it can only appear once! For instance whereas we can expand the wildcard */*.c, of course, we cannot have a double stem pattern substitution like $(patsubst %/%.o,%/foo/%.c,...). This restriction could be lifted in some future version of GNU Make, but it currently holds as far as I know.
Also there is a subtle difference between % and * in that % matches a nonempty sequence of characters. The wildcard pattern fo*o.c matches foo.c. The substitution pattern fo%o.c does not match foo.c, because then the stem % would be empty which is not allowed.
The following command works as expected interactively, in a terminal.
$ find . -name '*.foo' -o -name '*.bar'
./a.foo
./b.bar
$
However, if I do this, I get no results!
$ ftypes="-name '*.foo' -o -name '*.bar'"
$ echo $ftypes
-name '*.foo' -o -name '*.bar'
$ find . $ftypes
$
My understanding was/is that $ftypes would get expanded by bash before find got a chance to run. In which case, the ftypes approach should also have worked.
What is going on here?
Many thanks in advance.
PS: I have a need to dynamically build a list of file types (the ftypes variable above) to be given to find later in a script.
Both answers so far have recommended using eval, but that has a well-deserved reputation for causing bugs. Here's an example of the sort of bizarre behavior you can get with this:
$ touch a.foo b.bar "'wibble.foo'"
$ ftypes="-name '*.foo' -o -name '*.bar'"
$ eval find . $ftypes
./b.bar
Why didn't it find the file ./a.foo? It's because of exactly how that eval command got parsed. bash's parsing goes something like this (with some irrelevant steps left out):
bash looks for quotes first (none found -- yet).
bash substitutes variables (but doesn't go back and look for quotes in the substituted values -- this is what lead to the problem in the first place).
bash does wildcard matching (in this case it looks for files matching '*.foo' and '*.bar' -- note that it hasn't parsed the quotes, so it just treats them as part of the filename to match -- and finds 'wibble.foo' and substitutes it for '*.foo'). After this the command is roughly eval find . -name "'wibble.foo'" -o "'*.bar'". BTW, if it had found multiple matches things would've gotten even sillier by the end.
bash sees that the command on the line is eval, and runs the whole parsing process over on the rest of the line.
bash does quote matching again, this time finding two single-quoted strings (so it'll skip most parsing on those parts of the command).
bash looks for variables to substitute and wildcards to matching, etc, but there aren't any in the unquoted sections of the command.
Finally, bash runs find, passing it the arguments ".", "-name", "wibble.foo", "-o", "-name", and "*.bar".
find finds one match for "*.bar", but no match for "wibble.foo". It never even knows you wanted it to look for "*.foo".
So what can you do about this? Well, in this particular case adding strategic double-quotes (eval "find . $ftypes") would prevent the spurious wildcard substitution, but in general it's best to avoid eval entirely. When you need to build commands, an array is a much better way to go (see BashFAQ #050 for more discussion):
$ ftypes=(-name '*.foo' -o -name '*.bar')
$ find . "${ftypes[#]}"
./a.foo
./b.bar
Note that you can also build the options bit by bit:
$ ftypes=(-name '*.foo')
$ ftypes+=(-o -name '*.bar')
$ ftypes+=(-o -name '*.baz')
Simply prefix the line with eval to force the shell to expand and parse the command:
eval find . $ftypes
Without the eval, the '*.foo' is passed on literally instead of just *.foo (that is, the ' are suddenly considered to be part of the filename, so find is looking for files that start with a single quote and have an extension of foo').
The problem is that since $ftypes a single quoted value, find does see it as a single argument.
One way around it is:
$ eval find . $ftypes