What is the difference between % and * wildcards in Make? - makefile

I'm currently learning Make and am struggling to wrap my head around the wildcard concept. Specifically, it seems like there are two symbols that can represent wildcards: * and %
For example, say I want to to generate a variable that identifies all .c source files in the working directory. I would use the traditional * symbol in *.c However, if I use the patsubst function, I am required to use the % symbol instead of * symbol:
// WORKS
SRC = $(wildcard *.c) # list of source files
OBJS = $(patsubst %.c, %.o, $(SRC)) # list of object files
// DOES NOT WORK!!!!
SRC = $(wildcard *.c) # list of source files
OBJS = $(patsubst *.c, *.o, $(SRC)) # list of object files
Can someone explain the difference between * and % in the context of Make wildcards?

Can someone explain the difference between * and % in the context of Make wildcards?
TL;DR:
This is all specific to GNU make.
* and a few other characters are special to the wildcard function, but not to patsubst. This kind of pattern is expanded to the names of existing files and directories that match the pattern, possibly more than one for each pattern.
% is special to the patsubst function. This kind of pattern is used to select matching strings provided by the makefile or directly by make, in the process capturing the part matching the % for later use.
Both kinds have application to the target and prerequisite lists of rules, but their significance there is somewhat different from each other.
None of these are significant to make in recipes, but wildcard-style patterns are significant to the shell, and the shell will interpret them in recipes that it executes.
General
Understand first that $(wildcard) and $(patsubst) are features specific to GNU's implementation of make, and GNU make also attributes special significance to %, *, and a few other characters in rule targets and prerequisites. The POSIX specifications for make say nothing about any of that. GNU make is widely used these days, and with good reason, but it is not the only make you might encounter. If you want maximum portability among make implementations then you must avoid these altogether.
Understand also that a complete, albeit rather basic response to the question would be simply "yes, the wildcard and patsubst functions recognize different special characters." These functions do different things, so it is potentially useful that the special characters of one can be used as ordinary characters in the other.
Wildcards
The asterisk (*) is among the special characters recognized by the Bourne shell for "pathname expansion", which replaces patterns with the names of possibly-many existing files and directories matching the pattern. There are more characters than just * significant in pathname expansion, but % is not among them. Look up that term for a full description.
Additionally, there is the tilde (~), which make and some shells recognize for "tilde expansion", which involves interpreting the first segments of affected paths as specified users' home directories.
The GNU make documentation describes the characters and constructs it recognizes for pathname and tilde expansion as "wildcards", and it has this to say about them:
Wildcard expansion is performed by make automatically in targets and in prerequisites. In recipes, the shell is responsible for wildcard expansion. In other contexts, wildcard expansion happens only if you request it explicitly with the wildcard function.
(GNU make manual, section 4.4)
And that's where the wildcard function appearing in the question comes in -- if you want to perform the same wildcard expansion that make performs automatically on target and prerequisite names in some other context, such as a variable definition, then you can use the wildcard function to get it. Thus,
SRC = $(wildcard *.c)
results variable SRC representing a list of all the existing files in the working directory at the time the makefile is parsed whose names end with .c.
On the other hand, % is not significant for pathname or tilde expansion, so
SRC = $(wildcard %.c)
will expand to literally %.c.
Patterns
The shell documentation uses the term "pattern" to mean shell input that is interpreted to be subject to pathname expansion, rather than being literal. However, GNU make reserves the term for a different kind of pattern in which the % character features. This is relevant in two main areas: the patsubst function and pattern rules.
The patsubst function
The patsubst function computes one string or series of strings from another by replacing those that match a given pattern with a specified replacement. In a pattern, the first % character, if any, matches any number of characters, including zero. In this sense, it can be described as a wildcard, and the manual does, somewhat confusingly, use that term. If the replacement also contains at least one % then the first is replaced by the substring that was matched by the % in the pattern.
Note well that this has nothing inherently to do with file names, existing or otherwise.
Thus, with
SRC = main.c other.c
OBJ = $(patsubst %.c,%.o,$(SRC))
the %.c pattern matches first main.c (with % matching main) and then other.c (with % matching other), and the result for OBJ is the same as this:
OBJ = main.o other.o
regardless of whether any files named main.c, other.c, main.o, or other.o exist.
Pattern rules
If the target of a rule contains a %, then it is interpreted by GNU make as a pattern rule. Pattern rules use % in much the same way that patsubst does. When make needs to build a target but does not have a rule for that specific target, it will check whether the target name matches the target pattern of any pattern rule (including some built-in ones). There does not need to be any existing file of that name. If it finds a match, then it will use that rule to build the target. And in that case, for the purpose of building the target in question, the first % character, if any, in any prerequisite names specified in that rule will be replaced by the stem that matched the % in the target, much like in patsubst.
Combinations
Usually, only one of these kinds of pattern matching / expansion is desired in any given context, but sometimes they can usefully be combined. For example, consider this pattern rule:
%.o: %.c *.h
$(CC) $(CPPFLAGS) $(CFLAGS) -c -o $# $<
That says that any .o file can be built from a corresponding .c file and all the .h files in the working directory by applying the provided recipe.

The wildcard function expects wildcard characters (~, *, ? and [...] which match on file names) while the patsubst function expects a pattern (%) that operates on text. As such $(patsubst *.c, *.o, $(SRC)) means replace the literal string *.c with the string *.o and SRC probably does not contain any literal *.c strings.

Related

Makefile: reuse value of % of a pattern rule inside the recipe

In a Makefile I have:
images/schematic.pdf: images/schematic.svg
inkscape -D -z --file=$^ --export-pdf=$# --export-latex
sed -i "s:schematic:images/schematic:g" $#_tex
What this rule does is:
Use inkscape to convert to a latex-ready .pdf PDF file + its corresponding .pdf_tex text file (see this answer: https://tex.stackexchange.com/a/2107/104581)
Modify the mentioned .pdf_tex file so that it will not break latex compiling from a "higher" directory (namely . when the .pdf_tex file is in ./images)
My problem:
I have many rules in this form i. e. where only schematic is changed. I would like to use a pattern-rule that replaces schematic by %. And to use % in the recipe (in the sed command).
However the rule:
images/%.pdf: images/%.svg
inkscape -D -z --file=$^ --export-pdf=$# --export-latex
sed -i "s:%:images/%:g" $#_tex
does not works: % is interpreted literally in the recipe.
I also tried to replace % in the recipe by $% but this variable seems to be empty.
Unsatisfactory solution:
Add a line in the recipe to create a (make) variable that will hold the result of notdir(removeprefix($<)) (using this question or a call to bash because there is no removeprefix in GNU Make).
You want $*.
From https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html :
$*:
The stem with which an implicit rule matches (see How Patterns Match).
If the target is dir/a.foo.b and the target pattern is a.%.b
then the stem is dir/foo. The stem is useful for constructing names of
related files.
In a static pattern rule, the stem is part of the file name that matched the ‘%’ in the target pattern.
In an explicit rule, there is no stem; so ‘$*’ cannot be determined in that way. Instead, if the target name ends with a
recognized suffix (see Old-Fashioned Suffix Rules), ‘$*’ is set to the
target name minus the suffix. For example, if the target name is
‘foo.c’, then ‘$*’ is set to ‘foo’, since ‘.c’ is a suffix. GNU make
does this bizarre thing only for compatibility with other
implementations of make. You should generally avoid using ‘$*’ except
in implicit rules or static pattern rules.
If the target name in an explicit rule does not end with a recognized suffix, ‘$*’ is set to the empty string for that rule.
$% is an automatic variable, but it's for archive (.a library) members, and almost never useful.

GNU make - how to set an implicit pattern as a prerequisite

I have this implicit rule:
%.so: %.so.5
qnx_ln $< $#
I realized that for another target, I have to make all .so files the prerequisite for that target.
I tried this:
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs %.so
...
But I got the output
*** No rule to make target '%.so', needed by 'makegen'. Stop.
% prerequisite patterns can only be used in static and implicit pattern rules, where they match the respective % part of the target; when used in a regular rule % is a literal character.
You'll need to specify the dependencies literally, unless there is some correspondence between certain source filenames and the .so filenames that you can leverage, presumably you're already doing either of these to link the .so files in the first place.
As pointed out previously, no you can't do that because this is not how prerequisite patterns work. Maybe you gave the following a thought and rejected it but I suspect you might find the following a close-enough fit:
%.so.target: %.so.5
echo $< >> $(BUILD)/so.targets
SO_TARGETS=$(basename $(shell cat $(BUILD)/so.targets))
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs $(SO_TARGETS)
Maybe you are looking for a rule to match on every existing *.so file?
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs $(wildcard *.so)
...
However, if there are patterns which could generate *.so files which have not yet generated those files, they will (obviously) not be matched by the wildcard, which simply examines existing files. If that's what you actually want to accomplish, you'll probably want to enumerate the actual files, one way or another.

Extracting part of a match in a makefile rule

I have a makefile which generates a bunch of versions of an image in different places:
website/img/logo_256.png
website/img/logo_152.png
/tmp/logo_64.png
and so on (the /tmp/ generation is so I can later use those files to later generate a multiresolution .ico, the details of that aren't important).
I'd like a rule of the form
logo_%.png: ${SRC}
convert $^ -thumbnail $*x$* $#
but, $* brings in the matched directory too, so I get a command of the form:
convert logo_1024.png -thumbnail /tmp/64x/tmp/64 /tmp/logo_64.png
which is incorrect (I need 48x48, not /tmp/48x/tmp/48).
Or I can write
/tmp/logo_%.png: ${SRC}
convert $^ -thumbnail $*x$* $#
website/img/logo_%.png: ${SRC}
convert $^ -thumbnail $*x$* $#
which seems ugly.
I'm sure there are ways to break down and pattern match $# to get what I want, but I'm not a makefile guru, so this would take some research.
What's the easiest way to do this?
See the second half of the Automatic Variables in the GNU Make Manual:
Of the variables listed above, four have values that are single file names, and three have values that are lists of file names. These seven have variants that get just the file's directory name or just the file name within the directory. The variant variables' names are formed by appending ‘D’ or ‘F’, respectively. These variants are semi-obsolete in GNU make since the functions dir and notdir can be used to get a similar effect (see Functions for File Names). Note, however, that the ‘D’ variants all omit the trailing slash which always appears in the output of the dir function. Here is a table of the variants:
‘$(#D)’
The directory part of the file name of the target, with the trailing slash removed. If the value of ‘$#’ is dir/foo.o then ‘$(#D)’ is dir. This value is . if ‘$#’ does not contain a slash.
‘$(#F)’
The file-within-directory part of the file name of the target. If the value of ‘$#’ is dir/foo.o then ‘$(#F)’ is foo.o. ‘$(#F)’ is equivalent to ‘$(notdir $#)’.
‘$(*D)’
‘$(*F)’
The directory part and the file-within-directory part of the stem; dir and foo in this example.
‘$(%D)’
‘$(%F)’
The directory part and the file-within-directory part of the target archive member name. This makes sense only for archive member targets of the form archive(member) and is useful only when member may contain a directory name. (See Archive Members as Targets.)
‘$(<D)’
‘$(<F)’
The directory part and the file-within-directory part of the first prerequisite.
‘$(^D)’
‘$(^F)’
Lists of the directory parts and the file-within-directory parts of all prerequisites.
‘$(+D)’
‘$(+F)’
Lists of the directory parts and the file-within-directory parts of all prerequisites, including multiple instances of duplicated prerequisites.
‘$(?D)’
‘$(?F)’
Lists of the directory parts and the file-within-directory parts of all prerequisites that are newer than the target.
Edit:
As prompted by #Ian's comment I looked again and realized that this was not a complete solution. A complete solution follows.
The above F modifiers (and the $(notdir) function) will strip the path from the target filename. That's part of what is necessary.
Additional manipulation is required to extract only the numerical component from target like /some/path/logo_64.png.
The $(basename) function will strip off suffixes (as will $(patsubst %.png,%,$#) or $(#:%.png=) in a more specific fashion).
Combining those we get from /some/path/logo_64.png to logo_64. Handling things at this point depends heavily on what the data is going to look like and what assertions about it can be made. If logo_ is a static prefix then a simple $(patsubst logo_%,%,...) will work (as will the matching substitution reference like before).
If that is not guaranteed but the guarantee can be made that the dimension will be the last underscore separated component then $(lastword $(subst _, ,...)) can be used.
The rule needed is:
logo_%.png: ${SRC}
convert $^ -thumbnail $(*F)x$(*F) $#
The $(*F), is documented very briefly in the Make manual, as quoted in Etan's answer.
‘$(*F)’ The file-within-directory part of the stem; foo in this example.
The 'stem' ($*) is anything not explicit in the pattern. That includes the wildcard, and any implicit directories. So hence in the question it had the value /tmp/48, /tmp/ from the implicit directory, and 48 from the wildcard in the pattern. So of this combined stem, I need to select just the filename part, $(*F).
Alternatively, noting that the manual states:
These variants are semi-obsolete in GNU make since the functions dir and notdir can be used to get a similar effect
we can instead do:
logo_%.png: ${SRC}
convert $^ -thumbnail $(notdir $*)x$(notdir $*) $#
In a comment, Etan also linked to the How Patterns Match section of the manual, to help understand how the stem is constructed. I found this useful and wanted to bubble it up into an answer.

Fake dynamic files in Makefile

I want to run pocketlint on all **/*.js files.
.PHONY: lint_js2
LINT_JS = $(wildcard static/js/*.js static/js/**/*.js)
LINT_JS_TARGETS = $(addprefix lint__,$(LINT_JS))
#.PHONY: $(LINT_JS_TARGETS)
lint_js2: $(LINT_JS_TARGETS)
echo $<
lint__%: %
pocketlint $<
However, I get this error:
make: *** No rule to make target `lint__static/js/ad_list.js', needed by `lint_js2'. Stop.
Why lint__static/js/ad_list.js is not captured by lint__%?
If I uncomment second .PHONY, it echoes lint__static/js/ad_list.js, but does not invoke pocketlint static/js/ad_list.js. Why?
If my approach is wrong, what would be the right approach? Since tasks are independent, I would appreciate if make -j50 would do what I am expecting.
Thanks!
It's not clear what is intended: does pocketlint write a file named lint__static/js/ad_list.js, or is that really a phony filename? Anyway...
Reread the second paragraph of How Patterns Match:
When the target pattern does not contain a slash (and it usually does not), directory names in the file names are removed from the file name before it is compared with the target prefix and suffix. After the comparison of the file name to the target pattern, the directory names, along with the slash that ends them, are added on to the prerequisite file names generated from the pattern rule's prerequisite patterns and the file name.
In short, % generally matches just a filename, not a pathname with slashes in it. So lint__static/js/ad_list.js is not captured because actually it is only ad_list.js that is being matched against lint__%.
If you can arrange it so that the output files from pocketlint are static/js/lint__ad_list.js etc, then this could be made to work:
LINT_JS_TARGETS = $(foreach f,$(LINT_JS),$(dir $f)lint__$(notdir $f))
lint__%: %
pocketlint $<
Alternatively you can make % match pathnames by having the target pattern be a pathname (containing a slash):
LINT_JS_TARGETS = $(addprefix linted/,$(LINT_JS))
linted/%: %
pocketlint $<
This time % = static/js/ad_list.js does match the pattern rule.
In either case, you're going to have to have pocketlint produce output (if indeed it produces output) named differently than lint__static/*.
Implicit rule search is suppressed for phony targets (see Phony Targets, paragraph 5). So the rule involving pocketlint is never considered when lint__static/js/ad_list.js is phony.
It's not obvious why the result is Nothing to be done for (phony) lint__static/js/ad_list.js rather than No rule to make target lint__static/js/ad_list.js, but I wouldn't lose sleep over it!

How to call functions within order-only prerequisites?

Given this bit of Makefile:
# for pattern matching
$(OBJDIR) := build
# just to see if a level of indirection will work
my_dir = $(dir $(1))
$(OBJECTS) : $(OBJDIR)/% : $(HEADERS) $(SRCDIR)/% | % $(dir %) $(call my_dir,%)
#echo output-only = $|
This is a static pattern rule with order-only prerequisites.
Consider the target "build/utility/debug.js". The output of the above rule will be this:
output-only = utility/debug.js ./
The first component, "utility/debug.js", is properly copied from the stem (%).
The second component, "./", is the output of calling the dir function in the prerequisites list.
The third component, an empty string, is the output of calling my my_dir function in the prerequisites list.
If I change my_dir to this:
my_dir = $(1)
The output remains the same. If I change it to this:
my_dir = "foo"
Then make complains there is no rule to make "foo" (which is expected). It appears, then, that $(1) is not getting bound in the call to my_dir.
What's going on? Why can't I pass the stem to a function? I have a workaround that uses secondary expansion, but I want to know why I can't write my rule this way.
EDIT: I'm new to stackoverflow, forgive me if this is not the way things are done here.
I want $(1) because I am passing the stem as an argument to my_dir, as Alex pointed out.
I don't know why it was suggested I want "$". I don't believe $ by itself expands to anything in any context.
I know that automatic variables are only available in the recipe. I am not using an automatic variable in the prerequisites - I am using the stem:
Each target is matched against the target-pattern to extract a part of the target name, called the stem. This stem is substituted into each of the prereq-patterns to make the prerequisite names (one from each prereq-pattern). - the manual
The fact that the stem is available is demonstrated by the example: the stem expands to the correct value when used alone, but not when passed to a function.
As can be seen in this section of the GNU make manual, variable and function references in the list of prerequisites are immediately expanded, during the read in phase. This means, before any pattern matching is done, so the % has no special meaning yet; it is interpreted as a literal character in the two function references, $(dir %) and $(call my_dir,%), both having ./ as a result, which get merged in the reference to $| in the recipe.
I don't know of any other workaround than the one you already found, i.e. secondary expansion.
Note $1 is not a special variable that expands to anything interesting related to pattern rules (or static pattern rules). The $1 variable only has unique behavior within the context of a user-defined macro invoked by the $(call ...) function.
You wanted to use $*, not $1; $* is an automatic variable which expands to the stem of the target of the rule.
However, in all versions of make (including the POSIX standard definition of make), automatic variables (including $*, $<, $#, $^, etc.) are only available in the context of the recipe. They are not available in the context of the target or prerequisite lists. See the GNU make manual section on Automatic Variables for more details.
As you suggest, there is a GNU make-specific feature enabled by the .SECONDEXPANSION pseudo target which provides a way to avoid this limitation.

Resources