I have a makefile which generates a bunch of versions of an image in different places:
website/img/logo_256.png
website/img/logo_152.png
/tmp/logo_64.png
and so on (the /tmp/ generation is so I can later use those files to later generate a multiresolution .ico, the details of that aren't important).
I'd like a rule of the form
logo_%.png: ${SRC}
convert $^ -thumbnail $*x$* $#
but, $* brings in the matched directory too, so I get a command of the form:
convert logo_1024.png -thumbnail /tmp/64x/tmp/64 /tmp/logo_64.png
which is incorrect (I need 48x48, not /tmp/48x/tmp/48).
Or I can write
/tmp/logo_%.png: ${SRC}
convert $^ -thumbnail $*x$* $#
website/img/logo_%.png: ${SRC}
convert $^ -thumbnail $*x$* $#
which seems ugly.
I'm sure there are ways to break down and pattern match $# to get what I want, but I'm not a makefile guru, so this would take some research.
What's the easiest way to do this?
See the second half of the Automatic Variables in the GNU Make Manual:
Of the variables listed above, four have values that are single file names, and three have values that are lists of file names. These seven have variants that get just the file's directory name or just the file name within the directory. The variant variables' names are formed by appending ‘D’ or ‘F’, respectively. These variants are semi-obsolete in GNU make since the functions dir and notdir can be used to get a similar effect (see Functions for File Names). Note, however, that the ‘D’ variants all omit the trailing slash which always appears in the output of the dir function. Here is a table of the variants:
‘$(#D)’
The directory part of the file name of the target, with the trailing slash removed. If the value of ‘$#’ is dir/foo.o then ‘$(#D)’ is dir. This value is . if ‘$#’ does not contain a slash.
‘$(#F)’
The file-within-directory part of the file name of the target. If the value of ‘$#’ is dir/foo.o then ‘$(#F)’ is foo.o. ‘$(#F)’ is equivalent to ‘$(notdir $#)’.
‘$(*D)’
‘$(*F)’
The directory part and the file-within-directory part of the stem; dir and foo in this example.
‘$(%D)’
‘$(%F)’
The directory part and the file-within-directory part of the target archive member name. This makes sense only for archive member targets of the form archive(member) and is useful only when member may contain a directory name. (See Archive Members as Targets.)
‘$(<D)’
‘$(<F)’
The directory part and the file-within-directory part of the first prerequisite.
‘$(^D)’
‘$(^F)’
Lists of the directory parts and the file-within-directory parts of all prerequisites.
‘$(+D)’
‘$(+F)’
Lists of the directory parts and the file-within-directory parts of all prerequisites, including multiple instances of duplicated prerequisites.
‘$(?D)’
‘$(?F)’
Lists of the directory parts and the file-within-directory parts of all prerequisites that are newer than the target.
Edit:
As prompted by #Ian's comment I looked again and realized that this was not a complete solution. A complete solution follows.
The above F modifiers (and the $(notdir) function) will strip the path from the target filename. That's part of what is necessary.
Additional manipulation is required to extract only the numerical component from target like /some/path/logo_64.png.
The $(basename) function will strip off suffixes (as will $(patsubst %.png,%,$#) or $(#:%.png=) in a more specific fashion).
Combining those we get from /some/path/logo_64.png to logo_64. Handling things at this point depends heavily on what the data is going to look like and what assertions about it can be made. If logo_ is a static prefix then a simple $(patsubst logo_%,%,...) will work (as will the matching substitution reference like before).
If that is not guaranteed but the guarantee can be made that the dimension will be the last underscore separated component then $(lastword $(subst _, ,...)) can be used.
The rule needed is:
logo_%.png: ${SRC}
convert $^ -thumbnail $(*F)x$(*F) $#
The $(*F), is documented very briefly in the Make manual, as quoted in Etan's answer.
‘$(*F)’ The file-within-directory part of the stem; foo in this example.
The 'stem' ($*) is anything not explicit in the pattern. That includes the wildcard, and any implicit directories. So hence in the question it had the value /tmp/48, /tmp/ from the implicit directory, and 48 from the wildcard in the pattern. So of this combined stem, I need to select just the filename part, $(*F).
Alternatively, noting that the manual states:
These variants are semi-obsolete in GNU make since the functions dir and notdir can be used to get a similar effect
we can instead do:
logo_%.png: ${SRC}
convert $^ -thumbnail $(notdir $*)x$(notdir $*) $#
In a comment, Etan also linked to the How Patterns Match section of the manual, to help understand how the stem is constructed. I found this useful and wanted to bubble it up into an answer.
Related
I'm currently learning Make and am struggling to wrap my head around the wildcard concept. Specifically, it seems like there are two symbols that can represent wildcards: * and %
For example, say I want to to generate a variable that identifies all .c source files in the working directory. I would use the traditional * symbol in *.c However, if I use the patsubst function, I am required to use the % symbol instead of * symbol:
// WORKS
SRC = $(wildcard *.c) # list of source files
OBJS = $(patsubst %.c, %.o, $(SRC)) # list of object files
// DOES NOT WORK!!!!
SRC = $(wildcard *.c) # list of source files
OBJS = $(patsubst *.c, *.o, $(SRC)) # list of object files
Can someone explain the difference between * and % in the context of Make wildcards?
Can someone explain the difference between * and % in the context of Make wildcards?
TL;DR:
This is all specific to GNU make.
* and a few other characters are special to the wildcard function, but not to patsubst. This kind of pattern is expanded to the names of existing files and directories that match the pattern, possibly more than one for each pattern.
% is special to the patsubst function. This kind of pattern is used to select matching strings provided by the makefile or directly by make, in the process capturing the part matching the % for later use.
Both kinds have application to the target and prerequisite lists of rules, but their significance there is somewhat different from each other.
None of these are significant to make in recipes, but wildcard-style patterns are significant to the shell, and the shell will interpret them in recipes that it executes.
General
Understand first that $(wildcard) and $(patsubst) are features specific to GNU's implementation of make, and GNU make also attributes special significance to %, *, and a few other characters in rule targets and prerequisites. The POSIX specifications for make say nothing about any of that. GNU make is widely used these days, and with good reason, but it is not the only make you might encounter. If you want maximum portability among make implementations then you must avoid these altogether.
Understand also that a complete, albeit rather basic response to the question would be simply "yes, the wildcard and patsubst functions recognize different special characters." These functions do different things, so it is potentially useful that the special characters of one can be used as ordinary characters in the other.
Wildcards
The asterisk (*) is among the special characters recognized by the Bourne shell for "pathname expansion", which replaces patterns with the names of possibly-many existing files and directories matching the pattern. There are more characters than just * significant in pathname expansion, but % is not among them. Look up that term for a full description.
Additionally, there is the tilde (~), which make and some shells recognize for "tilde expansion", which involves interpreting the first segments of affected paths as specified users' home directories.
The GNU make documentation describes the characters and constructs it recognizes for pathname and tilde expansion as "wildcards", and it has this to say about them:
Wildcard expansion is performed by make automatically in targets and in prerequisites. In recipes, the shell is responsible for wildcard expansion. In other contexts, wildcard expansion happens only if you request it explicitly with the wildcard function.
(GNU make manual, section 4.4)
And that's where the wildcard function appearing in the question comes in -- if you want to perform the same wildcard expansion that make performs automatically on target and prerequisite names in some other context, such as a variable definition, then you can use the wildcard function to get it. Thus,
SRC = $(wildcard *.c)
results variable SRC representing a list of all the existing files in the working directory at the time the makefile is parsed whose names end with .c.
On the other hand, % is not significant for pathname or tilde expansion, so
SRC = $(wildcard %.c)
will expand to literally %.c.
Patterns
The shell documentation uses the term "pattern" to mean shell input that is interpreted to be subject to pathname expansion, rather than being literal. However, GNU make reserves the term for a different kind of pattern in which the % character features. This is relevant in two main areas: the patsubst function and pattern rules.
The patsubst function
The patsubst function computes one string or series of strings from another by replacing those that match a given pattern with a specified replacement. In a pattern, the first % character, if any, matches any number of characters, including zero. In this sense, it can be described as a wildcard, and the manual does, somewhat confusingly, use that term. If the replacement also contains at least one % then the first is replaced by the substring that was matched by the % in the pattern.
Note well that this has nothing inherently to do with file names, existing or otherwise.
Thus, with
SRC = main.c other.c
OBJ = $(patsubst %.c,%.o,$(SRC))
the %.c pattern matches first main.c (with % matching main) and then other.c (with % matching other), and the result for OBJ is the same as this:
OBJ = main.o other.o
regardless of whether any files named main.c, other.c, main.o, or other.o exist.
Pattern rules
If the target of a rule contains a %, then it is interpreted by GNU make as a pattern rule. Pattern rules use % in much the same way that patsubst does. When make needs to build a target but does not have a rule for that specific target, it will check whether the target name matches the target pattern of any pattern rule (including some built-in ones). There does not need to be any existing file of that name. If it finds a match, then it will use that rule to build the target. And in that case, for the purpose of building the target in question, the first % character, if any, in any prerequisite names specified in that rule will be replaced by the stem that matched the % in the target, much like in patsubst.
Combinations
Usually, only one of these kinds of pattern matching / expansion is desired in any given context, but sometimes they can usefully be combined. For example, consider this pattern rule:
%.o: %.c *.h
$(CC) $(CPPFLAGS) $(CFLAGS) -c -o $# $<
That says that any .o file can be built from a corresponding .c file and all the .h files in the working directory by applying the provided recipe.
The wildcard function expects wildcard characters (~, *, ? and [...] which match on file names) while the patsubst function expects a pattern (%) that operates on text. As such $(patsubst *.c, *.o, $(SRC)) means replace the literal string *.c with the string *.o and SRC probably does not contain any literal *.c strings.
I have a rule for the target
data/processed/21.12.2021/experiment6/written_piv21122021.005.exp6.mp4
in my makefile such that it has a dependency
data/raw/21.12.2021/experiment6/piv21122021.005.exp6.mov
Using subst functions, I'm trying to create the dependency by pattern matching as
%/written_*.mp4: \
$(subst processed,raw, $$*)/*.mov \
<do something>
However, the above rule can't find the *.mov dependency. i have tried many versions of $(subst processed,raw, $$*)/*.mov to match the dependency but didn't work.
How to do this? What is the correct syntax?
First, you can't do this:
%/written_*.mp4:
You can't combine a pattern % with a wildcard *. You have to realize that make works in two very discrete steps: first, all the makefiles are parsed and an internal representation of all the targets and prerequisites are constructed into a graph. Then, make walks that graph figuring out what needs to be built and how to build it, and running recipes.
Make variables, functions, and wildcards in targets and prerequisites are expanded when makefiles are parsed (in the first step). Automatic variables like $* are not set until a recipe is invoked (the second step), and patterns like % are not matched/expanded until make tries to decide how to build something (again in the second step).
So, a rule like:
%/written_*.mp4: $(subst processed,raw, $$*)/*.mov
can't work because the wildcard will expand to files matching the literal filename %/written_*.mp4 which clearly has no matches since you won't have a directory named %. In any event you can't use wildcards in targets because when make is parsing the makefile those targets won't exist (since that's what you want make to build) so the wildcards won't match anything. Also, $$* is the literal string $* and there is no processed string in that so the subst function will do nothing. And, even if /*.mov did match something it would put ALL the files matching that wildcard as prerequisites of every target so they'd all get rebuilt whenever any one changed.
And finally, you definitely should not use a backslash after your prerequisites: this just turns your recipe into prerequisites.
Your problem is very difficult to solve because your target and prerequisite differ in multiple distinct places and make doesn't support multiple % matching. You can get most of the way there with this:
data/processed/%.mp4: data/raw/%.mov
<do something>
However this is not quite right because the % in the target is written_... while in the prerequisite it's just ... and this is not possible to represent in make.
If you can rework the filenames so that instead of written_piv21122021.005.exp6.mp4 you can use piv21122021.005.exp6.written.mp4 (or even better you don't need the written_ prefix at all) then you can easily do this. If not you'll need to get very fancy to make this work.
Here is an imperfect kludge.
Delegate the work to another makefile I'll call adjunct.mk. In the main makefile:
data/processed/%.mp4:
#$(MAKE) -f adjunct.mk $#
And in adjunct.mk this ugly transformation:
.SECONDEXPANSION:
$(MAKECMDGOALS): $$(patsubst data/processed/%,data/raw/%,$$(subst written_,,$$(patsubst %.mp4,%.mov,$$#)))
...do whatever with $< and $#...
This incurs the usual cost of recursive Make: it blinds the higher-level Make to the dependency relations being tracked by the lower one. So if your Make must build that mov file, or you try to build the mp4 when the mov does not exist, then this solution will require some more careful pipefitting before it will work correctly.
I'm trying to write a Makefile where multiple source files (in my case they are markdown) create multiple target files (pdfs). However, the target files generated have extra characters in the file name that can't be predicted (it happens to be a version number encoded in the source), but ideally the Makefile would not have to read the source itself.
So, for example:
file1.md => file1-v1.pdf
file2.md => file2-v2.pdf
...
I can calculate source name given a target name (by excluding anything after the hyphen and adding .md), but cannot calculate target name given the source.
Is it possible to write a Makefile that builds only the targets where the source have been updated?
This will be ugly, but it will work.
As it often is with Make, our problem divides into these two problems:
1. construct a list of targets
2. build them
Suppose we have five md files which map to pdf files (whose names we don't know beforehand):
file1.md => file1-v1.pdf
file2.md => file2-v1.pdf
file3.md => file3-v1.pdf
file4.md => file4-v1.pdf
file5.md => file5-v1.pdf
We can't use the real output file names as targets, because we don't know them beforehand, but we see five input files and know that we must build one output file for each. For now, a fake target name will do:
file1-dummy.pdf: file1.md
zap file1.md
When Make executes this rule, it produces the file file1-v1.pdf. The fact that it doesn't produce a file named file1-dummy.pdf is disquieting, but not a serious problem. We can turn this into a pattern rule:
%-dummy.pdf: %.md
zap $<
Then all we have to do is turn the list of existing input files (file1.md, file2.md, ...) into a list of dummy targets (file1-dummy.pdf, file2-dummy.pdf, ...), and build them. So far, so good.
But suppose some of the output files already exist. If file2-v2.pdf already exists -- and is newer than file2.md -- then we would prefer that Make not rebuild it (by attempting to build file2-dummy.pdf). In that case we would prefer that file2-v2.pdf be in the target list, with a rule that worked like this:
file2-v2.pdf: file2.md
zap $<
This is not easy to turn into a pattern rule, because Make does not handle wildcards very well, and cannot cope with multiple wildcards in a single phrase, not without a lot of clumsiness. But there is a way to write one rule that will cover both cases. First note that we can obtain the part of a variable before the hyphen with this kludge:
$(basename $(subst -,.,$(VAR)))
Armed with this, and with secondary expansion, we can write a pattern rule that will work with both cases, and construct a target list that will exploit it:
# There are other ways to construct these two lists, but this will do.
MD := $(wildcard *.md)
PDF := $(wildcard *.pdf)
PDFROOTS := $(basename $(subst -,.,$(basename $(PDF))))
MDROOTS := $(filter-out $(PDFROOTS), $(basename $(MD)))
TARGETS:= $(addsuffix -foo.pdf, $(MDROOTS)) $(PDF)
.SECONDEXPANSION:
%.pdf: $$(basename $$(subst -,., $$*)).md
# perform actions on $<
Make's algorithm always starts with the final output product and works its way backwards to the source files, to see what needs to be updated.
Therefore, you HAVE to be able to enumerate the final output product as a target name and correlate that back to the inputs that generate that output, for make to work.
This is also why make is not a great tool for building Java, for example, since the output filenames don't map easily to the input file names.
So, you must have at least one target/prerequisite pair which is derivable (for implicit rules), or state-able (for explicit rules)--that is, known at the time you write the makefile. If you don't then a marker file is your only alternative. Note you CAN add extra generated, non-derivative prerequisites (for example, in compilers you can add header files as prerequisites that are not related to the source file name), in addition to the known prerequisite.
#Beta's answer is informative and helpful, but I needed a solution (using GNU Make 4.1) that worked when the destination filename bears no resemblance to the input filename, for example, if it is generated from its content. I came up with the following, which takes every file matching *.in, and creates a file by reading the contents of the source file, appending a .txt, and using it as a filename to create. (For example, if test.in exists and contains foo, the makefile will create a foo.txt file.)
SRCS := $(wildcard *.in)
.PHONY: all
all: all_s
define TXT_template =
$(2).txt: $(1)
touch $$#
ALL += $(2).txt
endef
$(foreach src,$(SRCS),$(eval $(call TXT_template, $(src), $(shell cat $(src)))))
.SECONDARY_EXPANSION:
all_s: $(ALL)
The explanation:
The define block defines the recipe needed to make the text file from the .in file. It's a function that takes two parameters; $(1) is the .in. file and $(2) is the contents of it, or the base of the output filename. Replace touch with whatever makes the output. We have to use $$# because eval will expand everything once, but we want $# to left after this expansion. Since we have to collect all the generated targets so we known what all the make, the ALL line accumulates the targets into one variable. The foreach line goes through each source file, calls the function with the source filename and the contents of the file (i.e. what we want to be the name of the target, here you'd normally use whatever script generates the desired filename), and then evaluates the resulting block, dynamically adding the recipe to make. Thanks to Beta for explaining .SECONDARY_EXPANSION; I needed it for reasons not entirely clear to me, but it works (putting all: $(ALL) at the top doesn't work). The all: at the top depends on the secondary expansion of all_s: at the bottom and somehow this magic makes it work. Comments welcome.
maybe try this ? or something along those lines
# makefile
SRCS=$(wildcard *.md)
PDFS=$(shell printf *.pdf)
$(PDFS): $(SRCS)
command ...
the printf *.pdf is meant to either expand to the first of the pdf files if they exist, else fail if they don't and that will signal to make that it should build. if this doesn't work i suggest maybe experimenting with find, ls or other listing tools (e.g. compgen, complete), maybe even in combination with xargs to get everything on one line.
In a Makefile I have:
images/schematic.pdf: images/schematic.svg
inkscape -D -z --file=$^ --export-pdf=$# --export-latex
sed -i "s:schematic:images/schematic:g" $#_tex
What this rule does is:
Use inkscape to convert to a latex-ready .pdf PDF file + its corresponding .pdf_tex text file (see this answer: https://tex.stackexchange.com/a/2107/104581)
Modify the mentioned .pdf_tex file so that it will not break latex compiling from a "higher" directory (namely . when the .pdf_tex file is in ./images)
My problem:
I have many rules in this form i. e. where only schematic is changed. I would like to use a pattern-rule that replaces schematic by %. And to use % in the recipe (in the sed command).
However the rule:
images/%.pdf: images/%.svg
inkscape -D -z --file=$^ --export-pdf=$# --export-latex
sed -i "s:%:images/%:g" $#_tex
does not works: % is interpreted literally in the recipe.
I also tried to replace % in the recipe by $% but this variable seems to be empty.
Unsatisfactory solution:
Add a line in the recipe to create a (make) variable that will hold the result of notdir(removeprefix($<)) (using this question or a call to bash because there is no removeprefix in GNU Make).
You want $*.
From https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html :
$*:
The stem with which an implicit rule matches (see How Patterns Match).
If the target is dir/a.foo.b and the target pattern is a.%.b
then the stem is dir/foo. The stem is useful for constructing names of
related files.
In a static pattern rule, the stem is part of the file name that matched the ‘%’ in the target pattern.
In an explicit rule, there is no stem; so ‘$*’ cannot be determined in that way. Instead, if the target name ends with a
recognized suffix (see Old-Fashioned Suffix Rules), ‘$*’ is set to the
target name minus the suffix. For example, if the target name is
‘foo.c’, then ‘$*’ is set to ‘foo’, since ‘.c’ is a suffix. GNU make
does this bizarre thing only for compatibility with other
implementations of make. You should generally avoid using ‘$*’ except
in implicit rules or static pattern rules.
If the target name in an explicit rule does not end with a recognized suffix, ‘$*’ is set to the empty string for that rule.
$% is an automatic variable, but it's for archive (.a library) members, and almost never useful.
I have this implicit rule:
%.so: %.so.5
qnx_ln $< $#
I realized that for another target, I have to make all .so files the prerequisite for that target.
I tried this:
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs %.so
...
But I got the output
*** No rule to make target '%.so', needed by 'makegen'. Stop.
% prerequisite patterns can only be used in static and implicit pattern rules, where they match the respective % part of the target; when used in a regular rule % is a literal character.
You'll need to specify the dependencies literally, unless there is some correspondence between certain source filenames and the .so filenames that you can leverage, presumably you're already doing either of these to link the .so files in the first place.
As pointed out previously, no you can't do that because this is not how prerequisite patterns work. Maybe you gave the following a thought and rejected it but I suspect you might find the following a close-enough fit:
%.so.target: %.so.5
echo $< >> $(BUILD)/so.targets
SO_TARGETS=$(basename $(shell cat $(BUILD)/so.targets))
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs $(SO_TARGETS)
Maybe you are looking for a rule to match on every existing *.so file?
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs $(wildcard *.so)
...
However, if there are patterns which could generate *.so files which have not yet generated those files, they will (obviously) not be matched by the wildcard, which simply examines existing files. If that's what you actually want to accomplish, you'll probably want to enumerate the actual files, one way or another.