I have defined a list of paths
paths = /src/nam1 /src/name2
And I'm trying to replace each of them with multiple paths like this
/src/nam1 -> /dist/nam1.foo /dist/nam1.bar
/src/nam2 -> /dist/nam2.foo /dist/nam2.bar
patsubst only allows me do to a 1:1 mapping
$(patsubst src/%,dist/%.foo,$(paths))
Is it possible to also do the mapping for .bar without repeating the whole thing?
What about:
$(foreach x,foo bar,$(patsubst src/%,dist/%.$(x),$(paths)))
Related
I'm currently learning Make and am struggling to wrap my head around the wildcard concept. Specifically, it seems like there are two symbols that can represent wildcards: * and %
For example, say I want to to generate a variable that identifies all .c source files in the working directory. I would use the traditional * symbol in *.c However, if I use the patsubst function, I am required to use the % symbol instead of * symbol:
// WORKS
SRC = $(wildcard *.c) # list of source files
OBJS = $(patsubst %.c, %.o, $(SRC)) # list of object files
// DOES NOT WORK!!!!
SRC = $(wildcard *.c) # list of source files
OBJS = $(patsubst *.c, *.o, $(SRC)) # list of object files
Can someone explain the difference between * and % in the context of Make wildcards?
Can someone explain the difference between * and % in the context of Make wildcards?
TL;DR:
This is all specific to GNU make.
* and a few other characters are special to the wildcard function, but not to patsubst. This kind of pattern is expanded to the names of existing files and directories that match the pattern, possibly more than one for each pattern.
% is special to the patsubst function. This kind of pattern is used to select matching strings provided by the makefile or directly by make, in the process capturing the part matching the % for later use.
Both kinds have application to the target and prerequisite lists of rules, but their significance there is somewhat different from each other.
None of these are significant to make in recipes, but wildcard-style patterns are significant to the shell, and the shell will interpret them in recipes that it executes.
General
Understand first that $(wildcard) and $(patsubst) are features specific to GNU's implementation of make, and GNU make also attributes special significance to %, *, and a few other characters in rule targets and prerequisites. The POSIX specifications for make say nothing about any of that. GNU make is widely used these days, and with good reason, but it is not the only make you might encounter. If you want maximum portability among make implementations then you must avoid these altogether.
Understand also that a complete, albeit rather basic response to the question would be simply "yes, the wildcard and patsubst functions recognize different special characters." These functions do different things, so it is potentially useful that the special characters of one can be used as ordinary characters in the other.
Wildcards
The asterisk (*) is among the special characters recognized by the Bourne shell for "pathname expansion", which replaces patterns with the names of possibly-many existing files and directories matching the pattern. There are more characters than just * significant in pathname expansion, but % is not among them. Look up that term for a full description.
Additionally, there is the tilde (~), which make and some shells recognize for "tilde expansion", which involves interpreting the first segments of affected paths as specified users' home directories.
The GNU make documentation describes the characters and constructs it recognizes for pathname and tilde expansion as "wildcards", and it has this to say about them:
Wildcard expansion is performed by make automatically in targets and in prerequisites. In recipes, the shell is responsible for wildcard expansion. In other contexts, wildcard expansion happens only if you request it explicitly with the wildcard function.
(GNU make manual, section 4.4)
And that's where the wildcard function appearing in the question comes in -- if you want to perform the same wildcard expansion that make performs automatically on target and prerequisite names in some other context, such as a variable definition, then you can use the wildcard function to get it. Thus,
SRC = $(wildcard *.c)
results variable SRC representing a list of all the existing files in the working directory at the time the makefile is parsed whose names end with .c.
On the other hand, % is not significant for pathname or tilde expansion, so
SRC = $(wildcard %.c)
will expand to literally %.c.
Patterns
The shell documentation uses the term "pattern" to mean shell input that is interpreted to be subject to pathname expansion, rather than being literal. However, GNU make reserves the term for a different kind of pattern in which the % character features. This is relevant in two main areas: the patsubst function and pattern rules.
The patsubst function
The patsubst function computes one string or series of strings from another by replacing those that match a given pattern with a specified replacement. In a pattern, the first % character, if any, matches any number of characters, including zero. In this sense, it can be described as a wildcard, and the manual does, somewhat confusingly, use that term. If the replacement also contains at least one % then the first is replaced by the substring that was matched by the % in the pattern.
Note well that this has nothing inherently to do with file names, existing or otherwise.
Thus, with
SRC = main.c other.c
OBJ = $(patsubst %.c,%.o,$(SRC))
the %.c pattern matches first main.c (with % matching main) and then other.c (with % matching other), and the result for OBJ is the same as this:
OBJ = main.o other.o
regardless of whether any files named main.c, other.c, main.o, or other.o exist.
Pattern rules
If the target of a rule contains a %, then it is interpreted by GNU make as a pattern rule. Pattern rules use % in much the same way that patsubst does. When make needs to build a target but does not have a rule for that specific target, it will check whether the target name matches the target pattern of any pattern rule (including some built-in ones). There does not need to be any existing file of that name. If it finds a match, then it will use that rule to build the target. And in that case, for the purpose of building the target in question, the first % character, if any, in any prerequisite names specified in that rule will be replaced by the stem that matched the % in the target, much like in patsubst.
Combinations
Usually, only one of these kinds of pattern matching / expansion is desired in any given context, but sometimes they can usefully be combined. For example, consider this pattern rule:
%.o: %.c *.h
$(CC) $(CPPFLAGS) $(CFLAGS) -c -o $# $<
That says that any .o file can be built from a corresponding .c file and all the .h files in the working directory by applying the provided recipe.
The wildcard function expects wildcard characters (~, *, ? and [...] which match on file names) while the patsubst function expects a pattern (%) that operates on text. As such $(patsubst *.c, *.o, $(SRC)) means replace the literal string *.c with the string *.o and SRC probably does not contain any literal *.c strings.
I have a rule for the target
data/processed/21.12.2021/experiment6/written_piv21122021.005.exp6.mp4
in my makefile such that it has a dependency
data/raw/21.12.2021/experiment6/piv21122021.005.exp6.mov
Using subst functions, I'm trying to create the dependency by pattern matching as
%/written_*.mp4: \
$(subst processed,raw, $$*)/*.mov \
<do something>
However, the above rule can't find the *.mov dependency. i have tried many versions of $(subst processed,raw, $$*)/*.mov to match the dependency but didn't work.
How to do this? What is the correct syntax?
First, you can't do this:
%/written_*.mp4:
You can't combine a pattern % with a wildcard *. You have to realize that make works in two very discrete steps: first, all the makefiles are parsed and an internal representation of all the targets and prerequisites are constructed into a graph. Then, make walks that graph figuring out what needs to be built and how to build it, and running recipes.
Make variables, functions, and wildcards in targets and prerequisites are expanded when makefiles are parsed (in the first step). Automatic variables like $* are not set until a recipe is invoked (the second step), and patterns like % are not matched/expanded until make tries to decide how to build something (again in the second step).
So, a rule like:
%/written_*.mp4: $(subst processed,raw, $$*)/*.mov
can't work because the wildcard will expand to files matching the literal filename %/written_*.mp4 which clearly has no matches since you won't have a directory named %. In any event you can't use wildcards in targets because when make is parsing the makefile those targets won't exist (since that's what you want make to build) so the wildcards won't match anything. Also, $$* is the literal string $* and there is no processed string in that so the subst function will do nothing. And, even if /*.mov did match something it would put ALL the files matching that wildcard as prerequisites of every target so they'd all get rebuilt whenever any one changed.
And finally, you definitely should not use a backslash after your prerequisites: this just turns your recipe into prerequisites.
Your problem is very difficult to solve because your target and prerequisite differ in multiple distinct places and make doesn't support multiple % matching. You can get most of the way there with this:
data/processed/%.mp4: data/raw/%.mov
<do something>
However this is not quite right because the % in the target is written_... while in the prerequisite it's just ... and this is not possible to represent in make.
If you can rework the filenames so that instead of written_piv21122021.005.exp6.mp4 you can use piv21122021.005.exp6.written.mp4 (or even better you don't need the written_ prefix at all) then you can easily do this. If not you'll need to get very fancy to make this work.
Here is an imperfect kludge.
Delegate the work to another makefile I'll call adjunct.mk. In the main makefile:
data/processed/%.mp4:
#$(MAKE) -f adjunct.mk $#
And in adjunct.mk this ugly transformation:
.SECONDEXPANSION:
$(MAKECMDGOALS): $$(patsubst data/processed/%,data/raw/%,$$(subst written_,,$$(patsubst %.mp4,%.mov,$$#)))
...do whatever with $< and $#...
This incurs the usual cost of recursive Make: it blinds the higher-level Make to the dependency relations being tracked by the lower one. So if your Make must build that mov file, or you try to build the mp4 when the mov does not exist, then this solution will require some more careful pipefitting before it will work correctly.
Wondering how to apply something such as pattern rules to accomplish the following in the Makefile:
FILES := a b c d
$(FILES).c:
run build $(FILE).c
# use $(FILE) without .c extension, etc.
Without having to name all of the values in FILES like a.c, b.c, etc. because I am going to use the names elsewhere.
If I try the above with make a.c, I get:
No rule to make target `a.c'. Stop.
The addsuffix built-in function, which already operates on lists, could be useful here:
targets := $(addsuffix .c,$(FILES))
However, you could simply use pattern rules instead. The automatic variable $* would correspond to the stem (i.e., the % part):
%.c:
run build $#
# use $* for filename without .c extension, etc.
I am trying to generate list of object files from source files in my makefile using patsubst
OUT_DIR=Out/
SRC=../../../Client2.4/Client/src/BrokerModule/BrokerApp.cpp
../../../Client2.4/Client/src/CommandMsgManager/CConfigModuleInfo.cpp
OBJ:= $(patsubst %src/%.cpp,${OUT_DIR}$%.o,$(SRC))
I want my OBJ variable to be
OBJ=Out/BrokerModule/BrokerApp.o Out/CommandMsgManager/CConfigModuleInfo.o
after patsubst but above patsubst is not producing the desired result. Please help.
There are some problems with the usage of patsubst, see my suggestion as followed,
OUT_DIR=Out/
SRC=../../../Client2.4/Client/src/BrokerModule/BrokerApp.cpp \
../../../Client2.4/Client/src/CommandMsgManager/CConfigModuleInfo.cpp
# add the definition of src
src=../../../Client2.4/Client/src/
# Modify the definition of OBJ
OBJ:= $(patsubst ${src}%.cpp,${OUT_DIR}%.o,$(SRC))
Filtered out the prepended ${src} and appended .cpp, and keep only
BrokerModule/BrokerApp.cpp & CommandMsgManager/CConfigModuleInfo.cpp.
And % is replaced by the text that matched the % in the previous step.
Patsubst can only handle patterns with one wildcard in it, unluckily. Moreover you are trying to take apart path names not the usual way at the file level. That means, as long as you neither know the prefix nor the postfix parts of the /src/ in your strings, you are out of luck as you can never say 'replace unknown prefix and conserve unknown postfix' (or the other way round).
The usual solution is to 'know' the prefix:
OUT_DIR=Out/
SRC_PATH := ../../../Client2.4/Client/src
SRC=../../../Client2.4/Client/src/BrokerModule/BrokerApp.cpp \
../../../Client2.4/Client/src/CommandMsgManager/CConfigModuleInfo.cpp
OBJ:= $(patsubst $(SRC_PATH)/%,${OUT_DIR}%,$(SRC))
$(info $(OBJ))
Another solution is to use e.g. the GNUmake table toolkit library of make functions (still beta but your problem can be solved):
include gmtt.mk
OUT_DIR=Out
SRC=../../../Client2.4/Client/src/BrokerModule/BrokerApp.cpp \
../../../Client5.6/Client/src/CommandMsgManager/CConfigModuleInfo.cpp
strip-till-last-src = src/$(call implode,$(call down-to,src/,$(call explode,/,$1)))
OBJ:= $(foreach a-path,$(SRC),$(OUT_DIR)/$(call strip-till-last-src,$(a-path)))
$(info $(OBJ))
I've git a project where, at some point in its Makefile, I'm filtering out stuff from a certain directory:
relevant = $(filter-out irrelevant/%,$^)
Now I want to use this in a VPATH-enabled environment. So the paths of my dependencies in $^ might not start with irrelevant any more, but instead something like ../src/irrelevant or similar.
Is there a way to filter-out anything that contains irrelevant, in any position? I.e. something like the following?
relevant = $(filter-out %irrelevant/%,$^)
This doesn't work, since apparently patterns for filter-out can contain only a single % wildcard. I know I could possibly achieve this via a shell invocation, grep or whatever, but I was hoping for some combinations of functions inside the Makefile.
Try
relevant = $(foreach a,$^,$(if $(findstring irrelevant,$a),,$a))