I have a rule for the target
data/processed/21.12.2021/experiment6/written_piv21122021.005.exp6.mp4
in my makefile such that it has a dependency
data/raw/21.12.2021/experiment6/piv21122021.005.exp6.mov
Using subst functions, I'm trying to create the dependency by pattern matching as
%/written_*.mp4: \
$(subst processed,raw, $$*)/*.mov \
<do something>
However, the above rule can't find the *.mov dependency. i have tried many versions of $(subst processed,raw, $$*)/*.mov to match the dependency but didn't work.
How to do this? What is the correct syntax?
First, you can't do this:
%/written_*.mp4:
You can't combine a pattern % with a wildcard *. You have to realize that make works in two very discrete steps: first, all the makefiles are parsed and an internal representation of all the targets and prerequisites are constructed into a graph. Then, make walks that graph figuring out what needs to be built and how to build it, and running recipes.
Make variables, functions, and wildcards in targets and prerequisites are expanded when makefiles are parsed (in the first step). Automatic variables like $* are not set until a recipe is invoked (the second step), and patterns like % are not matched/expanded until make tries to decide how to build something (again in the second step).
So, a rule like:
%/written_*.mp4: $(subst processed,raw, $$*)/*.mov
can't work because the wildcard will expand to files matching the literal filename %/written_*.mp4 which clearly has no matches since you won't have a directory named %. In any event you can't use wildcards in targets because when make is parsing the makefile those targets won't exist (since that's what you want make to build) so the wildcards won't match anything. Also, $$* is the literal string $* and there is no processed string in that so the subst function will do nothing. And, even if /*.mov did match something it would put ALL the files matching that wildcard as prerequisites of every target so they'd all get rebuilt whenever any one changed.
And finally, you definitely should not use a backslash after your prerequisites: this just turns your recipe into prerequisites.
Your problem is very difficult to solve because your target and prerequisite differ in multiple distinct places and make doesn't support multiple % matching. You can get most of the way there with this:
data/processed/%.mp4: data/raw/%.mov
<do something>
However this is not quite right because the % in the target is written_... while in the prerequisite it's just ... and this is not possible to represent in make.
If you can rework the filenames so that instead of written_piv21122021.005.exp6.mp4 you can use piv21122021.005.exp6.written.mp4 (or even better you don't need the written_ prefix at all) then you can easily do this. If not you'll need to get very fancy to make this work.
Here is an imperfect kludge.
Delegate the work to another makefile I'll call adjunct.mk. In the main makefile:
data/processed/%.mp4:
#$(MAKE) -f adjunct.mk $#
And in adjunct.mk this ugly transformation:
.SECONDEXPANSION:
$(MAKECMDGOALS): $$(patsubst data/processed/%,data/raw/%,$$(subst written_,,$$(patsubst %.mp4,%.mov,$$#)))
...do whatever with $< and $#...
This incurs the usual cost of recursive Make: it blinds the higher-level Make to the dependency relations being tracked by the lower one. So if your Make must build that mov file, or you try to build the mp4 when the mov does not exist, then this solution will require some more careful pipefitting before it will work correctly.
Related
Using GNU make, I am trying to solve a problem similar to make recipe execute twice — that is, to have a Makefile recipe run twice. In my case, however, the recipe is run under the .SECONDEXPANSION target, and the two different runs will be called with different parameters to generate different versions of the output file from the same input file. That is, with input file foo, this example Makefile should be callable via make foo.pdf or make foo.expanded.pdf to build one .pdf file, or make all to build both .pdf files:
.PHONY: all
all: foo.pdf foo.expanded.pdf
.SECONDEXPANSION:
%.expanded.pdf %.pdf: %
#echo building $(basename $#)
Of the two solutions given in that answer, the first is unsuitable because it always runs the rule twice; I want it run twice when the user asks for it.
The second solution posted there is conceptually what I am looking for and have implemented in the above example Makefile, with only the small problem that it doesn't work: although the all target lists both .pdf files as dependencies, only one is built when make all is run.
Is there a way to tell GNU make to build two different files using the same rule under a .SECONDEXPANSION?
EDIT: Clarified in problem description that the same input file is used to build both versions of the output file, and modified sample Makefile to include this dependency.
EDIT: I would like a solution as scalable as possible; that is, it should work if the input filename contains dots, specifying additional output file foo.reduced.pdf should require only adjusting the targets and recipe as appropriate, etc. This limits performing string surgery that relies on the filenames appearing exactly as given in this narrow example (e.g., changing the rule to %.pdf: $$(firstword $$(subst ., ,$$*)) fails if the input file could be either foo or foo.bar).
You are probably looking for Pattern-specific Variable Values. Let's assume your recipe depends on a make variable named BUILDFLAGS that takes value normal by default and special for the "expanded" targets. Then this Makefile:
BUILDER := builder
BUILDFLAGS := normal
.PHONY: all
all: foo.pdf foo.expanded.pdf
%.expanded.pdf: BUILDFLAGS := special
%.pdf:
$(BUILDER) $(BUILDFLAGS) $#
should do about what you want with the same rule for all targets, plus one pattern-specific variable value declaration. Replace builder, normal and special with what makes sense in your case. Demo:
$ make foo.pdf
builder normal foo.pdf
$ make foo.expanded.pdf
builder special foo.expanded.pdf
$ make
builder normal foo.pdf
builder special foo.expanded.pdf
Your problem has nothing to do with .SECONDEXPANSION. You can just drop that and the problem will be the same.
Your problem is that you are using a pattern rule with multiple target patterns, and expecting that it works similar to an explicit rule with multiple targets. But it does not (and in fact you cannot have a rule with both pattern and explicit targets).
For a pattern rule with multiple target patterns, Make matches the same pattern to all the %, including multiple times in the targets, and then assumes that it just has to execute the recipe with that pattern once, and it will make all the matched targets.
In your case the best way is to use multiple rules (I changed your recipe because using echo as a Make recipe is a bad idea):
.PHONY: all
all: foo.expanded.pdf foo.pdf
RECIPE = touch $#
%.expanded.pdf:
$(RECIPE)
%.pdf:
$(RECIPE)
I'm trying to write a Makefile where multiple source files (in my case they are markdown) create multiple target files (pdfs). However, the target files generated have extra characters in the file name that can't be predicted (it happens to be a version number encoded in the source), but ideally the Makefile would not have to read the source itself.
So, for example:
file1.md => file1-v1.pdf
file2.md => file2-v2.pdf
...
I can calculate source name given a target name (by excluding anything after the hyphen and adding .md), but cannot calculate target name given the source.
Is it possible to write a Makefile that builds only the targets where the source have been updated?
This will be ugly, but it will work.
As it often is with Make, our problem divides into these two problems:
1. construct a list of targets
2. build them
Suppose we have five md files which map to pdf files (whose names we don't know beforehand):
file1.md => file1-v1.pdf
file2.md => file2-v1.pdf
file3.md => file3-v1.pdf
file4.md => file4-v1.pdf
file5.md => file5-v1.pdf
We can't use the real output file names as targets, because we don't know them beforehand, but we see five input files and know that we must build one output file for each. For now, a fake target name will do:
file1-dummy.pdf: file1.md
zap file1.md
When Make executes this rule, it produces the file file1-v1.pdf. The fact that it doesn't produce a file named file1-dummy.pdf is disquieting, but not a serious problem. We can turn this into a pattern rule:
%-dummy.pdf: %.md
zap $<
Then all we have to do is turn the list of existing input files (file1.md, file2.md, ...) into a list of dummy targets (file1-dummy.pdf, file2-dummy.pdf, ...), and build them. So far, so good.
But suppose some of the output files already exist. If file2-v2.pdf already exists -- and is newer than file2.md -- then we would prefer that Make not rebuild it (by attempting to build file2-dummy.pdf). In that case we would prefer that file2-v2.pdf be in the target list, with a rule that worked like this:
file2-v2.pdf: file2.md
zap $<
This is not easy to turn into a pattern rule, because Make does not handle wildcards very well, and cannot cope with multiple wildcards in a single phrase, not without a lot of clumsiness. But there is a way to write one rule that will cover both cases. First note that we can obtain the part of a variable before the hyphen with this kludge:
$(basename $(subst -,.,$(VAR)))
Armed with this, and with secondary expansion, we can write a pattern rule that will work with both cases, and construct a target list that will exploit it:
# There are other ways to construct these two lists, but this will do.
MD := $(wildcard *.md)
PDF := $(wildcard *.pdf)
PDFROOTS := $(basename $(subst -,.,$(basename $(PDF))))
MDROOTS := $(filter-out $(PDFROOTS), $(basename $(MD)))
TARGETS:= $(addsuffix -foo.pdf, $(MDROOTS)) $(PDF)
.SECONDEXPANSION:
%.pdf: $$(basename $$(subst -,., $$*)).md
# perform actions on $<
Make's algorithm always starts with the final output product and works its way backwards to the source files, to see what needs to be updated.
Therefore, you HAVE to be able to enumerate the final output product as a target name and correlate that back to the inputs that generate that output, for make to work.
This is also why make is not a great tool for building Java, for example, since the output filenames don't map easily to the input file names.
So, you must have at least one target/prerequisite pair which is derivable (for implicit rules), or state-able (for explicit rules)--that is, known at the time you write the makefile. If you don't then a marker file is your only alternative. Note you CAN add extra generated, non-derivative prerequisites (for example, in compilers you can add header files as prerequisites that are not related to the source file name), in addition to the known prerequisite.
#Beta's answer is informative and helpful, but I needed a solution (using GNU Make 4.1) that worked when the destination filename bears no resemblance to the input filename, for example, if it is generated from its content. I came up with the following, which takes every file matching *.in, and creates a file by reading the contents of the source file, appending a .txt, and using it as a filename to create. (For example, if test.in exists and contains foo, the makefile will create a foo.txt file.)
SRCS := $(wildcard *.in)
.PHONY: all
all: all_s
define TXT_template =
$(2).txt: $(1)
touch $$#
ALL += $(2).txt
endef
$(foreach src,$(SRCS),$(eval $(call TXT_template, $(src), $(shell cat $(src)))))
.SECONDARY_EXPANSION:
all_s: $(ALL)
The explanation:
The define block defines the recipe needed to make the text file from the .in file. It's a function that takes two parameters; $(1) is the .in. file and $(2) is the contents of it, or the base of the output filename. Replace touch with whatever makes the output. We have to use $$# because eval will expand everything once, but we want $# to left after this expansion. Since we have to collect all the generated targets so we known what all the make, the ALL line accumulates the targets into one variable. The foreach line goes through each source file, calls the function with the source filename and the contents of the file (i.e. what we want to be the name of the target, here you'd normally use whatever script generates the desired filename), and then evaluates the resulting block, dynamically adding the recipe to make. Thanks to Beta for explaining .SECONDARY_EXPANSION; I needed it for reasons not entirely clear to me, but it works (putting all: $(ALL) at the top doesn't work). The all: at the top depends on the secondary expansion of all_s: at the bottom and somehow this magic makes it work. Comments welcome.
maybe try this ? or something along those lines
# makefile
SRCS=$(wildcard *.md)
PDFS=$(shell printf *.pdf)
$(PDFS): $(SRCS)
command ...
the printf *.pdf is meant to either expand to the first of the pdf files if they exist, else fail if they don't and that will signal to make that it should build. if this doesn't work i suggest maybe experimenting with find, ls or other listing tools (e.g. compgen, complete), maybe even in combination with xargs to get everything on one line.
I have this implicit rule:
%.so: %.so.5
qnx_ln $< $#
I realized that for another target, I have to make all .so files the prerequisite for that target.
I tried this:
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs %.so
...
But I got the output
*** No rule to make target '%.so', needed by 'makegen'. Stop.
% prerequisite patterns can only be used in static and implicit pattern rules, where they match the respective % part of the target; when used in a regular rule % is a literal character.
You'll need to specify the dependencies literally, unless there is some correspondence between certain source filenames and the .so filenames that you can leverage, presumably you're already doing either of these to link the .so files in the first place.
As pointed out previously, no you can't do that because this is not how prerequisite patterns work. Maybe you gave the following a thought and rejected it but I suspect you might find the following a close-enough fit:
%.so.target: %.so.5
echo $< >> $(BUILD)/so.targets
SO_TARGETS=$(basename $(shell cat $(BUILD)/so.targets))
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs $(SO_TARGETS)
Maybe you are looking for a rule to match on every existing *.so file?
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs $(wildcard *.so)
...
However, if there are patterns which could generate *.so files which have not yet generated those files, they will (obviously) not be matched by the wildcard, which simply examines existing files. If that's what you actually want to accomplish, you'll probably want to enumerate the actual files, one way or another.
I'm trying to build several executables in one make instance, as suggested by
Recursive Make Considered Harmful.
The arguments of that paper apply to my project because some of my source files are generated, and go into multiple executables.
The object files for each of these executables go into separate directories, because they are compiled with different preprocessor flags.
How can I specify separate vpath settings for each of these executables, so that source files with duplicate filenames in separate directories go into the executables where I want them to go?
Setting vpath before the rules to build the object files for one executable, and erasing it afterwards (by not giving any directories) doesn't have the desired effect. Apparently, the last setting is used for all the rules.
Solutions I see currently:
Rename source filenames to use unique names so that I can use a global vpath setting
Instead of vpath, use separate rules for each source directory, with the source directory in the prerequisite pattern (*)
Recursive make, with separate vpath settings in each make instance, and somehow deal with the resulting trouble.
Use something different than make.
Is there a better solution, or which one of the above would you prefer?
(*) The solution with separate rules looks like this:
build/$(PROGRAM)/%.o: %.c
$(COMPILE_RECIPE)
build/$(PROGRAM)/%.o: $($(PROGRAM)_SOURCE_DIR)/%.c
$(COMPILE_RECIPE)
Ok for my current project, but would become ugly fast, if there where more than one directory different in the source paths of the executables
Edit: Test for suggestion of #Etan, which shows that $^ is empty, when the prerequisites appear only in pattern rules (y/y.c) - this only works if each dependency is given directly, as for x/x.c.
.PHONY: all
all: build/x.o build/y.o
build/x.o: x/x.c
# the following rule is ignored:
build/%.o: y/%.c
# because there is a matching pattern rule with a recipe:
build/%.o:
#echo \"$#\" from \"$^\"
#touch $#
The files x/x.c, y/y.c and the directory build exist.
output:
"build/x.o" from "x/x.c"
"build/y.o" from ""
Tested with GNU Make 3.82.90
First, it's never correct to use VPATH or vpath to find derived files (files that are built by make)... that includes "source files that are generated" (if they are generated by make, and not before make is invoked somehow). VPATH/vpath can only be used to find source files (files that are not built by make). See How not to use VPATH for more information.
Second, vpath is a global setting, not a per-target setting. Whenever make wants to find a prerequisite and that prerequisite doesn't exist "normally", make will use the vpath settings to find it. There's no facility in vpath to say "for this target use these vpath settings, for that target use those". In an ideal world, you would be able to set VPATH as a target-specific variable but this doesn't work.
There are various ways to generate prerequisites based on a target: you might find some of the suggestions on metaprogramming in make helpful.
Lastly, the makefile example you give regarding $^ is not correct:
# the following rule is ignored:
build/%.o: y/%.c
# because there is a matching pattern rule with a recipe:
build/%.o:
#echo \"$#\" from \"$^\"
#touch $#
You cannot specify a pattern rule without any recipe and have it "add" prerequisite patterns to some other existing pattern rule. It works for explicit rules, to add more prerequisites to a target, but specifying a pattern rule with no recipe simply deletes the pattern rule.
So, $^ is empty for build/y.o because there are no prerequisites defined in the pattern rule and you've not defined any explicit prerequisites for build/y.o like you did for build/x.o.
ETA: The reference to metaprogramming was to suggest that you can use it to make your second option, which is the one I would use, be less typing and easier maintenance. Pick the method you prefer and auto-generate the various pattern rules. This might be termed a "fifth option".
Other than that any of the four you suggest are fine, and there isn't another option you're missing that I'm aware of.
I'm using GNUmake to generate a deployment version of some web content from a source directory. I want to be rules that will run some files through compression tools (e.g. YUI compressor), and then for anything that doesn't have a rule, just copy it.
So for example:
# Rule with all the $(WWW_OUT_DIR)/*.* files as a prerequisite
all: $(WWW_OUT_FILES)
# Generic rule to perform compression on Javascript files.
$(WWW_OUT_DIR)/%.js: $(WWW_SRC_DIR)/%.js
$(YUI_COMP) $(YUI_COMP_OPTS) $< > $#
# Generic rule to perform compression on CSS files.
$(WWW_OUT_DIR)/%.css: $(WWW_SRC_DIR)/%.css
$(YUI_COMP) $(YUI_COMP_OPTS) $< > $#
# TDB Rule to magically handle everything else? (This doesn't work)
$(WWW_OUT_DIR)/%.%: $(WWW_SRC_DIR)/%.%
cp $< $#
How do I accomplish what that last rule is trying to do? I.e. For everything in $(WWW_OUT_FILES) that's not a .js or .css, just copy it? If possible, I want to retain the dependency on the corresponding input file.
You are almost right, the only thing to fix is the last pattern rule, just remove redundant percent symbol:
$(WWW_OUT_DIR)/%: $(WWW_SRC_DIR)/%
cp $< $#
Also keep in mind that as of GNU Make 3.82 pattern search algorithm has been modified a bit (from Changelog):
The pattern-specific variables and pattern rules are now applied in the
shortest stem first order instead of the definition order (variables
and rules with the same stem length are still applied in the definition
order). This produces the usually-desired behavior where more specific
patterns are preferred.
It is exactly what you want in case if you use the most recent version of Make. In order to get your Makefile compatible with other versions of GNU Make (including versions earlier than 3.82), the rule must be defined after other ones (as in the original question).
UPD.
A good example from here:
Prior to version 3.82, when gmake finds multiple matches during a pattern search, it prefers patterns declared earlier in the makefile over patterns declared later. As of 3.82, gmake instead prefers the pattern that results in the shortest stem. That sounds a bit confusing thanks to the jargon, but I think this will actually cause gmake to better adhere to the principle of least astonishment. Here’s an example:
all: sub/foo.x
%.x:
#echo "Prefer first match (stem is $*)."
sub/%.x:
#echo "Prefer most specific match (stem is $*)."
Compare the output from gmake 3.81 and 3.82:
gmake 3.81
Prefer first match (stem is sub/foo).
gmake 3.82
Prefer most specific match (stem is foo).
gmake 3.82 prefers the second pattern because it is a more specific match than the first. Note that this is a significant backwards-incompatibility compared with previous versions of gmake!