Why is make running a recipe before its prerequisites? - makefile

I have a make file with some pattern rules¹ that end up as prerequisites for another rule. A minimum example that shows the symptoms I'm puzzled by is this:
.PHONY: clean default one two
default: clean one two
clean:
#rm -f {one,two}.{a,b}
one two: %: %.a %.b
#echo TARGET $# PREREQUISITES $^
%.a %.b:
#echo Prereq $#
#touch $#
The output I expect when running this would be:
Prereq one.a
Prereq one.b
TARGET one PREREQUISITES one.a one.b
Prereq two.a
Prereq two.b
TARGET two PREREQUISITES two.a two.b
Instead, only the first prerequisite gets built, make gives me this:
Prereq one.a
TARGET one PREREQUISITES one.a one.b
Prereq two.a
TARGET two PREREQUISITES two.a two.b
As you can see, the recipe itself parses these correctly and knows that it should have been built after both prerequisites, but the prerequisite rule hasn't actually been run.
Incidentally, I'm using order-only prerequisites for the second item here but it doesn't really matter: the same problem is exhibited either way.
If I run the same target a second time, then it builds the second prerequisite. In other words two passes finally gets me the result I need:
$ make clean
$ make one
Prereq one.a
TARGET one PREREQUISITES one.a one.b
$ make one
Prereq one.b
TARGET one PREREQUISITES one.a one.b
The first time it runs, only the first prerequisite exists and I'm left with a broken one. On the second pass one.a exists, so it decides to build one.b. Now that both prerequisites exist my one builds properly.
If I spell out the prerequisite target variations as two separate recipes (repeat the same block for both %a: and %.b patterns separately) then both prerequisites get built for each target. However this would make my file quite a bit more complex and I'd like to understand why this breaks.
¹ This make file has some other problems but I've isolated this issue to an reproducible case so I don't think its other idiosyncrasies are the issue here.

GNU Make has what we may consider a little "quirk" in the sense that multiple-target pattern rules don't work exactly like several rules. In this situation, the rule is triggered only once by a single target, and Make expects the rule to make all targets at once. From the GNU Make documentation:
Pattern rules may have more than one target. Unlike normal rules, this does not act as many different rules with the same prerequisites and recipe. If a pattern rule has multiple targets, make knows that the rule’s recipe is responsible for making all of the targets. The recipe is executed only once to make all the targets. [...]
So what is happening here is:
checks the dependencies for one: finds one.a and one.b from the pattern %: %.a %.b;
finds one pattern rule for both one.a and one.b;
executes the recipe with target ($#) set to one.a (the one that triggered the rule);
marks both one.a and one.b as updated, even though the recipe only created one.a;
thinks all depencies for one have been satisfied and go on to do the same with two.
The reason why Make touches different files when you call it again is because now one.a and two.a are up-to-date. So the targets that trigger the rule become one.b and two.b. If you remove only one.a before calling Make a second time, it will create one.a and two.b.
So you have at least three possible solutions. One is to spell out the targets that behave like you'd expects—which you said is not a good solution, but it's worth mentioning here anyway. Another is to break that %.a %.b into two rules, which achieves the same but might be easier considering your needs. And the third is to do what Make expects and create both targets in the same recipe—e.g., you can use the stem in the recipe:
%.a %.b:
#echo Processing target $# from stem $*
touch $*.a $*.b
Another thing that I noticed while looking into this problem is that your MCVE's clean is broken. By default, Make will use /bin/sh, which will not expand things like {a,b}. You can set SHELL=/bin/bash to change that.

Related

Makefile: Pattern rule as first rule

I know that make usually executes the first target if called without any arguments. But what happens if the first target is a pattern rule? I have a Makefile here that looks as follows:
%.o: %.cc
gcc -c -o $# $<
main: main.o helper.o
gcc main.o helper.o -o $#
From my understanding of make, just calling it w/o any arguments should probably lead to some kind of error in this case because the first target, which is as far as I understood always the default target, does not make sense if make is not given any arguments. But when I call make with this Makefile, it instead builds the main target (and, of course, recursively the targets main.o and helper.o as well).
So, is it always true that make will ignore the pattern rules when looking for the first target? And is it somehow considered bad style to put those in front of the target that one really wants to be the default one? In my opinion, this is somehow confusing.
From the GNU make manual:
The order of rules is not significant, except for determining the
default goal: the target for make to consider, if you do not otherwise
specify one. The default goal is the target of the first rule in the
first makefile. If the first rule has multiple targets, only the first
target is taken as the default. There are two exceptions: a target
starting with a period is not a default unless it contains one or more
slashes, ‘/’, as well; and, a target that defines a pattern rule has
no effect on the default goal. (See Defining and Redefining Pattern
Rules.)

defer prerequisite expansion until after a (different) target creation

I want to be able use the result of a target created in a rule in the prerequisite of another rule in GNU make. So for example:
PREREQ = $(shell echo "reading target1" >&2; cat target1)
target1:
echo "prereq" > $#
target2: target1 $(PREREQ)
echo foo > $#
target2 should depend on prereq as read from the target1 file, but that is not in the file until the target1 recipe is executed.
Granted this is very contrived example with I am sure lots of suggestions about how to refactor this particular example but I'm not looking to refactor this example. This is just a simplified example of my more complicated problem where I need to derive prerequisites from the contents of a file that is not created until a recipe in the Makefile is executed.
The question is, [how] can I make expansion of $(PREREQ) (and therefore the execution of the $(shell cat target1) defer until after the target1 rule is actually executed?
Update: I tried .SECONDARYEXPANSION: but that doesn't seem to do the job:
$ make -d target2
...
reading target1
cat: target1: No such file or directory
...
Updating goal targets....
Considering target file 'target2'.
File 'target2' does not exist.
Considering target file 'target1'.
File 'target1' does not exist.
Finished prerequisites of target file 'target1'.
Must remake target 'target1'.
echo "prereq" > target1
[ child management ]
Successfully remade target file 'target1'.
Finished prerequisites of target file 'target2'.
Must remake target 'target2'.
echo foo > target2
[ child management ]
Successfully remade target file 'target2'.
As you can see, "reading target" was only printed once at the very beginning demonstrating that PREREQ is not expanded again due to the .SECONDEXPANSION: and the list of targets Considered for target2 did not include prereq.
Deferring the expansion of the prerequisite $(PREREQ) can be achieved by conditionally creating the target2 and relying on recursion:
ifndef expand-prereq
target2: target1
$(MAKE) --no-print-directory -f $(lastword $(MAKEFILE_LIST)) $# expand-prereq=y
else
target2: target1 $(PREREQ)
echo foo > $#
endif
The first time make runs for this makefile, the variable expand-prereq is not defined and therefore, the first targe2 rule is generated as a result of the conditional. This kind of dummy rule makes possible to update target1 without expanding $(PREREQ).
Matching this rule results in target1 being updated (since target1 is a prerequisite of this rule) and make being called recursively for the same makefile and with target2 as target.
The second time make is (recursively) invoked, the variable expand-prereq was defined by means of the command-line argument expand-prereq=y, so the second target2 rule is generated as a result of the else branch this time. This rule is the one that actually produces the target target2. Note that before this rule can be matched, target1 has been already created as a side effect of the first dummy rule, so the expansion of $(PREREQ) happens after target1 has been created (what you were looking for).
You could write the complete rule for target2 to a separate file and -include it:
Including Other Makefiles
How Makefiles Are Remade
The exact mechanics will depend on your specific use case, and it may well be impossible to achieve what we need using this approach, but it supports a variety of styles for automated dependency generation.
There are several solutions:
GNU make 4.4 has been released! Haven't tried yet, but the release notes claim that secondary expansion only expands the prerequisites when they're considered. Furthermore you can delay execution with the .WAIT special prerequisite. That works fine, I tested. If .WAIT really delays the second expansion of the prerequisites after .WAIT, you're good.
Recursive make. Restart make after the prerequisites for the second rule were updated. This is a bit lame solution, can't recommend it.
Produce the prerequisites into make include file(s). Make automatically restarts after updating include files (re-exec). I'm currently using this method, and it works great. Better than recursive make but still slow, as all the makefiles have to be parsed again. A possible solution is comparing the old and new prerequisites, and only updating the include file, if its content changed. The second rule also needs to be modified to do nothing, if the content changed. Make will run all rules before restarting, but if they don't update their targets, after the restart they'll be executed again, now with the proper prerequisites. An interesting feature of make is that you can define make variables inside a recipe, and use them in other recipes (but not in the prereq list, unless #1 above reall works).

Recipe that produces multiple targets

In our makefile, we have one recipe that links together all our object and library files to make an executable (an .elf file). As a side effect, this step also produces a map-file and an Intel .hex file:
$(ELF_FILE) : <list of dependencies here>
<linker command line>
Until now, since we never actually had a $(MAP_FILE) or a $(HEX_FILE) target, when ever another target depended on one of the $(ELF_FILE)'s side products, we simply declared it to be dependent on $(ELF_FILE), even if the recipe of that target didn't want to access the $(ELF_FILE) itself at all. For instance:
# Target that needs map-file, which is a side product of the $(ELF_FILE) target.
$(TARGET_THAT_NEEDS_MAP_FILE) : $(ELF_FILE)
<build-recipe>
# Target that needs hex-file, which is also a side product of the $(ELF_FILE) target.
$(TARGET_THAT_NEEDS_HEX_FILE) : $(ELF_FILE)
<build-recipe>
We have recently found out that a recipe can be used for more than one target, like so:
$(MAP_FILE) $(HEX_FILE) $(ELF_FILE) : <list of dependencies here>
<linker command line>
With this new-found knowledge, we figured we could get rid of the above "hack" and just directly state each target's direct dependencies:
$(TARGET_THAT_NEEDS_MAP_FILE) : $(MAP_FILE)
<build-recipe>
$(TARGET_THAT_NEEDS_HEX_FILE) : $(HEX_FILE)
<build-recipe>
Having implemented these changes, we now observe an odd effect that makes us suspect that we've either misunderstood this multiple-targets-one-recipe feature of make, or we're not using it correctly. The odd effect is that the recipe that produces the .elf, .map and .hex files now appears to run twice. This doesn't seem to have caused any immediate problems, but it does seem to indicate that something is fishy here. So my question, can our new approach work at all, or should we stick to the hack I described above?
EDIT: We're running our make in a multi-threaded manner (i.e. with -j).
It might be that when make is trying to update a target (whether it is $(MAP_FILE), $(HEX_FILE) or $(ELF_FILE), it does not know that its recipe will also update another target, therefore it starts a recipe for that one too, even if it's the same.
Of course, that would only happen when using the -j option. (Did you had the possibility to try without ?)
To illustrate :
$(TARGET): $(ELF_FILE) $(MAP_FILE)
<update target>
Here make will try to update $(ELF_FILE) and $(MAP_FILE) and fire the recipe twice. (That should also applies if the dependencies are on different target, as long as the targets are updated by a one execution of make and that there is no bottlenecks between them.
I'm not completely sure about that, though, make might be able to know that this is the same recipe.
======
This answer might be of use to you.
Specifically :
However, if your output files and your input file share a common base,
you CAN write a pattern rule like this:
%.foo %.bar %.baz : %.boz ; $(BUILDIT)
Strangely, for implicit rules with multiple targets GNU make assumes
that a single invocation of the recipe WILL build all the targets, and it will behave exactly as you want.
MadScientist
It refers to that part of the make manual :
Pattern rules may have more than one target. Unlike normal rules, this
does not act as many different rules with the same prerequisites and
recipe. If a pattern rule has multiple targets, make knows that the
rule’s recipe is responsible for making all of the targets. The recipe
is executed only once to make all the targets. When searching for a
pattern rule to match a target, the target patterns of a rule other
than the one that matches the target in need of a rule are incidental:
make worries only about giving a recipe and prerequisites to the file
presently in question. However, when this file’s recipe is run, the
other targets are marked as having been updated themselves.
EDIT:
Gnu Make has now gained a feature that would support this usecase (in version 4.3) : grouped explicit targets. It allows make to be aware that one recipe generate several targets, and it used like this (from the gnu make manual) :
foo bar biz &: baz boz
echo $^ > foo
echo $^ > bar
echo $^ > biz
foo, bar, and biz are generated by this rule (note the use of &: instead of :.
Full documentation : https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html ("Rules with Grouped Targets")

searchpath for prereqisites, like vpath, but only for some pattern rules

I'm trying to build several executables in one make instance, as suggested by
Recursive Make Considered Harmful.
The arguments of that paper apply to my project because some of my source files are generated, and go into multiple executables.
The object files for each of these executables go into separate directories, because they are compiled with different preprocessor flags.
How can I specify separate vpath settings for each of these executables, so that source files with duplicate filenames in separate directories go into the executables where I want them to go?
Setting vpath before the rules to build the object files for one executable, and erasing it afterwards (by not giving any directories) doesn't have the desired effect. Apparently, the last setting is used for all the rules.
Solutions I see currently:
Rename source filenames to use unique names so that I can use a global vpath setting
Instead of vpath, use separate rules for each source directory, with the source directory in the prerequisite pattern (*)
Recursive make, with separate vpath settings in each make instance, and somehow deal with the resulting trouble.
Use something different than make.
Is there a better solution, or which one of the above would you prefer?
(*) The solution with separate rules looks like this:
build/$(PROGRAM)/%.o: %.c
$(COMPILE_RECIPE)
build/$(PROGRAM)/%.o: $($(PROGRAM)_SOURCE_DIR)/%.c
$(COMPILE_RECIPE)
Ok for my current project, but would become ugly fast, if there where more than one directory different in the source paths of the executables
Edit: Test for suggestion of #Etan, which shows that $^ is empty, when the prerequisites appear only in pattern rules (y/y.c) - this only works if each dependency is given directly, as for x/x.c.
.PHONY: all
all: build/x.o build/y.o
build/x.o: x/x.c
# the following rule is ignored:
build/%.o: y/%.c
# because there is a matching pattern rule with a recipe:
build/%.o:
#echo \"$#\" from \"$^\"
#touch $#
The files x/x.c, y/y.c and the directory build exist.
output:
"build/x.o" from "x/x.c"
"build/y.o" from ""
Tested with GNU Make 3.82.90
First, it's never correct to use VPATH or vpath to find derived files (files that are built by make)... that includes "source files that are generated" (if they are generated by make, and not before make is invoked somehow). VPATH/vpath can only be used to find source files (files that are not built by make). See How not to use VPATH for more information.
Second, vpath is a global setting, not a per-target setting. Whenever make wants to find a prerequisite and that prerequisite doesn't exist "normally", make will use the vpath settings to find it. There's no facility in vpath to say "for this target use these vpath settings, for that target use those". In an ideal world, you would be able to set VPATH as a target-specific variable but this doesn't work.
There are various ways to generate prerequisites based on a target: you might find some of the suggestions on metaprogramming in make helpful.
Lastly, the makefile example you give regarding $^ is not correct:
# the following rule is ignored:
build/%.o: y/%.c
# because there is a matching pattern rule with a recipe:
build/%.o:
#echo \"$#\" from \"$^\"
#touch $#
You cannot specify a pattern rule without any recipe and have it "add" prerequisite patterns to some other existing pattern rule. It works for explicit rules, to add more prerequisites to a target, but specifying a pattern rule with no recipe simply deletes the pattern rule.
So, $^ is empty for build/y.o because there are no prerequisites defined in the pattern rule and you've not defined any explicit prerequisites for build/y.o like you did for build/x.o.
ETA: The reference to metaprogramming was to suggest that you can use it to make your second option, which is the one I would use, be less typing and easier maintenance. Pick the method you prefer and auto-generate the various pattern rules. This might be termed a "fifth option".
Other than that any of the four you suggest are fine, and there isn't another option you're missing that I'm aware of.

What is the behaviour of an order-only prerequisite where the prerequisite is a phony target?

I am wondering if the order-only prerequisite loses its order-only precedence if it is a phony target. Consider the following:
%.make: unpack_chroot
schroot $(CHROOT) make $*
%.copy: | unpack_chroot
rsync -a input/$*/ $(CHROOT)/input/$*/
unpack_chroot: input/chroot.tar.gz
mkdir -p $(CHROOT)
tar -C $(CHROOT) -zxf $<
.PHONY: unpack_chroot
All %.make and %.copy targets are .PHONY. Some of these targets depend on files being copied into the chroot, others do not. Those that do are defined with explicit dependencies:
a.make: a.copy
c.make: c.copy
However, if unpack_chroot is not an order-only prerequisite and was unpacked as part of the processing of prerequisites for another make target not in the same make process, unpack_chroot will be considered up-to-date when %.copy runs and will not remake %.copy; at least that is what I have seen. Currently, unpack_chroot is not phony and gets created. I want to make it phony, but want to clarify the behaviour.
According to this email ( http://kolpackov.net/pipermail/notes/2004-January/000001.html ) it seems that the only difference between normal and order only prerequisites is that the latter will not be included in automatic make variables such as $^.
From some tests I ran it seems that this is the only difference. In all other respects they are treated the same as normal prerequisites.

Resources