Background:
We have a Makefile that sits in the middle of a dev-ops pipeline that does things in unusual ways, for legacy reasons not worth describing. This has worked fine for years, but upon upgrading to GNU Make 4.4, it started generating a new warning of the form:
Makefile:X: warning: pattern recipe did not update peer target 'Y'.
I'm 99% sure this warning is harmless for our use case, but the new warning output is tripping CI failures in our pipeline. This Makefile is invoked by an automated script and the output is also parsed by an automated script, neither of which can easily be changed.
Minimal reproducer:
Makefile:
%-seq %-par :: %.cpp force
$(MAKE) do_complicated_stuff SRC=$< TGT=$#
do_complicated_stuff:
#echo doing complicated stuff with SRC=$(SRC) TGT=$(TGT)
touch $(TGT)
%-seq :: %.c force
echo Error: this rule should not be run in this MRE
exit 1
.PHONY: force
Command:
$ rm -f *-{seq,par} ; touch foo.cpp ; make --no-print-directory foo-seq
make do_complicated_stuff SRC=foo.cpp TGT=foo-seq
doing complicated stuff with SRC=foo.cpp TGT=foo-seq
touch foo-seq
Makefile:2: warning: pattern recipe did not update peer target 'foo-par'.
Here make was invoked to build the foo-seq target, but it's complaining about the foo-par target which does not exist and was not even mentioned. A given make command will be invoked to build exactly one target at a time, and the .PHONY dependency ensures the rule will be run (regardless of whether or not Make considers dependencies to be up-to-date). Make will be invoked (by the script) many times in the same directory to build each test.
Question:
Why is GNU Make 4.4 suddenly generating this new warning for an idiom that has always silently worked correctly in the past, and what is the minimal change to silence this harmless warning?
Constraints/Requirements:
The solution probably needs to involve a pattern rule of some kind, because the set of possible source file names cannot be encoded in the Makefile. The name is provided only on the command line (which cannot be changed), but the rule needs to match it against the existence of the source file to ensure the correct rule is selected and executed.
In the real Makefile both of the rules modeled above are much more complicated than shown here, so we'd like to avoid duplicating the first rule for each of the target patterns (which does silence the warning, but causes maintainability problems).
Finally, for portability reasons the solution needs to continue functioning correctly without new warnings for all versions of GNU Make back to v3.80 (possibly negotiable if there's really no better solution).
The current plan is that in the next release of GNU make, this will become an error not a warning. So you should address it now. The reason for the change is that there are other errors in how GNU make handles patterns that cannot be fixed without changing this.
The problem is this rule:
%-seq %-par :: %.cpp force
$(MAKE) do_complicated_stuff SRC=$< TGT=$#
I'm not sure what you intend for this rule to actually do, but what it tells make it will do is a single invocation of this recipe will create both targets %-seq and %-par. That's what multiple patterns in a single rule means, and has always meant.
If your recipe does not actually build both of those targets then you will see this issue.
The simplest thing to do is write the rule twice, once for each target, which will work on all versions of GNU make and was always the correct way to write it:
%-seq :: %.cpp force
$(MAKE) do_complicated_stuff SRC=$< TGT=$#
%-par :: %.cpp force
$(MAKE) do_complicated_stuff SRC=$< TGT=$#
ETA
Wanting to avoid "duplicating the recipe" is trivial enough. Just put it into a variable:
define COMPLEX_RECIPE
$(MAKE) do_complicated_stuff SRC=$< TGT=$#
endef
%-seq :: %.cpp force ; $(COMPLEX_RECIPE)
%-par :: %.cpp force ; $(COMPLEX_RECIPE)
Related
I have these recipes in my Makefile. They generate cross-compiled objects for ARM architecture and link them into an elf binary:
%.ao: %.c
$(ARM_CC) $(ARM_CPPFLAGS) $(ARM_FLAGS) $(CFLAGS) -c -o $# $<
%.elf: %.ao startup_stm32f0xx.ao system_stm32f0xx.ao
$(ARM_CC) $(ARM_FLAGS) $other_arguments -o $# $^
This works fine from a clean build.
Contrary to my expectation, if I then say touch foo.c; make foo.elf, gmake responds with
make: 'foo.elf' is up to date.
If I try to make foo.ao, gmake says that it, too , is up to date.
What am I missing?
Edit after reading the comments:
TLDR: I did have multiple rules matching the same target, as John Bollinger alluded and HardcoreHenry said specifically.
In addition to the rules above, there's a rule for assembly sources so I can use those vendor files:
%.ao: %.s
$(ARM_CC) $(ARM_CPPFLAGS) $(ARM_FLAGS) $(CFLAGS) -c -o $# $<
I had been debugging some macros, and used -save-temps to look at preprocessor output. This option also writes .s files. So after I'd run make foo.elf, I'd have the following in my directory:
foo.c
foo.i
foo.s
foo.ao
foo.elf
I can touch foo.c, but make sees that there's a foo.s which is older than foo.ao, and produces the output that it does. On a clean build, there is no foo.s, so make finds the %.c:%.ao rule and the build proceeds from foo.c.
(BTW, .ao stands for ARM object. In addition to cross-compiling for AMR, I compile many of the sources to run unit tests on the host, using the built-in .o:.c rule)
I'm not a fan of pattern rules.
Make can make very strange decisions on which rules apply depending on whatever is lying around on your hard disks.
It's all a bit arbitrary.
Much better IMHO to tell make exactly what files you need for a target.
It's pretty easy too.
Just prefix your pattern rule with the list of targets you actually want it to apply to.
This makes it a Static Pattern Rule.
objects := main.ao tools.ao devices.ao# etc
${objects}: %.ao: %.c
$(ARM_CC) $(ARM_CPPFLAGS) $(ARM_FLAGS) $(CFLAGS) -c -o $# $<
%.elf: ${objects} startup_stm32f0xx.ao system_stm32f0xx.ao
$(ARM_CC) $(ARM_FLAGS) $other_arguments -o $# $^
As an added bonus, make now won't try to create the pre-existing startup_stm32f0xx.ao and system_stm32f0xx.ao.
Usually I find it nicer to list the source files, but YMMV:
sources := main.c tools.c devices.c
objects := $(patsubst $.c,%.ao,${sources})
(P.S. Using a Static Pattern Rule doesn't really give you any advantage over a normal rule in this noddy case. I just wanted to show a small tweak that would make your makefiles much more consistent in their behaviour.)
I know it's bad form to use an answer to respond to another answer, but I ran out of space in a comment to #bobbogo's answer.
Sorry but I can't agree with your assessment of pattern rules. It's not true that you will get "strange decisions" based on "whatever is lying around on your harddisks", and it's certainly not arbitrary.
There is one advantage of static pattern rules over pattern rules, and that is also its downside: a static pattern rule is a shorthand for creating an explicit rule, so that rule will always be used to build that target. A pattern rule, on the other hand, is just one possible way to build a target: if the prerequisites of a pattern rule don't exist and can't be made, then make keeps going and looks for other pattern rules that might be able to build that target.
So if you have multiple possible ways you can build a target then an explicit rule cannot be used for that.
The problem with pattern rules is that if NO pattern rule applies then make just assumes there is no rule to build that target. If the target exists then make simply says "up to date" (as we see in the question) since there's no rule to build it. That can be confusing to users.
If you use an explicit rule (including a static pattern rule) and some prerequisite doesn't exist and can't be created, then make will exit with an error, which can make it easier to figure out what went wrong.
In a Makefile with
%.o: %.c
#echo Compiling $< ...
I'm getting the warning '%'-style pattern rules are a GNU make extension when I run autoreconf --install (of autoconf version 2.69). The warning is not very clear, but makes me think that there's something to be added to configure.ac.
I conducted searches on google.com, duckduckgo.com and yahoo.com, but they all don't seem to be able to differentiate between the large number of build logs they index (why should they...) which makes the search painful. I figured that:
I can silence the warning by adding AM_INIT_AUTOMAKE([-Wno-portability]) to configure.ac (found in a post of the openais mailing list) which seems not great because simply silencing a warning is generally not a good idea in a technical environment - please tell me if GNU autotools is an exception.
Replace
%.o: %.c
with
.c.o:
That's a suffix rule doing the same thing, and it's more portable. If your rule involves a suffix that is not known to make, list it in the prerequisites for the special .SUFFIXES target:
.SUFFIXES: .hack .win
.hack.win:
# build a .win file from a .hack file.
More on how this works in detail here. They recommend to use pattern rules instead of suffix rules because they're clearer and more general, which is true, but as autoconf notes, they are indeed less portable. So if that is a worry (if you want to build on BSD/Mac OS and not install GNU make, basically), fall back on the "old-fashioned suffix rules."
If you have a pattern rule that cannot be replaced by a suffix rule, another possible replacement that automake doesn't complain about is a static pattern rule. This is similar to a pattern rule but requires a list of targets it applies to. Instead of saying
%.o: %.c
You would have to say
OBJS = foo.o bar.o baz.o # list all .o files here
$(OBJS): %.o: %.c
Or more generally,
target-pattern: prerequisite-pattern
is replaced by
target-list: target-pattern: prerequisite-pattern
Winteermute covered almost all aspects of the issue. I will add one of my own frustrations here. In my make file, I have been using a pattern rule to install shell scripts without their .sh suffix. Automake does not like pattern rules and give you warning. You can simply ignore the warning for now. My configure.ac file:
AM_INIT_AUTOMAKE([-Wall -Wportability])
In my Makefile.am
dist_bin_SCRIPTS = foobar
# pattern rule
% : %.sh
cp $< $#
chmod +x $#
In my script directory I have the foobar.sh file.
I have not been able to figure out a better way to specify a suffix rule to cover the case where the target does not have an extension although I really tried hard to search on the internet and read the manuals backward a few times.
The reason I am distributing the *.sh file is that the user does not have to know the implementation details. They just need to remember the name of the executable. Furthermore, asking the user to type the extra .sh is really a waste of life.
Is there a way how to ask gmake to never run two targets from a set in parallel?
I don't want to use .NOTPARALLEL, because it forces the whole Makefile to be run sequentially, not just the required part.
I could also add dependencies so that one depends on another, but then (apart from being ugly) I'd need to build all of them in order to build the last one, which isn't necessary.
The reason why I need this is that (only a) part of my Makefile invokes ghc --make, which takes care of its dependencies itself. And it's not possible to run it in parallel on two different targets, because if the two targets share some dependency, they can rewrite each other's .o file. (But ghc is fine with being called sequentially.)
Update: To give a specific example. Let's say I need to compile two programs in my Makefile:
prog1 depends on prog1.hs and mylib.hs;
prog2 depends on prog2.hs and mylib.hs.
Now if I invoke ghc --make prog1.hs, it checks its dependencies, compiles both prog1.hs and mylib.hs into their respective object and interface files, and links prog1. The same happens when I call ghc --make prog2.hs. So if they the two commands get to run in parallel, one will overwrite mylib.o of the other one, causing it to fail badly.
However, I need that neither prog1 depends on prog2 nor vice versa, because they should be compilable separately. (In reality they're very large with a lot of modules and requiring to compile them all slows development considerably.)
Hmmm, could do with a bit more information, so this is just a stab in the dark.
Make doesn't really support this, but you can sequential-ise two targets in a couple of ways. First off, a real use for recursive make:
targ1: ; recipe1...
targ2: ; recipe2...
both-targets:
${MAKE} targ1
${MAKE} targ2
So here you can just make -j both-targets and all is fine. Fragile though, because make -j targ1 targ2 still runs in parallel. You can use dependencies instead:
targ1: ; recipe1...
targ2: | targ1 ; recipe2...
Now make -j targ1 targ2 does what you want. Disadvantage? make targ2 will always try to build targ1 first (sequentially). This may (or may not) be a show-stopper for you.
EDIT
Another unsatisfactory strategy is to explicitly look at $MAKECMDGOALS, which lists the targets you specified on the command-line. Still a fragile solution as it is broken when someone uses dependencies inside the Makefile to get things built (a not unreasonable action).
Let's say your makefile contains two independent targets targ1 and targ2. Basically they remain independent until someone specifies on the command-line that they must both be built. In this particular case you break this independence. Consider this snippet:
$(and $(filter targ1,${MAKECMDGOALS)),$(filter targ2,${MAKECMDGOALS}),$(eval targ1: | targ2))
Urk! What's going on here?
Make evaluates the $(and)
It first has to expand $(filter targ1,${MAKECMDGOALS})
Iff targ1 was specified, it goes on to expand $(filter targ2,${MAKECMDGOALS})
Iff targ2 was also specified, it goes on to expand the $(eval), forcing the serialization of targ1 and targ2.
Note that the $(eval) expands to nothing (all its work was done as a side-effect), so that the original $(and) always expands to nothing at all, causing no syntax error.
Ugh!
[Now that I've typed that out, the considerably simpler prog2: | $(filter prog1,${MAKECMDGOALS})
occurs to me. Oh well.]
YMMV and all that.
I'm not familiar with ghc, but the correct solution would be to get the two runs of ghc to use different build folders, then they can happily run in parallel.
Since I got stuck at the same problem, here is another pointer in the direction that make does not provide the functionality you describe:
From the GNU Make Manual:
It is important to be careful when using parallel execution (the -j switch; see Parallel Execution) and archives. If multiple ar commands run at the same time on the same archive file, they will not know about each other and can corrupt the file.
Possibly a future version of make will provide a mechanism to circumvent this problem by serializing all recipes that operate on the same archive file. But for the time being, you must either write your makefiles to avoid this problem in some other way, or not use -j.
What you are attempting, and what I was attempting (using make to insert data in a SQLite3 database) suffers from the exact same problem.
I needed to separate the compilation from other steps (cleaning, building dirs and linking), as I wanted to run the compilation with more core processes and the -j flag.
I managed to solve this, with different makefiles including and calling each other. Only the "compile" make file is running in parallel with all the cores, the rest of the process is syncronous.
I divided my makefile in 3 separate scripts:
settings.mk: contains all the variables and flag definitions
makefile: has all the targets except the compilation one (It has .NOTPARALLEL directive). It calls compile.mk with -j flag
compile.mk: contains only the compile operation (without .NOTPARALLEL)
In settings.mk I have:
CC = g++
DB = gdb
RM = rm
MD = mkdir
CP = cp
MAKE = mingw32-make
BUILD = Debug
DEBUG = true
[... all other variables and flags needed, directories etc ...]
In makefile I have Link and compilation target as these:
include .makefiles/settings.mk
[... OTHER TARGETS (clean, directories etc)]
compilation:
#echo Compilation
#$(MAKE) -f .makefiles/compile.mk --silent -j 8 -Oline
#Link
$(TARGET): compilation
#echo -e Linking $(TARGET)
#$(CC) $(LNKFLAGS) -o $(TARGETDIR)/$(TARGET) $(OBJECTS) $(LIBDIRS) $(LIB)
#Non-File Targets
.PHONY: all prebuild release rebuild clean resources directories run debug
.NOTPARALLEL: all
# include dependency files (*.d) if available
-include $(DEPENDS)
And this is my compile.mk:
include .makefiles/settings.mk
#Defauilt
all: $(OBJECTS)
#Compile
$(BUILDDIR)/%.$(OBJEXT): $(SRCDIR)/%.$(SRCEXT)
#echo -e Compiling: $<
#$(MD) -p $(dir $#)
#$(CC) $(COMFLAGS) $(INCDIRS) -c $< -o $#
#Non-File Targets
.PHONY: all
# include dependency files (*.d) if available
-include $(DEPENDS)
Until now, it's working.
Note that I'm calling compile.mk with -j flag AND -Oline so that parallel processing doesn't mess up with the output.
Any syntax color can be setted in the makefile main script, since the -O flag invalidates escape color codes.
I hope it can help.
I had a similar problem so ended up solving it on the command line, like so:
make target1; make target2
to force it to do the targets sequentially.
I tried to write a make rule of this form:
lib%.so: computations/%.h
make -C computations/ -f makefile $<
Output:
make -C computations/ -f makefile computations/test.h
make[1]: Entering directory `/home/shai/csm/csm2/src/computations'
make[1]: *** No rule to make target `computations/test.h'. Stop.
make[1]: Leaving directory `/home/shai/csm/csm2/src/computations'
make: *** [libtest.so] Error 2
Well, it seems that it looks for %.h after switching library. No biggie, I'll just try removing
lib%.so: %.h
make -C computations/ -f makefile $<
but now it doesn't recognize the rule at all!
Output:
shai#ubuntu:~/csm/csm2/src$ make libtest.so
`make: *** No rule to make target `libtest.so'. Stop.
Is make playing with my mind? How could a change in the dependencies make it stop recognizing the target?
How could a change in the dependencies make it stop recognizing the target?
Most likely because it can't find a file that fits the pattern of the dependency, because test.h is in another directory. You could use VPATH to have it search for files in other directories.
Edit: but this still won't fix your problem completely, because $< will be substituted by computations/test.h, which won't be found in the directory computations (as in the first error you got). You might try $(notdir $<), but I think it's an ugly hack (if it works; I haven't tried it). This results from the design of your makefile and project structure.
You're using make wrong.
Make's design is actually fairly simple at heart: you tell it
how to create targets (The body of the rule).
which targets it needs to create first (The dependency list).
Make then topological sorts the dependency information so it executes the bodies in the right order.
Your rule with $< (which expands to the list of dependencies) means you're telling the invoked make to create the dependency %.h not the target lib%.so.
The rule has to create the target. If the dependency needs to be created, you need to tell make how to do so with another, separate rule, not try to ensure it is up to date in this rule.
As you're trying to coordinate across directories, I'll recommend the paper Recursive make considered harmful, which can often be a nice way to organize things, though not the only possible way to do things.
It might just be the way you copy/pasted it, but you're missing a tab in the second form.
When I change a Makefile, its rules may have changed, so they should be reevaluated, but make doesn't seem to think so.
Is there any way to say, in a Makefile, that all of its targets, no matter which, depend on the Makefile itself?
(Regardless of its name.)
I'm using GNU make.
This looks like one more simple, useful, logical thing that Make should be able to do, but isn't.
Here is a workaround. If the clean rule is set up correctly, Make can execute it whenever the makefile has been altered, using an empty dummy file as a marker.
-include dummy
dummy: Makefile
#touch $#
#$(MAKE) -s clean
This will work for most targets, that is targets that are actual files and that are removed by clean, and any targets that depend on them. Side-effect targets and some PHONY targets will slip through the net.
Since GNU make version 4.3 it is now possible with the use of those two special variable:
.EXTRA_PREREQS
To add new prerequisite to every target
MAKEFILE_LIST
To get the path of the make file
To have every target depend on the current make file:
Put near the top of the file (before any include since it would affect the MAKEFILE_LIST) the following line:
.EXTRA_PREREQS:= $(abspath $(lastword $(MAKEFILE_LIST)))
To have every target depend on the current make file and also the make files which were included
Put the following line at the end of your file:
.EXTRA_PREREQS+=$(foreach mk, ${MAKEFILE_LIST},$(abspath ${mk}))
The only answer I know to this is to add makefile explicitly to the dependencies. For example,
%.o: %.c makefile
$(CC) $(CFLAGS) -c $<