According to the gnu make documentation, if a rule generates multiple targets by a single invocation (for instance with a recipe executing a tool with multiple output files), you can use the '&:' rule syntax to tell make.
I'm getting warnings when using this syntax, about target '&', however, when having multiple (but unique) targets in multiple rules. As if make mistakes the ampersand for a target name instead of being part of the target-prerequisite separator.
In my original project I've got two rules having multiple targets and a recipe generating those targets from a single statement/tool. The targets are unique for each of the two rules. I've created the following simple example to demonstrate the warning generated by make:
all: file_abbc
.PHONY: all clean
clean:
del /Q file_*
file_abbc: file_ab file_bc
copy file_ab+file_bc file_abbc
file_ab file_bc &: file_a file_b file_c
copy file_a+file_b file_ab
copy file_b+file_c file_bc
file_a file_b file_c &: content
copy content file_a
copy content file_b
copy content file_c
Warnings from running make on Windows on the above:
Makefile:17: warning: overriding recipe for target '&'
Makefile:13: warning: ignoring old recipe for target '&'
Why is make complaining about target '&' ?
You're using an old version of GNU make that doesn't understand &: rules, so just treats & as a file name. Use make --version to see which version you are running and upgrade if it is older that 4.3
Related
I have this version of makefile
[sbsuser#compute-00-01 415]$ make --version GNU Make 3.81
I have directory SOMATIC where I have 3 file . I want to produce a only one output. This is what I wrote.
`
OUTSOMATIC=SOMATIC
FINAL=FINAL
INPUT=$(wildcard $(OUTSOMATIC)/*.vcf)
OUTSORT2= $(patsubst $(OUTSOMATIC)/%.vcf,$(FINAL)/%somatic.ensemble.gz,$(INPUT))
$(info lista $(OUTSORT2))
$(info lista $(INPUT))
.PHONY: all
all: $(INPUT) $(OUTSOMATIC) $(OUTSORT2) $(FINAL)
$(FINAL)/%somatic.ensemble.gz: $(OUTSOMATIC)/%.vcf $(INPUT)
~/jdk1.8.0_121/bin/java -XX:+UseSerialGC -Xms1g -Xmx10g -jar /illumina/software/PROG2/bcbio-variation-recall-0.1.7 ensemble -n 1 $(FINAL)/somatic_ensemble.gz /illumina/software/database/database_2016/hg19_primary.fa $^
`
With this script make 3 time the same files. I don't understand how to create only one output from list of input to use in the same time.
What is the best way to do this?
If I change $(FINAL)/%somatic.ensemble.gz: in $(FINAL)/somatic.ensemble.gz I have this error:
make: *** No rule to make target FINAL/415_merge_mutect2.somaticsomatic.ensemble.gz', needed byall'. Stop`
You probably should review the GNU make manual introductory sections where they describe how make works.
Let's look at your makefile; first you define some variables. Let's assume that you have the files SOMATIC/foo.vcf, SOMATIC/bar.vcf, and SOMATIC/baz.vcf. Then the variables you created will have these values, after they are expanded:
OUTSOMATIC = SOMATIC
FINAL = FINAL
INPUT = SOMATIC/foo.vcf SOMATIC/bar.vcf SOMATIC/baz.vcf
Now your patsubst finds all words in INPUT that match the pattern SOMATIC/%.vcf and replace that with FINAL/%somatic.ensemble.gz, where the part that matches the % in the input is substituted into the output:
OUTSORT2 = FINAL/foosomatic.ensemble.gz FINAL/barsomatic.ensemble.gz FINAL/bazsomatic.ensemble.gz
Now, make sees that you've defined an all target. Since it's the first target in the makefile this is the target that will be run by default. After expansion, it will look like this:
all: SOMATIC/foo.vcf SOMATIC/bar.vcf SOMATIC/baz.vcf SOMATIC FINAL/foosomatic.ensemble.gz FINAL/barsomatic.ensemble.gz FINAL/bazsomatic.ensemble.gz FINAL
So, make will try to build every prerequisite of the all target to be sure it's up to date. First it tries to build the SOMATIC/*.vcf files. Those files already exist and make doesn't have any rules about how to rebuild them, so it assumes they're up to date.
Next it tries to build the SOMATIC file. This is a directory and it also has no rule to be built, so make assumes that's up to date as well.
Next make tries to build the target FINAL/foosomatic.ensemble.gz. Make does have a rule that can build it, you've created one:
$(FINAL)/%somatic.ensemble.gz: $(OUTSOMATIC)/%.vcf $(INPUT)
~/jdk1.8.0_121/bin/java ...
This matches the target you want to build, with a % value of foo, so then make substitutes the % in the prerequisite for foo and finds that SOMATIC/foo.vcf exists and doesn't need to be rebuilt, so it runs your recipe. However your recipe doesn't actually create the target FINAL/foosomatic.ensemble.gz; it creates the target FINAL/somatic_ensemble.gz. So this rule is broken because it tells make it will do one thing, but it does something else.
You should always ensure all your recipes build the file represented by the automatic variable $#; that will ensure that you and make agree on the meaning of your rule. If you want your recipe to build some other file, then your rule is written incorrectly.
Next make does the same thing with the next prerequisite of all: FINAL/barsomatic.ensemble.gz. Since that file doesn't exist, make tries to build it using the pattern rule, but again that creates the same output file.
And again for the third .gz file FINAL/bazsomatic.ensemble.gz. That's why things are run three times.
If you change the pattern rule to an explicit rule building FINAL/somatic.ensemble.gz, which is what you want, then make can't find any way to build the prerequisites of the all target so it gives this error.
Your problem is the creation of OUTSORT2. You want to create only one output file, but you've set OUTSORT2 to contain three different files, so make tries to create all three files. You want this:
OUTSOMATIC = SOMATIC
FINAL = FINAL
INPUT = $(wildcard $(OUTSOMATIC)/*.vcf)
OUTSORT2 = $(FINAL)/somatic.ensemble.gz
.PHONY: all
all: $(OUTSORT2)
$(OUTSORT2): $(INPUT)
~/jdk1.8.0_121/bin/java -XX:+UseSerialGC -Xms1g -Xmx10g -jar /illumina/software/PROG2/bcbio-variation-recall-0.1.7 ensemble -n 1 $# /illumina/software/database/database_2016/hg19_primary.fa $^
I have another make-like tool that produces an XML as an artifact after parsing my makefile which I'll then further process with Python.
It'd simplify things for me - a lot - if I could have make consider every single prerequisite to be an actual target because then this other tool
will classify each and every file as a "job".
This is a fragment of my makefile:
.obj/eventlookupmodel.o: C:/Users/User1/Desktop/A/PROJ/src/AL2HMIBridge/LookupModels/eventlookupmodel.cpp C:\Users\User1\Desktop\A\PROJ\src\AL2HMIBridge\LookupModels\eventlookupmodel.h \
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qabstractitemmodel.h \
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qvariant.h \
...
I'd want for make to think I have a dummy rule for each prerequisite such as below:
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qvariant.h:
#echo target pre= $#
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qabstractitemmodel.h:
#echo target pre=$#
C:/Users/User1/Desktop/A/PROJ/src/AL2HMIBridge/LookupModels/eventlookupmodel.cpp :
#echo target pre=$#
C:\Users\User1\Desktop\A\PROJ\src\AL2HMIBridge\LookupModels\eventlookupmodel.h:
#echo target pre=$#
I don't care about the exact form of the rule just that each file is considered an actual target.
My method of passing in this rule would be by setting the MAKEFILES variable like so
make all MAKEFILES=Dummy.mk
with Dummy.mk containing this rule so that I do not modify the makefiles.
I've tried the following so far.
Dummy.mk:
%.h:
#echo header xyz = $#
%:
#echo other xyz= $#
This partially works.
I run make all --trace --print-data-base MAKEFILES=Dummy.mk and I can see that
make does "bind" the %.h: rule to the header files. In the --print-data-base section, I see that rule being assigned to the header files.
C:/Users/User1/Desktop/A/QNX_SDK/target/qnx6/usr/include/stddef.h:
# Implicit rule search has been done.
# Implicit/static pattern stem: 'C:/Users/User1/Desktop/A/QNX_SDK/target/qnx6/usr/include/stddef'
# Last modified 2016-05-27 12:39:16
# File has been updated.
# Successfully updated.
# recipe to execute (from '#$(QMAKE) top_builddir=C:/Users/User1/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/ top_srcdir=C:/Users/User1/Desktop/A/HMI_FORGF/ -Wall CONFIG+=release CONFIG+=qnx_build_release_with_symbols CONFIG+=rtc_build -o Makefile C:/Users/User1/Desktop/A/HMI_FORGF/src/HmiLogging/HmiLogging.pro
', line 2):
#echo header xyz = $#
However, I do NOT see the "echo header xyz $#"-rule being executed.
Regarding the %: rule, it is neither executed for the .cpp files nor "bound" to them in the --print-data-base section.
However, it is bound and executed for existing targets which have no suffix i.e.
all: library binary
binary: | library
ifs: | library
For the %: rule, the reason for this behavior is because of 10.5.5 Match-Anything Pattern Rules: If you do not mark the match-anything rule as terminal, then it is non-terminal. A non-terminal match-anything rule cannot apply to a file name that indicates a specific type of data. A file name indicates a specific type of data if some non-match-anything implicit rule target matches it.
If I make it non-terminal - no double colon - then the rule doesn't apply to built-in types like .cppunless I un-define the built-in rules that negate my intended %: rule.
If I make it terminal, "it does not apply unless its prerequisites actually exist". But a .h or .cpp doesn't technically have prerequisites; can I just create a dummy file and have that as its prerequisite?
NOTE: This has NOTHING to do with gcc -M generation. Yes the -M option would help in the specific case of header and source files but this question is for more generic targets and prerequisites that already exist in the makefile when make is launched.
This may take a few iterations. Try:
%.h: null
#echo header xyz = $#
%: null
#echo other xyz= $#
null:
#:
Try generating static pattern rules for the header files. See one of the answers to Make ignoring Prerequisite that doesn't exist.
Static pattern rules only apply to an explicit list of target files like this:
$(OBJECTS): %.o: %.c
*recipe here*
where the variable OBJECTS is defined earlier in the makefile to be a list of target files (separated by spaces), for example:
OBJECTS := src/fileA.c src/fileB.c src/fileC.c
Note that you can use the various make utility functions to build that list of target files. For example, $(wildcard pattern), $(addsuffix), etc.
You should also ensure that the recipe "touches" the header file to change the timestamp.
I've found that using static pattern rules instead of pattern rules fixes problems where make doesn’t build prerequisites that don’t exist, or deletes files that you want.
Here is an example of using wildcard to copy files from one directory to another.
# Copy images to build/images
img_files := $(wildcard src/images/*.png src/images/*.gif src/images/*.jpg \
src/images/*.mp3)
build_images := $(subst src/,$(BUILD_DIR)/,$(img_files))
$(build_images): $(BUILD_DIR)/images/% : src/images/%
mkdir -p $(dir $#)
cp -v -a $< $#
There are other make functions like addprefix that could be used to generate a more complex file specification.
This question is based on another question of mine here: Getting basename and notdir to work in prerequisite (dependency) list.
I'm using a Makefile to generate some figures automatically and
efficiently.
My figures are generated in ../thesis/figures using Octave .m
files that are in the current directory where the Makefile also is.
Each .m file, e.g. figure1.m, may generate several figures, e.g.
figure1.p1.tex and figure1.p2.tex (and their dependecies, which
are also generated by figure1.m). These .tex files are then to be compiled using LaTeX (a single run of pdflatex figure1.p1.tex suffices in this case; so, there is no need for latexmk or other alternatives).
The Makefile I have so far is
OCTAVE = octave --jit-compiler --no-gui --quiet
PDFLATEX = pdflatex
FIGDIR = ../thesis/figures
TEXTARGETS = $(wildcard $(FIGDIR)/*.tex)
.PHONY: figures
figures: $(TEXTARGETS)
.SECONDEXPANSION:
$(TEXTARGETS): %.tex : $$(basename $$(notdir %)).m
$(OCTAVE) $<
$(PDFLATEX) $#
A dry run with make -n shows me
octave --jit-compiler --no-gui --quiet figure1.m
pdflatex ../thesis/figures/figure1.p1.tex
octave --jit-compiler --no-gui --quiet figure1.m
pdflatex ../thesis/figures/figure1.p2.tex
make: *** No rule to make target `figure2.m', needed by `../thesis/figures/figure2.tex'. Stop.
There are two issues here:
1) Both figure1.p1.tex and figure.p2.tex are generated by the first run of figure1.m by octave. Is there a way to treat all targets with the same basename (or other pattern) as a prerequisite as one, so that there is no more than one invocation of octave per .m file?
2) figure2.tex was made using some other means than an .m file. How can I tell make to ignore a rule if its prerequisite does not exist. I know how to do that for an explicit prerequisite:
target: prereq
recipe
prereq:
But what to do in this case with the prerequisite being derived from the target's name?
GNU make can be taught that multiple targets are created by one command invocation by using a pattern rule for those targets.
From Pattern Rule Examples:
This pattern rule has two targets:
%.tab.c %.tab.h: %.y
bison -d $<
This tells make that the recipe ‘bison -d x.y’ will make both x.tab.c and x.tab.h. If the file foo depends on the files parse.tab.o and scan.o and the file scan.o depends on the file parse.tab.h, when parse.y is changed, the recipe ‘bison -d parse.y’ will be executed only once, and the prerequisites of both parse.tab.o and scan.o will be satisfied. (Presumably the file parse.tab.o will be recompiled from parse.tab.c and the file scan.o from scan.c, while foo is linked from parse.tab.o, scan.o, and its other prerequisites, and it will execute happily ever after.)
So you could use something like
figure1.%1.tex figure1.%2.tex: figure1.m
$(OCTAVE) $<
but to do that for N output files where N is variable would require generating an included makefile that pulled that information out of the .m file (or similar).
If a file exists that matches a target but no matching prerequisite file is found make will just use the file it found and ignore the rule (the rule doesn't apply). You shouldn't have to do anything for that.
If, however, the file would otherwise match the rule (but you don't want make to follow the rule for that file) then you can cancel just that application by giving that file an explicit target. Like this.
figure2.tex: ;
Assume I have a make rule:
.PHONY:gen
gen: auto.template
generate-sources auto.template
that creates a bunch of files, for example auto1.src, auto2.src, auto3.src and so on.
If I now have rules to build targets from *.src files, like this:
$(patsubst %.src,%.target,$(wildcard *.src)): %.target: %.src
build $< > $#
How can I tell make to first execute the gen rule and then expand the preconditions for the second rule template? GNU extensions are welcome.
Note: I would like to keep it in one make invocation; A trivial solution to this would be to put the second rule in a secondary Makefile.secondrun and call $(MAKE) -f Makefile.secondrun after gen was processed. But I was wondering if there is a better option.
Building off Beta's answer, here's how you can do it using makefile remaking in GNU make, which is not the same thing as recursive make. Rather, it updates an included makefile using a rule in the main makefile, then restarts the original make instance. This is how *.d dependency files are typically generated and used.
# Get the list of auto-generated sources. If this file doesn't exist, or if it is older
# than auto.template, it will get built using the rule defined below, according to the
# standard behavior of GNU make. If autosrcs.mk is rebuilt, GNU make will automatically
# restart itself after autosrcs.mk is updated.
include autosrcs.mk
# Once we have the list of auto-generated sources, getting the list of targets to build
# from them is a simple pattern substitution.
TARGETS=$(patsubst %.src,%.target,$(AUTO_SRCS))
all: $(TARGETS)
# Rule describing how to build autosrcs.mk. This generates the sources, then computes
# the list of autogenerated sources and writes that to autosrcs.mk in the form of a
# make variable. Note that we use *shell* constructs to get the list of sources, not
# make constructs like $(wildcard), which could be expanded at the wrong time relative
# to when the source files are actually created.
autosrcs.mk: auto.template
./generate-sources auto.template
echo "AUTO_SRCS=`echo *.src`" > autosrcs.mk
# How to build *.target files from *.src files.
%.target: %.src
#echo 'build $< > $#'
Short answer: you can't. Make determines all of the rules it will have to execute before it executes any rule.
Longer answer: maybe you can. As you say, you can use recursive Make explicitly, or surreptitiously by, say, building a file which your makefile will include (I'm looking at you, Jack Kelly). Or if you could somehow obtain a list of the files which gen will build, you could write a rule around that. Or you could take a leap of faith like this:
%.target: %.src
build $< > $#
%.src: gen;
What does .PHONY mean in a Makefile? I have gone through this, but it is too complicated.
Can somebody explain it to me in simple terms?
By default, Makefile targets are "file targets" - they are used to build files from other files. Make assumes its target is a file, and this makes writing Makefiles relatively easy:
foo: bar
create_one_from_the_other foo bar
However, sometimes you want your Makefile to run commands that do not represent physical files in the file system. Good examples for this are the common targets "clean" and "all". Chances are this isn't the case, but you may potentially have a file named clean in your main directory. In such a case Make will be confused because by default the clean target would be associated with this file and Make will only run it when the file doesn't appear to be up-to-date with regards to its dependencies.
These special targets are called phony and you can explicitly tell Make they're not associated with files, e.g.:
.PHONY: clean
clean:
rm -rf *.o
Now make clean will run as expected even if you do have a file named clean.
In terms of Make, a phony target is simply a target that is always out-of-date, so whenever you ask make <phony_target>, it will run, independent from the state of the file system. Some common make targets that are often phony are: all, install, clean, distclean, TAGS, info, check.
Let's assume you have install target, which is a very common in makefiles. If you do not use .PHONY, and a file named install exists in the same directory as the Makefile, then make install will do nothing. This is because Make interprets the rule to mean "execute such-and-such recipe to create the file named install". Since the file is already there, and its dependencies didn't change, nothing will be done.
However if you make the install target PHONY, it will tell the make tool that the target is fictional, and that make should not expect it to create the actual file. Hence it will not check whether the install file exists, meaning: a) its behavior will not be altered if the file does exist and b) extra stat() will not be called.
Generally all targets in your Makefile which do not produce an output file with the same name as the target name should be PHONY. This typically includes all, install, clean, distclean, and so on.
NOTE: The make tool reads the makefile and checks the modification time-stamps of the files at both the side of ':' symbol in a rule.
Example
In a directory 'test' following files are present:
prerit#vvdn105:~/test$ ls
hello hello.c makefile
In makefile a rule is defined as follows:
hello:hello.c
cc hello.c -o hello
Now assume that file 'hello' is a text file containing some data, which was created after 'hello.c' file. So the modification (or creation) time-stamp of 'hello' will be newer than that of the 'hello.c'. So when we will invoke 'make hello' from command line, it will print as:
make: `hello' is up to date.
Now access the 'hello.c' file and put some white spaces in it, which doesn't affect the code syntax or logic then save and quit. Now the modification time-stamp of hello.c is newer than that of the 'hello'. Now if you invoke 'make hello', it will execute the commands as:
cc hello.c -o hello
And the file 'hello' (text file) will be overwritten with a new binary file 'hello' (result of above compilation command).
If we use .PHONY in makefile as follow:
.PHONY:hello
hello:hello.c
cc hello.c -o hello
and then invoke 'make hello', it will ignore any file present in the pwd 'test' and execute the command every time.
Now suppose, that 'hello' target has no dependencies declared:
hello:
cc hello.c -o hello
and 'hello' file is already present in the pwd 'test', then 'make hello' will always show as:
make: `hello' is up to date.
.PHONY: install
means the word "install" doesn't represent a file name in this
Makefile;
means the Makefile has nothing to do with a file called "install"
in the same directory.
It is a build target that is not a filename.
The special target .PHONY: allows to declare phony targets, so that make will not check them as actual file names: it will work all the time even if such files still exist.
You can put several .PHONY: in your Makefile :
.PHONY: all
all : prog1 prog2
...
.PHONY: clean distclean
clean :
...
distclean :
...
There is another way to declare phony targets : simply put :: without prerequisites :
all :: prog1 prog2
...
clean ::
...
distclean ::
...
The :: has other special meanings, see here, but without prerequisites it always execute the recipes, even if the target already exists, thus acting as a phony target.
The best explanation is the GNU make manual itself: 4.6 Phony Targets section.
.PHONY is one of make's Special Built-in Target Names. There are other targets that you may be interested in, so it's worth skimming through these references.
When it is time to consider a .PHONY target, make will run its recipe
unconditionally, regardless of whether a file with that name exists or
what its last-modification time is.
You may also be interested in make's Standard Targets such as all and clean.
There's also one important tricky treat of ".PHONY" - when a physical target depends on phony target that depends on another physical target:
TARGET1 -> PHONY_FORWARDER1 -> PHONY_FORWARDER2 -> TARGET2
You'd simply expect that if you updated TARGET2, then TARGET1 should be considered stale against TARGET1, so TARGET1 should be rebuild. And it really works this way.
The tricky part is when TARGET2 isn't stale against TARGET1 - in which case you should expect that TARGET1 shouldn't be rebuild.
This surprisingly doesn't work because: the phony target was run anyway (as phony targets normally do), which means that the phony target was considered updated. And because of that TARGET1 is considered stale against the phony target.
Consider:
all: fileall
fileall: file2 filefwd
echo file2 file1 >fileall
file2: file2.src
echo file2.src >file2
file1: file1.src
echo file1.src >file1
echo file1.src >>file1
.PHONY: filefwd
.PHONY: filefwd2
filefwd: filefwd2
filefwd2: file1
#echo "Produced target file1"
prepare:
echo "Some text 1" >> file1.src
echo "Some text 2" >> file2.src
You can play around with this:
first do 'make prepare' to prepare the "source files"
play around with that by touching particular files to see them updated
You can see that fileall depends on file1 indirectly through a phony target - but it always gets rebuilt due to this dependency. If you change the dependency in fileall from filefwd to file, now fileall does not get rebuilt every time, but only when any of dependent targets is stale against it as a file.
I often use them to tell the default target not to fire.
superclean: clean andsomethingelse
blah: superclean
clean:
#echo clean
%:
#echo catcher $#
.PHONY: superclean
Without PHONY, make superclean would fire clean, andsomethingelse, and catcher superclean; but with PHONY, make superclean won't fire the catcher superclean.
We don't have to worry about telling make the clean target is PHONY, because it isn't completely phony. Though it never produces the clean file, it has commands to fire so make will think it's a final target.
However, the superclean target really is phony, so make will try to stack it up with anything else that provides deps for the superclean target — this includes other superclean targets and the % target.
Note that we don't say anything at all about andsomethingelse or blah, so they clearly go to the catcher.
The output looks something like this:
$ make clean
clean
$ make superclean
clean
catcher andsomethingelse
$ make blah
clean
catcher andsomethingelse
catcher blah