GPATH has a special feature.
It allows Make to recursively search for a missing target, by repeatedly applying the same VPATH on a given target.
Thus: all can turn to D/D/D/D/D/all (VPATH applied 5 times!), and so on.
This rather bizarre - and frankly buggy - relocation, that Make performs on its targets, doesn't occur for a "normal" VPATH. Thus, for a "regular" VPATH, Make will still apply the VAPTH once, and only once.
But, as soon as you add GPATH into the mix, Make goes out of its way, by doing a fully-exhaustive recursive-vpath-search, relocating the missing target to a directory - as deep as make could possible match - by repeatedly and recursively applying again and again the same VPATH.
For example, given a makefile, like:
# Force Make, to do a "directory-search" (via VPATH), for the non-existent file 'all'
$(shell rm -rf 'all')
# Make sure, that all of the (vpath) files - 'D/all', 'D/D/all' and so on - DO NOT exist!
$(shell rm -rf 'D')
VPATH = D
all :
#echo '$#'
D/all :
D/D/all :
D/D/D/all :
D/D/D/D/all :
D/D/D/D/D/all :
Executing, I get:
# 'VAPTH' alone. (i.e. no 'GPATH').
$ make
D/all
# Adding 'GPATH = D', causes Make to do an "exhaustive" recursive-directory-search.
# This recursive serach, has no limits, regarding the number of times, Make will apply the same VPATH on a given target.
$ make -f makefile -f <(echo 'GPATH = D')
D/D/D/D/D/all
Bug or feature? What do you think?
(Versions note: 3.82 and up).
Related
I have this version of makefile
[sbsuser#compute-00-01 415]$ make --version GNU Make 3.81
I have directory SOMATIC where I have 3 file . I want to produce a only one output. This is what I wrote.
`
OUTSOMATIC=SOMATIC
FINAL=FINAL
INPUT=$(wildcard $(OUTSOMATIC)/*.vcf)
OUTSORT2= $(patsubst $(OUTSOMATIC)/%.vcf,$(FINAL)/%somatic.ensemble.gz,$(INPUT))
$(info lista $(OUTSORT2))
$(info lista $(INPUT))
.PHONY: all
all: $(INPUT) $(OUTSOMATIC) $(OUTSORT2) $(FINAL)
$(FINAL)/%somatic.ensemble.gz: $(OUTSOMATIC)/%.vcf $(INPUT)
~/jdk1.8.0_121/bin/java -XX:+UseSerialGC -Xms1g -Xmx10g -jar /illumina/software/PROG2/bcbio-variation-recall-0.1.7 ensemble -n 1 $(FINAL)/somatic_ensemble.gz /illumina/software/database/database_2016/hg19_primary.fa $^
`
With this script make 3 time the same files. I don't understand how to create only one output from list of input to use in the same time.
What is the best way to do this?
If I change $(FINAL)/%somatic.ensemble.gz: in $(FINAL)/somatic.ensemble.gz I have this error:
make: *** No rule to make target FINAL/415_merge_mutect2.somaticsomatic.ensemble.gz', needed byall'. Stop`
You probably should review the GNU make manual introductory sections where they describe how make works.
Let's look at your makefile; first you define some variables. Let's assume that you have the files SOMATIC/foo.vcf, SOMATIC/bar.vcf, and SOMATIC/baz.vcf. Then the variables you created will have these values, after they are expanded:
OUTSOMATIC = SOMATIC
FINAL = FINAL
INPUT = SOMATIC/foo.vcf SOMATIC/bar.vcf SOMATIC/baz.vcf
Now your patsubst finds all words in INPUT that match the pattern SOMATIC/%.vcf and replace that with FINAL/%somatic.ensemble.gz, where the part that matches the % in the input is substituted into the output:
OUTSORT2 = FINAL/foosomatic.ensemble.gz FINAL/barsomatic.ensemble.gz FINAL/bazsomatic.ensemble.gz
Now, make sees that you've defined an all target. Since it's the first target in the makefile this is the target that will be run by default. After expansion, it will look like this:
all: SOMATIC/foo.vcf SOMATIC/bar.vcf SOMATIC/baz.vcf SOMATIC FINAL/foosomatic.ensemble.gz FINAL/barsomatic.ensemble.gz FINAL/bazsomatic.ensemble.gz FINAL
So, make will try to build every prerequisite of the all target to be sure it's up to date. First it tries to build the SOMATIC/*.vcf files. Those files already exist and make doesn't have any rules about how to rebuild them, so it assumes they're up to date.
Next it tries to build the SOMATIC file. This is a directory and it also has no rule to be built, so make assumes that's up to date as well.
Next make tries to build the target FINAL/foosomatic.ensemble.gz. Make does have a rule that can build it, you've created one:
$(FINAL)/%somatic.ensemble.gz: $(OUTSOMATIC)/%.vcf $(INPUT)
~/jdk1.8.0_121/bin/java ...
This matches the target you want to build, with a % value of foo, so then make substitutes the % in the prerequisite for foo and finds that SOMATIC/foo.vcf exists and doesn't need to be rebuilt, so it runs your recipe. However your recipe doesn't actually create the target FINAL/foosomatic.ensemble.gz; it creates the target FINAL/somatic_ensemble.gz. So this rule is broken because it tells make it will do one thing, but it does something else.
You should always ensure all your recipes build the file represented by the automatic variable $#; that will ensure that you and make agree on the meaning of your rule. If you want your recipe to build some other file, then your rule is written incorrectly.
Next make does the same thing with the next prerequisite of all: FINAL/barsomatic.ensemble.gz. Since that file doesn't exist, make tries to build it using the pattern rule, but again that creates the same output file.
And again for the third .gz file FINAL/bazsomatic.ensemble.gz. That's why things are run three times.
If you change the pattern rule to an explicit rule building FINAL/somatic.ensemble.gz, which is what you want, then make can't find any way to build the prerequisites of the all target so it gives this error.
Your problem is the creation of OUTSORT2. You want to create only one output file, but you've set OUTSORT2 to contain three different files, so make tries to create all three files. You want this:
OUTSOMATIC = SOMATIC
FINAL = FINAL
INPUT = $(wildcard $(OUTSOMATIC)/*.vcf)
OUTSORT2 = $(FINAL)/somatic.ensemble.gz
.PHONY: all
all: $(OUTSORT2)
$(OUTSORT2): $(INPUT)
~/jdk1.8.0_121/bin/java -XX:+UseSerialGC -Xms1g -Xmx10g -jar /illumina/software/PROG2/bcbio-variation-recall-0.1.7 ensemble -n 1 $# /illumina/software/database/database_2016/hg19_primary.fa $^
I have a very bizzare problem with GNU make. I have the following files:
a/x.html
b/Makefile
b/c/Makefile
The contents of a/x.html are irrelevant. The contents of b/Makefile are as follows:
SRC=../a
all: x.html
%.html: ${SRC}/%.html
rsync $< $#
The contents of b/c/Makefile are the same, except for the definition of SRC:
SRC=../../a
If I run make in b/c/ the result is as expected:
rsync ../../a/x.html x.html
and x.html gets copied from a/ to b/c/.
However, if I run make in b/ the output I get is several lines of:
make: stat: ../a/../a/.. (repeated many times) ../a/x.html: File name too long
It seems that make is applying the rule for %.html recursively, but why? Is there something obvious I am missing?
To build a target that matches the pattern %.html (i.e. any target name that ends in .html), make applies the rule if it can build the dependency (target built from the original target with ../a/ prepended).
You ask to build x.html. This matches the pattern %.html, so the rule applies: make sees if it can build ../a/x.html.
../a/x.html matches the pattern %.html, so the rule applies: make sees if it can build ../a/../a/x.html.
../../a/x.html matches the pattern %.html, so the rule applies, etc.
The stem character can match any part of a path, including directory separators.
You can see what make is trying by running make -r -d (-d to show debugging output, -r to turn off built-in rules which would cause a huge amount of noise).
When you're in b/c, this stops at step 2 because ../../a/x.html exists but ../../../../a/x.html doesn't.
One way to fix this is to list the files on which you want to act. You can build that list from the list of files that already exist in ../a:
$(notdir $(wildcard ${SRC}/*.html)): %.html: ${SRC}/%.html
rsync $< $#
This has the downside that if the HTML files in ../a are themselves built by a rule in b/Makefile, then running make in b won't built them in a pristine source directory. This shouldn't be a problem though: it would be unusual to have a makefile in b build things outside b.
Another approach which doesn't have this defect is to use an absolute path.
%.html: $(abspath ${SRC})/%.html
rsync $< $#
From the docs:
$?
The names of all the prerequisites that are newer than the target,
with spaces between them.
So, given a makefile:
# Force make to search for 'foo' in the VPATH directory
$(shell rm -rf foo)
# If 'D' is a "regular" file, we remove it first.
$(shell rm -rf D)
$(shell mkdir D)
# Suggest a VPATH-file, for Make to "associate" with 'foo'.
$(shell touch D/foo)
$(shell sleep 1)
# Target 'all' is newer than prerequisite 'D/foo'
$(shell touch all)
VPATH = D
all : foo phony
echo '$?'
foo ::
touch '$#'
.PHONY: phony
.PRECIOUS : D/foo
And running, I get:
$ make -r
touch 'D/foo'
echo 'D/foo phony'
D/foo phony
# Try again, but this time, with parallel-execution mode.
$ make -r -j
touch 'D/foo'
echo 'phony'
phony
Here, we have 2 serious issues:
Given the simple and explicit recipe to "touch" the prerequisite foo, which Make clearly executes - hence will guarantee that foo will be "newer" than all - Make still does not expand $? to D/foo, at-least in the 2nd case above (i.e. for the parallel-execution (-j) mode). Why?
If you come up with an explanation for the above, shouldn't it also explain, why in the 1st case (non-parallel execution), $? - does indeed - get expanded to D/foo.
I guess, I had an assumption, that parallel vs. non-parallel aside, Make will always pause before executing a target, and first check if all of its prerequisites had already finished their respective builds.
So, shouldn't the $? variable be identically expanded for both cases?
I think there are two issues going on here.
The first is that double-colon rules appear to act like phony targets in that they force make to consider the target as "newer" regardless of actual modification time. (This is why the non-parallel version behaves the way it does. Change from foo :: to foo : and you don't get foo in the $? output at all.)
The second thing is that, despite that, using parallel mode seems to force make back into considering modification times of its prerequisites (so the previous behavior is avoided).
This is conjecture and not definitive since I haven't dug through the code to see if this is actually happening but it explains the results here (it also explains the results on the other, nearly identical, question here).
Before I start, I'll mention that I'm not using GNU Make in this case for building a C/C++ project.
Makefile:
DEST_DIR = build/
SRC_DIR = src/
$(SRC_DIR)a/ : $(SOMETHING_ELSE)
$(DO_SOMETHING_TO_GENERATE_A_DIR)
$(DEST_DIR)% : $(SRC_DIR)%
cp -r $^ $#
ALL_DEPS += <SOMETHING>
... more code which appends to ALL_DEPS ...
.PHONY: all
all : $(ALL_DEPS)
I've got some files not generated via Make rules in $(SRC_DIR). (For the sake of this example, let's say there's a directory $(SRC_DIR)b/ and a file $(SRC_DIR)c .)
I want to append to ALL_DEPS all targets which represent files or directories in $(DEST_DIR) so that "make all" will run all of the available $(DEST_DIR)% rules.
I thought to do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(wildcard $(SRC_DIR)*)))
But of course, that doesn't catch anything that hasn't yet been made. (i.e. it doesn't append $(DEST_DIR)a/ to the list because $(SRC_DIR)a/ doesn't yet exist when the $(wildcard ...) invocation is evaluated and the shell doesn't include it in the results returned by the $(wildcard ...) invocation.)
So, rather than a function which finds all (currently-existing) files matching a pattern, I need one which finds all targets matching a pattern. Then, I could do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(targetwildcard $(SRC_DIR)*)))
If it matters any, I've got much of the GNU Make code split across multiple files and included by a "master" Makefile. The ALL_DEPS variable is appended to in any of these files which has something to add to it. This is in an attempt to keep the build process modular as opposed to dropping it all in one monster Makefile.
I'm definitely still learning GNU Make, so it's not unlikely that I'm missing something fairly obvious. If I'm just going about this all wrong, please let me know.
Thanks!
It is simply not possible to do what you're trying to do; you're trying to get make to recognise something that doesn't exist.
This is part of the reason why, in general, wildcards are bad (the other being that you can end up including stuff you didn't mean to). The right thing to do here is to explicitly create a list of source files (ls -1 | sed -e 's/\(.*\)/sources+=\1/' > dir.mk) and perform the patsubst transformation on that list.
If you have additional files that are generate as part of the build, then you can append them to that list and their rules will be found as you'd expect.
I setup make to build my tree using recursive make. So the setup is
A/Makefile a.c
A/B/Makefile a.c
A/B/C/Makefile a.c
where if I issue the command make all from level A/ then make will travel down the tree building everything and then come back up. Each Makefile contains a list of folders below it to build. There is a common.inc file in the root which is read in each Makefile.
This is just a standard layout for recursive make, and nothing new. The details is gives in many places. here and here are examples.
My question is this: many times I'd like to do make all but only build things in the current folder, and not actually travel down the tree, may be because I want to test some changes in the current folder at this time. So I end up editing the current folder's Makefile by commenting out the SUBDIRS=A B C which lists all folders below, or by adding new special targets for this folder only. Both are annoying things to have to keep doing.
Does any one have an idea or a small example of a recursive makefile that uses a switch to tell it if it should travel down the tree or not when called? may be there is a way to call make and pass it some flag at the command line, and this flag is used to remove SUBDIRS=A B C ..... list so it only stops at the current folder level?
Just to be clear. I am using standard SUBDIRS in the Rules.mk, which each Makefile in the tree includes. Here is the part. I copied this from the net long time ago
$(SUBDIRS)::
#if test -d $#; then \
set $(EXIT_ON_ERROR); \
echo "cd $#; make $#"; \
cd $#; make $#; \
set +e; \
else \
echo "Skipping non-directory $#..."; \
fi \
$(CLICK_STOPWATCH);
endif
and in each folder Makefile I write
SUBDIRS = A B C
include Rules.mk
all:: .......
Then I just write make all to build. If there is a way to do make all LOOP=0 where LOOP is some value I pass it or an option or a string or something and then change the above SUBDIRS logic to check for the value of this LOOP and based on the value then do the recursive make or not, then the problem is solved. The default can be to LOOP=1 if it is missing from the command line.
But I do not know enough Make to program this type of logic.
You should use power of rules' depencies. Add your sources files to the dependencies of the rule called from the "root Makefile". If these files are up to date, the recursivity in an folder will stop because the rule is 'up-to-date', and nothing will be done.
Don't add .PHONY for all your rules in the sub-directory Makefile, otherwise recursives rules will be called.
Play with the dependencies of the rules can be the key to not make recursive call, but if you modify sources in each folder and want to build only from the root Makfile, you have to create another rules. With make all, the make binary may not know if you want build all your projet or not (if all your sources has been modified).
EDIT: choice by the command line
You're near the answer, you can set env var while calling your make all and test the value to decide calling recursivly or not.
CC=g++
SUBDIR=a b
all: ${SUBDIR} main.cc
${CC} main.cc
${SUBDIR}:
ifneq ($(MK_LOOP), 0)
#echo "trust the recursivity !"
${MAKE} -C $#
endif
.PHONY: ${SUBDIR}
If you don't set the MK_LOOP var or you set to something else than 0, it will not be equal to 0 so recursive Makefile will be call; if you set to 0, $(SUBDIR) rule do nothing
42SH $ MK_LOOP=0 make # no recur
42SH $ make all
trust the recursivity !
42SH $ make all MK_LOOP=1 # recur by default; same as : make all
trust the recursivity !
42SH $