make .SECONDEXPANSION recipe execute twice - makefile

Using GNU make, I am trying to solve a problem similar to make recipe execute twice — that is, to have a Makefile recipe run twice. In my case, however, the recipe is run under the .SECONDEXPANSION target, and the two different runs will be called with different parameters to generate different versions of the output file from the same input file. That is, with input file foo, this example Makefile should be callable via make foo.pdf or make foo.expanded.pdf to build one .pdf file, or make all to build both .pdf files:
.PHONY: all
all: foo.pdf foo.expanded.pdf
.SECONDEXPANSION:
%.expanded.pdf %.pdf: %
#echo building $(basename $#)
Of the two solutions given in that answer, the first is unsuitable because it always runs the rule twice; I want it run twice when the user asks for it.
The second solution posted there is conceptually what I am looking for and have implemented in the above example Makefile, with only the small problem that it doesn't work: although the all target lists both .pdf files as dependencies, only one is built when make all is run.
Is there a way to tell GNU make to build two different files using the same rule under a .SECONDEXPANSION?
EDIT: Clarified in problem description that the same input file is used to build both versions of the output file, and modified sample Makefile to include this dependency.
EDIT: I would like a solution as scalable as possible; that is, it should work if the input filename contains dots, specifying additional output file foo.reduced.pdf should require only adjusting the targets and recipe as appropriate, etc. This limits performing string surgery that relies on the filenames appearing exactly as given in this narrow example (e.g., changing the rule to %.pdf: $$(firstword $$(subst ., ,$$*)) fails if the input file could be either foo or foo.bar).

You are probably looking for Pattern-specific Variable Values. Let's assume your recipe depends on a make variable named BUILDFLAGS that takes value normal by default and special for the "expanded" targets. Then this Makefile:
BUILDER := builder
BUILDFLAGS := normal
.PHONY: all
all: foo.pdf foo.expanded.pdf
%.expanded.pdf: BUILDFLAGS := special
%.pdf:
$(BUILDER) $(BUILDFLAGS) $#
should do about what you want with the same rule for all targets, plus one pattern-specific variable value declaration. Replace builder, normal and special with what makes sense in your case. Demo:
$ make foo.pdf
builder normal foo.pdf
$ make foo.expanded.pdf
builder special foo.expanded.pdf
$ make
builder normal foo.pdf
builder special foo.expanded.pdf

Your problem has nothing to do with .SECONDEXPANSION. You can just drop that and the problem will be the same.
Your problem is that you are using a pattern rule with multiple target patterns, and expecting that it works similar to an explicit rule with multiple targets. But it does not (and in fact you cannot have a rule with both pattern and explicit targets).
For a pattern rule with multiple target patterns, Make matches the same pattern to all the %, including multiple times in the targets, and then assumes that it just has to execute the recipe with that pattern once, and it will make all the matched targets.
In your case the best way is to use multiple rules (I changed your recipe because using echo as a Make recipe is a bad idea):
.PHONY: all
all: foo.expanded.pdf foo.pdf
RECIPE = touch $#
%.expanded.pdf:
$(RECIPE)
%.pdf:
$(RECIPE)

Related

Matching a dependency using `subst` function in Makefile

I have a rule for the target
data/processed/21.12.2021/experiment6/written_piv21122021.005.exp6.mp4
in my makefile such that it has a dependency
data/raw/21.12.2021/experiment6/piv21122021.005.exp6.mov
Using subst functions, I'm trying to create the dependency by pattern matching as
%/written_*.mp4: \
$(subst processed,raw, $$*)/*.mov \
<do something>
However, the above rule can't find the *.mov dependency. i have tried many versions of $(subst processed,raw, $$*)/*.mov to match the dependency but didn't work.
How to do this? What is the correct syntax?
First, you can't do this:
%/written_*.mp4:
You can't combine a pattern % with a wildcard *. You have to realize that make works in two very discrete steps: first, all the makefiles are parsed and an internal representation of all the targets and prerequisites are constructed into a graph. Then, make walks that graph figuring out what needs to be built and how to build it, and running recipes.
Make variables, functions, and wildcards in targets and prerequisites are expanded when makefiles are parsed (in the first step). Automatic variables like $* are not set until a recipe is invoked (the second step), and patterns like % are not matched/expanded until make tries to decide how to build something (again in the second step).
So, a rule like:
%/written_*.mp4: $(subst processed,raw, $$*)/*.mov
can't work because the wildcard will expand to files matching the literal filename %/written_*.mp4 which clearly has no matches since you won't have a directory named %. In any event you can't use wildcards in targets because when make is parsing the makefile those targets won't exist (since that's what you want make to build) so the wildcards won't match anything. Also, $$* is the literal string $* and there is no processed string in that so the subst function will do nothing. And, even if /*.mov did match something it would put ALL the files matching that wildcard as prerequisites of every target so they'd all get rebuilt whenever any one changed.
And finally, you definitely should not use a backslash after your prerequisites: this just turns your recipe into prerequisites.
Your problem is very difficult to solve because your target and prerequisite differ in multiple distinct places and make doesn't support multiple % matching. You can get most of the way there with this:
data/processed/%.mp4: data/raw/%.mov
<do something>
However this is not quite right because the % in the target is written_... while in the prerequisite it's just ... and this is not possible to represent in make.
If you can rework the filenames so that instead of written_piv21122021.005.exp6.mp4 you can use piv21122021.005.exp6.written.mp4 (or even better you don't need the written_ prefix at all) then you can easily do this. If not you'll need to get very fancy to make this work.
Here is an imperfect kludge.
Delegate the work to another makefile I'll call adjunct.mk. In the main makefile:
data/processed/%.mp4:
#$(MAKE) -f adjunct.mk $#
And in adjunct.mk this ugly transformation:
.SECONDEXPANSION:
$(MAKECMDGOALS): $$(patsubst data/processed/%,data/raw/%,$$(subst written_,,$$(patsubst %.mp4,%.mov,$$#)))
...do whatever with $< and $#...
This incurs the usual cost of recursive Make: it blinds the higher-level Make to the dependency relations being tracked by the lower one. So if your Make must build that mov file, or you try to build the mp4 when the mov does not exist, then this solution will require some more careful pipefitting before it will work correctly.

How to do Makefile Pattern Rules with List of Files

Wondering how to apply something such as pattern rules to accomplish the following in the Makefile:
FILES := a b c d
$(FILES).c:
run build $(FILE).c
# use $(FILE) without .c extension, etc.
Without having to name all of the values in FILES like a.c, b.c, etc. because I am going to use the names elsewhere.
If I try the above with make a.c, I get:
No rule to make target `a.c'. Stop.
The addsuffix built-in function, which already operates on lists, could be useful here:
targets := $(addsuffix .c,$(FILES))
However, you could simply use pattern rules instead. The automatic variable $* would correspond to the stem (i.e., the % part):
%.c:
run build $#
# use $* for filename without .c extension, etc.

Writing pattern rules for complex build commands with GNU Make

I'm working with a makefile system using mcs as its build command. Briefly, mcs is invoked with:
source files: Passed directly;
resources: with -resource:;
a target: with -target: and can be exe, winexe, library or module.
And produces an *.exe, *.dll or *.netmodule. The -target: parameter is certain in the last two cases but an *.exe can be built with either exe or winexe.
Currently, each directory's makefile includes what boils down to this:
RESOLVED_SOURCES := $(addprefix $(srcdir)/,$(SOURCES))
RESOLVED_RESOURCES := $(addprefix $(srcdir)/,$(RESOURCES))
S_PARAM := $(RESOLVED_SOURCES)
R_PARAM := $(addprefix -resource:,$(RESOLVED_RESOURCES))
all: $(OUTPUT)
$(OUTPUT): $(RESOLVED_SOURCES) $(RESOLVED_RESOURCES)
$(MCS) $(MCSARGS) $(S_PARAM) $(R_PARAM) -target:$(TARGET) -out:$(OUTPUT)
And in each project directory the variables OUTPUT, TARGET, SOURCES and RESOURCES are defined before inclusion.
Even with everything I've left out it's fairly complicated and suffers from only being able to build a single concrete target where it's included, without duplicating the build rule.
To avoid the duplication and increase flexibility I'd like to define some pattern rules for this complex build command but it's difficult:
each compilation unit generally has many inputs;
there's no way to incrementally add each input;
some inputs need different handling.
e.g. For mcs these patterns seem possible:
%.exe: %.cs
$(MCS) $(MCSARGS) -target:$(target) -out:$# $<
%.dll: %.cs
$(MCS) $(MCSARGS) -target:library -out:$# $<
%.netmodule: %.cs
$(MCS) $(MCSARGS) -target:module -out:$# $<
As pattern rules go they aren't too bad but they aren't useful for most applications of mcs as there's only a single source file being passed.
If the user adds source file prerequisites for a target then the above $< could be swapped for $^ to pass them all but that won't work for resource files. The order only list could be used for passing resource file prerequisites but would prevent rebuilds when only resource files were updated.
Is there a general technique for turning this sort of complex command's invocation into a pattern rule or set of rules?
There are a number of problems:
the build command distinguishes more than one prerequisite type;
they must all be passed at once;
updates to any of them must cause a rebuild, they are all regular prerequisites.
Ideas:
delegating the inputs you want to separate to input files and then reading them back in allows you to have a simple target invoking the build;
using secondary expansion you can define those input files based on the contents of "pattern variables";
general search should work.
For example:
multi_A := foo
multi_B := bar
all: multi.foo
%.foo: .a.% .b.%
#echo $*: $(foreach _,$^,$(file < $_))
#touch $#
.SECONDEXPANSION:
.a.%: $$($$*_A)
$(file > $#,$^)
.b.%: $$($$*_B)
$(file > $#,$(addprefix -b:,$^))
To start with using secondary expansion seems dubious. Apart from that the immediate issue with this is that the user has to know that any regular prerequisites will be read and passed to the build command.
On the other hand, knowing that, the user can supplement a target with extra input parameters by depending on a target that writes them to another input file.

makefile: from 3 input generate one output

I have this version of makefile
[sbsuser#compute-00-01 415]$ make --version GNU Make 3.81
I have directory SOMATIC where I have 3 file . I want to produce a only one output. This is what I wrote.
`
OUTSOMATIC=SOMATIC
FINAL=FINAL
INPUT=$(wildcard $(OUTSOMATIC)/*.vcf)
OUTSORT2= $(patsubst $(OUTSOMATIC)/%.vcf,$(FINAL)/%somatic.ensemble.gz,$(INPUT))
$(info lista $(OUTSORT2))
$(info lista $(INPUT))
.PHONY: all
all: $(INPUT) $(OUTSOMATIC) $(OUTSORT2) $(FINAL)
$(FINAL)/%somatic.ensemble.gz: $(OUTSOMATIC)/%.vcf $(INPUT)
~/jdk1.8.0_121/bin/java -XX:+UseSerialGC -Xms1g -Xmx10g -jar /illumina/software/PROG2/bcbio-variation-recall-0.1.7 ensemble -n 1 $(FINAL)/somatic_ensemble.gz /illumina/software/database/database_2016/hg19_primary.fa $^
`
With this script make 3 time the same files. I don't understand how to create only one output from list of input to use in the same time.
What is the best way to do this?
If I change $(FINAL)/%somatic.ensemble.gz: in $(FINAL)/somatic.ensemble.gz I have this error:
make: *** No rule to make target FINAL/415_merge_mutect2.somaticsomatic.ensemble.gz', needed byall'. Stop`
You probably should review the GNU make manual introductory sections where they describe how make works.
Let's look at your makefile; first you define some variables. Let's assume that you have the files SOMATIC/foo.vcf, SOMATIC/bar.vcf, and SOMATIC/baz.vcf. Then the variables you created will have these values, after they are expanded:
OUTSOMATIC = SOMATIC
FINAL = FINAL
INPUT = SOMATIC/foo.vcf SOMATIC/bar.vcf SOMATIC/baz.vcf
Now your patsubst finds all words in INPUT that match the pattern SOMATIC/%.vcf and replace that with FINAL/%somatic.ensemble.gz, where the part that matches the % in the input is substituted into the output:
OUTSORT2 = FINAL/foosomatic.ensemble.gz FINAL/barsomatic.ensemble.gz FINAL/bazsomatic.ensemble.gz
Now, make sees that you've defined an all target. Since it's the first target in the makefile this is the target that will be run by default. After expansion, it will look like this:
all: SOMATIC/foo.vcf SOMATIC/bar.vcf SOMATIC/baz.vcf SOMATIC FINAL/foosomatic.ensemble.gz FINAL/barsomatic.ensemble.gz FINAL/bazsomatic.ensemble.gz FINAL
So, make will try to build every prerequisite of the all target to be sure it's up to date. First it tries to build the SOMATIC/*.vcf files. Those files already exist and make doesn't have any rules about how to rebuild them, so it assumes they're up to date.
Next it tries to build the SOMATIC file. This is a directory and it also has no rule to be built, so make assumes that's up to date as well.
Next make tries to build the target FINAL/foosomatic.ensemble.gz. Make does have a rule that can build it, you've created one:
$(FINAL)/%somatic.ensemble.gz: $(OUTSOMATIC)/%.vcf $(INPUT)
~/jdk1.8.0_121/bin/java ...
This matches the target you want to build, with a % value of foo, so then make substitutes the % in the prerequisite for foo and finds that SOMATIC/foo.vcf exists and doesn't need to be rebuilt, so it runs your recipe. However your recipe doesn't actually create the target FINAL/foosomatic.ensemble.gz; it creates the target FINAL/somatic_ensemble.gz. So this rule is broken because it tells make it will do one thing, but it does something else.
You should always ensure all your recipes build the file represented by the automatic variable $#; that will ensure that you and make agree on the meaning of your rule. If you want your recipe to build some other file, then your rule is written incorrectly.
Next make does the same thing with the next prerequisite of all: FINAL/barsomatic.ensemble.gz. Since that file doesn't exist, make tries to build it using the pattern rule, but again that creates the same output file.
And again for the third .gz file FINAL/bazsomatic.ensemble.gz. That's why things are run three times.
If you change the pattern rule to an explicit rule building FINAL/somatic.ensemble.gz, which is what you want, then make can't find any way to build the prerequisites of the all target so it gives this error.
Your problem is the creation of OUTSORT2. You want to create only one output file, but you've set OUTSORT2 to contain three different files, so make tries to create all three files. You want this:
OUTSOMATIC = SOMATIC
FINAL = FINAL
INPUT = $(wildcard $(OUTSOMATIC)/*.vcf)
OUTSORT2 = $(FINAL)/somatic.ensemble.gz
.PHONY: all
all: $(OUTSORT2)
$(OUTSORT2): $(INPUT)
~/jdk1.8.0_121/bin/java -XX:+UseSerialGC -Xms1g -Xmx10g -jar /illumina/software/PROG2/bcbio-variation-recall-0.1.7 ensemble -n 1 $# /illumina/software/database/database_2016/hg19_primary.fa $^

GNU make - how to set an implicit pattern as a prerequisite

I have this implicit rule:
%.so: %.so.5
qnx_ln $< $#
I realized that for another target, I have to make all .so files the prerequisite for that target.
I tried this:
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs %.so
...
But I got the output
*** No rule to make target '%.so', needed by 'makegen'. Stop.
% prerequisite patterns can only be used in static and implicit pattern rules, where they match the respective % part of the target; when used in a regular rule % is a literal character.
You'll need to specify the dependencies literally, unless there is some correspondence between certain source filenames and the .so filenames that you can leverage, presumably you're already doing either of these to link the .so files in the first place.
As pointed out previously, no you can't do that because this is not how prerequisite patterns work. Maybe you gave the following a thought and rejected it but I suspect you might find the following a close-enough fit:
%.so.target: %.so.5
echo $< >> $(BUILD)/so.targets
SO_TARGETS=$(basename $(shell cat $(BUILD)/so.targets))
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs $(SO_TARGETS)
Maybe you are looking for a rule to match on every existing *.so file?
makegen: $(TEAM_ROOT)HMI_FORGF/src/src.pro module_dirs $(wildcard *.so)
...
However, if there are patterns which could generate *.so files which have not yet generated those files, they will (obviously) not be matched by the wildcard, which simply examines existing files. If that's what you actually want to accomplish, you'll probably want to enumerate the actual files, one way or another.

Resources