I am using makefiles to manage dependencies in computational experiments. It would very often be useful to have targets with multiple patterns.
For example, I may have some conditions, say A and B, and different folds (i.e. splits of data) of a cross-validation experiment, say 1 through 10.
Currently I just copy and paste the same commands for A and B, and pattern-match on the folds, but if there are more than two conditions it quickly becomes a maintenance problem.
output-%.A: input-%
run A input-$* output-$*.A
output-%.B: input-%
run B input-$* output-$*.B
Is there a better way of doing this in makefile?
If not, what other tool would solve this issue?
One drawback of Make is that it doesn't handle wildcards well. But there is a way to minimize the redundancy in this makefile.
First, let's use automatic variables to clean up the rules a little bit:
output-%.A: input-%
run A $< $#
output-%.B: input-%
run B $< $#
Then we define a rule template, use call to evaluate it for A and for B, and eval to interpret those things as actual rules:
define fold-rule
output-%.$(1): input-%
#echo run $(1) $$< $$# # <-- note the doubled '$'
endef
$(eval $(call fold-rule,A))
$(eval $(call fold-rule,B))
Finally we put the conditions in a variable, and iterate using foreach:
CONDITIONS := A B
define fold-rule
output-%.$(1): input-%
#echo run $(1) $$< $$#
endef
$(foreach x, $(CONDITIONS), $(eval $(call fold-rule,$(x))))
Now when you want to add a new condition, just modify the first line.
It isn't pretty, but it works.
Using a few features of makepp, like using variable as macro-functions, and early evalution with $[] it is easy:
define runner
output-%.$1: input-%
run $1 $$(input) $$(output)
enddef
$[runner A]
$[runner B]
$[runner C]
If you want to create a lot of rules, this may still be too verbose. Problem with the foreach function is that it joins the results with a space. So to separate the rules, you can put a dummy empty expression the last line:
define runner
output-%.$1: input-%
run $1 $$(input) $$(output)
$()
enddef
$[foreach type,D E F,$[runner $[type]]]
Related
I have a complicated set of rules I need to write to generate a rather large number of "parameterised" output files and thought that, rather than expand them all out by hand, I could repeatedly "include" a template file with sets of rules and use (GNU)make's facility for allowing "simply expanded" variables to avoid the pain.
(In the past I've always been using the "recursively expanded" variable approach, so this is new to me)
As a trivial example of what I thought would work, I tried putting the following in a Makefile
Targ:=A
Param1:=Pa
Param2:=Qa
$(Targ):
#echo expect A, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
Targ:=B
Param1:=Pb
Param2:=Qb
$(Targ):
#echo expect B, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
Targ:=C
Param1:=Pc
Param2:=Qc
$(Targ):
#echo expect C, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
The eventual plan was to replace the rules with an include file containing dozens of different rules, each referencing the various "parameter" variables.
However, what I get is...
prompt> make A
expect A, get C, Target is A. Params are Pc and Qc
prompt> make B
expect B, get C, Target is B. Params are Pc and Qc
Essentially, unlike each rule's target, which is picking up the intended definition, the $(Targ), $(Param1), and $(Param2) in each rule's command is instead being run with the final definition.
Does anyone know how to prevent this, i.e. how do you force the command to use the definition at the time it is encountered in the Makefile?
Simple vs recursive expansion makes no difference here; regardless of which you use you'll see the same behavior. A GNU make variable is global and obviously can have only one value.
You have to understand when variables are expanded. The documentation provides a detailed description of this. Targets and prerequisites are expanded when the makefile is read in, so the value of Targ as the makefile is being parsed is used.
Recipes are expanded when the recipe is to be invoked, which is not until after all makefiles are parsed and make starts to build targets. At that time of course the variable Targ has its last set value.
Without knowing what your makefile really does it's hard to suggest an alternative. One option is to use target-specific variables:
Targ := A
$(Targ): LocalTarg := $(Targ)
$(Targ):
#echo expect A, get $(LocalTarg), Target is $#
Another option is to use constructed variable names:
Targ := A
Targ_$(Targ) := $(Targ)
$(Targ):
#echo expect A, get $(Targ_$#), Target is $#
Apologies for answering my own question, but I now realised it is possible to solve the issue I was having by running make recursively.
E.g. if the parameter variables for the rules are Targ, Param1 and Param2 then
#Set up "default" values for the parameters (As #madscientist points out,
#these will safely be overridden by the defs on the #(make) commands below
Targ=XXXXXXXXXX
Param=XXXXXXXXXX
Param2=XXXXXXXXXX
Recursing=
#
# define N (==3) templated rule(s)
#
$(Targ)%a:
#echo Run Combo_a $(Targ) $(Param1) $(Param2) $#
$(Targ)%b:
#echo Run Combo_b $(Targ) $(Param2) $(Param1) reversed $#
$(Targ)%c:
#echo Run Combo_c $(Param1) $(Targ) $(Param2) mixed again $#
#
#Enumerate "M" (==2) sets of parameters,
# (Except if we are already recursing because unrecognised targets may cause
# it to descend forever)
#
ifneq ($(Recursing), Yes)
Set1%:
#$(MAKE) Targ=Set1 Param1=foo Param2=bar Recursing=Yes $#
Set2%:
#$(MAKE) Targ=Set2 Param1=ray Param2=tracing Recursing=Yes $#
endif
This then allows N*M different combos for N+M typing cost.
eg. (removing messages from make re recursion)
>make Set1.a
Run Combo_a Set1 foo bar Set1.a
>make Set2.c
Run Combo_c ray Set2 tracing mixed again Set2.c
According to the make documentation, pattern rules with multiple outputs run only once while normal rules may run multiple times. I'd like to run pattern rules multiple times to share a recipe. I know I can share them by using a define but that comes with a different set of problems.
I am using gnu make version 3.8.1
This is the makefile I'd like to use. Is there a way to have it run the pattern rule four times?
build: test.a.x test.a.y test.b.x test.b.y
clean:
rm -f test.a.x test.a.y test.b.x test.b.y
%.a.x: optAB=a
%.a.x: optXY=x
%.a.y: optAB=a
%.a.y: optXY=y
%.b.x: optAB=b
%.b.x: optXY=x
%.b.y: optAB=b
%.b.y: optXY=y
%.a.x %.a.y %.b.x %.b.y: %.v
#echo optAB: $(optAB) optXY: $(optXY) $# $<
touch $#
The relevant line in the documentation is
If a pattern rule has multiple targets, make knows that the rule's recipe is responsible for making all of the targets.
So one way to get around this would be to define multiple targets. One way to achieve this is to use foreach/eval/call:
build: test.a.x test.a.y test.b.x test.b.y
clean:
rm -f test.a.x test.a.y test.b.x test.b.y
%.a.x: optAB=a
%.a.x: optXY=x
%.a.y: optAB=a
%.a.y: optXY=y
%.b.x: optAB=b
%.b.x: optXY=x
%.b.y: optAB=b
%.b.y: optXY=y
optset1 = a b
optset2 = x y
define target_template
%.$(1).$(2): %.v
#echo optAB: $$(optAB) optXY: $$(optXY) $$# $$<
touch $$#
endef
$(foreach opt1,$(optset1),$(foreach opt2,$(optset2),$(eval $(call target_template,$(opt1),$(opt2)))))
We define a template for the rule, and then call/eval that template for each of the options, to generate multiple rules, one for each target. Note the $$ in the template definition are to ensure the variables are not expanded when the template is defined, but only when it is evaluated to generate each target.
I am trying to compile for different software directories with different optimization levels etc. I created the following makefile to do so:
OWNER = betsy molly fred
DOG = poodle mutt doberman
COLOUR = brown red yellow
ATTR = big small
LEGS = 0 3
#we want every possible combination to be excercised
OUTPUT_STUFF = $(foreach own,$(OWNER),$(foreach dog,$(DOG),$(foreach col,$(COLOUR),$(foreach attr,$(ATTR),$(foreach legs,$(LEGS),new/$(own)/$(dog)/$(col)/$(attr)/$(legs)/dogInfo.txt)))))
.PHONY: all
all: $(OUTPUT_STUFF)
define PROGRAM_template
own = $(1)
dog = $(2)
col = $(3)
attr = $(4)
legs = $(5)
BUILD_DIR = new/$(own)/$(dog)/$(col)/$(attr)/$(legs)
#for each build directory, we are going to put a file in it containing the build dir. string
$$(BUILD_DIR)/dogInfo.txt:
#echo "$$#"
mkdir $$(BUILD_DIR)
#echo "$$(BUILD_DIR)" > $$(BUILD_DIR)/dogInfo.txt
endef
#call the function many times
$(foreach own,$(OWNER),$(foreach dog,$(DOG),$(foreach col,$(COLOUR),$(foreach attr,$(ATTR),$(foreach legs,$(LEGS),$(eval $(call PROGRAM_template,$(own),$(dog),$(col),$(attr),$(legs))))))))
As you can see, this simple test program loops through different combinations of owner, dog etc. The end goal is to have a directory, new, that has all owners as dirs, and in those, all dogs, etc. At the bottom is just a file with the path in it.
When I run this, the output is:
new/betsy/poodle/brown/big/0/dogInfo.txt
mkdir new/fred/doberman/yellow/small/3
mkdir: cannot create directory `new/fred/doberman/yellow/small/3': No such file or directory
make: *** [new/betsy/poodle/brown/big/0/dogInfo.txt] Error 1
So, for some reason, the target is ok, but the seemingly exact same variable is the last in my loops. Fundamentally, I don't understand what is happening that well.
Weird foreach + user-defined function behavior in Makefiles seems to answer, but I don't fully get it. In my mind, when the function is called, it fills in all instances with one $, and the escaped ones become $(BUILD_DIR). It then 'pastes' the code to the temporary file, and after it's done all the calls it evaluates the file, substituting the variables as normal.
One (ugly) solution I thought of is to make the BUILD_DIR variable different every time like so:
B_D_$(1)_$(2)_$(3)_$(4)_$(5) = ~~~
Alex is correct (although I think he means recipe, not receipt :-)). The best way to debug complex eval issues is to replace the eval function with a call to info instead. So if you have something like:
$(foreach A,$(STUFF),$(eval $(call func,$A)))
then you can rewrite this as:
$(foreach A,$(STUFF),$(info $(call func,$A)))
Now make will print out to you exactly what the eval is going to parse. It's usually pretty clear, looking at the makefile output, what the problem is. In your case you'll see something like this in the output (leaving out all the extra variable settings):
BUILD_DIR = new/betsy/poodle/brown/big/0
$(BUILD_DIR)/dogInfo.txt:
#echo "$$#"
mkdir $(BUILD_DIR)
#echo "$(BUILD_DIR)" > $(BUILD_DIR)/dogInfo.txt
BUILD_DIR = new/betsy/poodle/brown/big/3
$(BUILD_DIR)/dogInfo.txt:
#echo "$$#"
mkdir $(BUILD_DIR)
#echo "$(BUILD_DIR)" > $(BUILD_DIR)/dogInfo.txt
etc. Notice how you're setting the global variable BUILD_DIR every time. In make, variables have only one value (at a time). While make is reading the makefile it expands the target and prerequisite lists immediately, so whatever value BUILD_DIR has at that time will be used for targets/prerequisites, so this works for you.
But when make finishes reading the makefile, the value of BUILD_DIR will always be the last thing you set it to; in this case new/fred/doberman/yellow/small/3. Now make starts to invoke the recipes for each target, and when it does that it will expand BUILD_DIR in the recipes then, and so ALL the recipes will get that same value.
As Alex points out, you should ensure that your recipe uses only automatic variables like $#, which are set correctly for each rule. If you do that you'll notice that you don't really need to redefine the rule at all because it's actually the same recipe for all the targets. And if you notice THAT, you'll notice you don't need the whole eval or call complexity in the first place.
All you have to do is compute the names of all the targets, then write a single rule:
ALLDOGINFO = $(foreach own,$(OWNER),$(foreach dog,$(DOG),$(foreach col,$(COLOUR),$(foreach attr,$(ATTR),$(foreach legs,$(LEGS),new/$(own)/$(dog)/$(col)/$(attr)/$(legs)/dogInfo.txt)))))
$(ALLDOGINFO):
#echo "$#"
mkdir $(dir $#)
#echo "$(dir $#)" > $#
If you don't want the trailing slash you have to use $(patsubst %/,%,$(dir $#)) instead.
The problem is that when $$(BUILD_DIR) is evaluated in receipt, the loop is already complete. The solution is to rewrite the receipt:
$$(BUILD_DIR)/dogInfo.txt:
#echo "$$#"
mkdir $$(#D)
#echo "$$(#D)" > $$#
I don't think your problem is necessarily with something to do with make. This command:
mkdir new/fred/doberman/yellow/small/3
will fail if one of the parent directories (for example, yellow) doesn't already exist. The error it spits out in this case is the one you're getting, so it seems likely this is the case. If you want a command that makes all parent directories of a given directory as needed, you should run mkdir -p, like this:
mkdir -p $$(BUILD_DIR)
See the mkdir man page for a full description of what -p does.
I want to define a few similar rules, so I decided to try to define them inside of a function:
COMPILED_JS_FILES=$(COMPILED_JS_FILES) $(foreach m,$(notdir $(wildcard src/$(1)/*.$(2))),$(TARGET_DIR)/$(1)/$(m))
$(TARGET_DIR)/$(1)/%.$(2) : src/$(1)/%.$(2)
$(CAT) $< > $#
endef
$(eval $(call COMPILE_JS,modules,js))
$(eval $(call COMPILE_JS,modules,jsm))
$(eval $(call COMPILE_JS,models,js))
$(eval $(call COMPILE_JS,firefox,js))
$(eval $(call COMPILE_JS,helpers,js))
However, the $< and $# variables inside the rule evaluate to empty strings, presumably because they are being defined by the function that is running them, as opposed to being saved until the rule is evaluated.
I am curious about the answer to this question, but also into other reasonable solutions to this problem (other than restructuring the directory structure).
I actually figured out the answer (in this case, I don't know about in general) - $$< and $$#. Furthermore $$(CAT) will delay the expansion of $(CAT) until the evaluation of the rule.
This is an old problem with makefiles. There really isn't a good, general way to get around it within Make, because you are right: the $< and $# just are not available when you need them.
When I have a problem like this, I tend to write a support script outside Make. However, GNU Make does provide the .SECONDEXPANSION target, an obscure hack intended to make some tasks like yours possible to complete. You might investigate it, though my own experience is that .SECONDEXPANSION is not always a clean solution.
I have to convert a set of file (let's say format fa) into another format (fb) by a command (fa2fb). Each target fb depends only on one fa file.
Data structure is in a format like this:
source:
./DATA/L1/fa/L1.fa
./DATA/L2/fa/L2.fa
...
./DATA/Ln/fa/Ln.fa
target:
./DATA/L1/fb/L1.fb
./DATA/L2/fb/L2.fb
...
./DATA/Ln/fb/Ln.fb
How can I implement it with make?
I have tried this but of course it did not work:
./DATA/%/fb/%.fb : ./DATA/%/fa/%.fb
#fa2fb $< $#
Is there any simple solution without changing the data directories?
Many thanks!
Use secondary expansion and the subst function to create a rule where the prerequisites are constructed as a more complex function of the target name:
.SECONDEXPANSION:
DATA/%.fb: $$(subst fb,fa,$$#)
#fa2fb $< $#
Note that this approach assumes that fb will not occur anywhere else in the filename (which holds true if all of your filenames are of the form DATA/Ln/fb/Ln.fb, for some integer n).
This may be the sloppiest makefile I have ever written.
define template
$(2) : $(1)
echo hi
endef
sources=DATA/L1/fa/L1.fa DATA/L2/fa/L2.fa
$(foreach source,$(sources),$(eval $(call template,$(source),$(subst /fa/,/fb/,$(subst .fa,.fb,$(source))))))
The idea is to define a macro to generate your rules, then use foreach and eval+call to invoke it once for each source. The source is the first argument to the call, so it becomes $(1) in the macro. The second argument is just the transformation from a source file name to a destination file name; it becomes $(2) in the macro.
Replace echo hi with your own rule and you should be good to go. And be sure to write a nice big clear comment or someday someone will surely show up at your door with a baseball bat.
This is basically the same as Nemo's answer. I just tried to make the foreach call a bit more readable, by creating a list of modules, containing simply L1 L2 ... Ln, instead of the list of full source names.
MODULES := $(notdir $(wildcard ./DATA/L*))
define rule
./DATA/$(1)/fb/$(1).fb: ./DATA/$(1)/fa/$(1).fa
#fa2fb $< $#
endef
$(foreach module, $(MODULES), $(eval $(call rule,$(module))))