multi-wildcard pattern rules of GNU Make - makefile

I want to write something like regex:
SRC:="a.dat.1 a.dat.2"
$(SRC): %.dat.%: (\\1).rlt.(\\2)
dat2rlt $^ $#
so that a.dat.1 and a.dat.2 will give a.rlt.1 and a.rlt.2.
In GNU Make info page, it says "the % can be used only once".
Is there some trick to achieve this in GNU Make?

I'm afraid what you are trying to do is not possible the way you suggest to do it, since - as you already mention - (GNU) make only allows a single stem '%', see http://www.gnu.org/software/make/manual/make.html#Pattern-Rules:
A pattern rule looks like an ordinary rule, except that its target
contains the character ‘%’ (exactly one of them).
Without it, creating such 'multi-dimensional' targets is cumbersome.
One way around this is by rebuilding the name of the dependency in the command (rather than in the dependency list):
SRC := a.dat.1 a.dat.2
all : $(SRC:%=%.dat2rlt)
%.dat2rlt :
dat2rtl $(word 1,$(subst ., ,$*)).rlt.$(word 2,$(subst ., ,$*)) $*
Of course, however, this way you would lose the dependency, it will not rebuild once the rlt has been updated.
The only way I can see to address that is by generating the rules explicitly:
SRC := a.dat.1 a.dat.2
all : $(SRC)
define GEN_RULE
$1.dat.$2 : $1.rlt.$2
dat2rtl $$< $$#
endef
$(foreach src,$(SRC),$(eval $(call GEN_RULE,$(word 1,$(subst ., ,$(src))),$(word 3,$(subst ., ,$(src))))))

Using named variables, we can write more readable code (based on answer of Paljas):
letters:=a b c
numbers:=1 2 3 4
define GEN_RULE
$(letter).dat.$(number) : $(letter).rlt.$(number)
./rlt2dat $$< $$#
endef
$(foreach number,$(numbers), \
$(foreach letter,$(letters), \
$(eval $(GEN_RULE)) \
) \
)
We can generate SRC in a similar way. Note that using that method SRC will contain all the combinations. That may or may not be beneficial.

Building on the answer of Erzsébet Frigó, you might additionally choose to:
in the inner loop, eval not the macro itself but the result of calling it
name the macro after program you're calling, dat2rtl
in combination, allowing you to
refer to the program name using make's ${0}
define a target, ${0}s (expanding to dat2rts - note the pluralization) with preconditions of all combinations of letters and numbers on which dat2r2 was called
Like this:
letters:=a b c
numbers:=1 2 3 4
define rlt2dat
${0}s::$(letter).dat.$(number)
$(letter).dat.$(number): $(letter).rlt.$(number)
./${0} $$< $$#
endef
$(foreach number,$(numbers), \
$(foreach letter,$(letters), \
$(eval $(call rlt2dat))))
allowing you to build all rlt2dat targets as:
make rlt2dats

For the limited example you gave, you can use a pattern with one %.
SRC := a.dat.1 a.dat.2
${SRC}: a.dat.%: a.rlt.%
dat2rlt $^ $#
$* in the recipe will expand to whatever the % matched.
Note that the "s around your original macro are definitely wrong.
Have a look at .SECONDEXPANSION in the manual for more complicated stuff (or over here).

Related

Makefile: how to set up prerequisites for two lists of files

I have two lists of files as prerequisites
input_i.xx
config_j.yy
and I need to run all of their combinations. A single one looks like this:
input1_config3.output: input1.xx config3.yy
run_script $^
Also in reality, their names are not numbered, but I already have their stems defined in INPUTS and CONFIGS. With that, I can generate all the targets together
TARGETS:=$(foreach input,$(INPUTS),$(foreach config,$(CONFIGS),$(input)_$(config).output))
But I have difficulty with the prerequisites. It seems I need to
get basename
split on _
add the extensions .xx and .yy
.SECONDEXPANSION
$(TARGETS): $(basename $#)
run_script $^
Can someone show me how to do that? Not sure if this the proper way, maybe a bottom-up way is easier?
make is not really suitable for keeping track of an M x N matrix of results. The fundamental problem is that you can't have two stems in a rule, so you can't say something like
# BROKEN
input%{X}_config%{Y}.output: input%{X}.xx config%{Y}.yy
As a rough approximation, you could use a recursive make rule to set a couple of parameters, and take it from there, but this is rather clumsy.
.PHONY: all
all:
$(MAKE) -$(MAKEFLAGS) X=1 Y=6 input1_config6.output
$(MAKE) -$(MAKEFLAGS) X=1 Y=7 input1_config7.output
$(MAKE) -$(MAKEFLAGS) X=2 Y=6 input2_config6.output
:
input$X_config$Y.output: input$X.xx config$Y.yy
run_script $^
It would be a lot easier if you provided a complete sample example with a complete set of targets and prerequisites and exactly what you wanted to happen.
Using .SECONDEXPANSION might work, but you're not using it correctly; please re-read the documentation. The critical aspect of .SECONDEXPANSION is that you have to escape the variables that you want to avoid expanding until the second pass. In your example you've not escaped anything, so .SECONDEXPANSION isn't actually doing anything at all here. However, as #tripleee points out it's not easy to use multiple variable values in a single target.
To do this more easily you'll probably want to use eval. Something like this:
define DECLARE
$1_$2.output: $1.xx $2.yy
TARGETS += $1_$2.output
endef
TARGETS :=
$(foreach input,$(INPUTS),$(foreach config,$(CONFIGS),$(eval $(call DECLARE,$(input),$(config)))))
$(TARGETS):
run_script $^
I have another solution using include and bash for loop.
include trees.mk
trees.mk:
#for input in $(INPUTS); do \
for config in $(CONFIGS); do \
echo $${input}_$$config.output : $${input}.xx $$config.yy; \
echo -e '\t run_scipt $$^ ';\
done \
done > $#
At the beginning, trees.mk doesn't exist. The double for loops write out the rule to the target using file redirection >$#.
I got this idea from Managing Projects with GNU Make, Third Edition By Robert Mecklenburg, on
page 56

GNU Make | How to create dependency between source code hierarchy and compiler output h-hy?

Some code from Makefile:
tempDir := ...
javaSources := $(wildcard src/java/**/%.java)
javaClasses := $(subst src/java, $(tempDir)/java/classes, $(subst .java,.class, $(javaSources)))
$(javaClasses): $(javaSources)
mkdir -p $(tempDir)/java/classes || true
javac \
-d $(tempDir)/java/classes \
-cp $(tempDir)/java/classes \
$?
How to create a pattern rule (like here) to preserve in / out order?
#MadScientist
First, your wildcard won't work. GNU make uses only basic shell globbing, which means it can't understand advanced globbing like ** meaning "search all subdirectories". Second, % is not a shell globbing character at all so you're just looking for files that are literally named %.java.
Instead you probably want something like this:
javaSources := $(shell find src/java -name '*.java')
Next, to create the javaClasses content you really don't want to use subst because it substitutes everywhere which can give false matches (e.g., $(subst .x,.y,foo.xbar) will yield foo.ybar which is probably not what you want).
Something like this is simpler to understand:
javaClasses := $(patsubst src/java/%.java,$(tempdir)/java/classes/%.class,$(javaSources))
Finally, you are repeating exactly the same error you made in the previous question, where you tried to list all the targets and all the prerequisites in the same rule. Just as I said for that question, that is not right.
The answer is exactly the same as in the previous question: you should write a pattern rule that describes how to build one single target from one single source file.
And again you need an all target or similar which depends on all the outputs.
In complement to MadScientist answer, you should probably use a pattern rule like:
$(tempDir)/java/classes/%.class: src/java/%.java
mkdir -p $(dir $#)
javac -d $(dir $#) -cp $(dir $#) $<
(not sure what -cp should be in this case, it depends on your specific project). And as MadScientist also suggested, you will need:
.PHONY: all
all: $(javaClasses)
such that you can call make all to compile all the source files that need to be. Put it before any other explicit target if you want all to be the default goal (the goal make selects if you just call make), or use the .DEFAULT_GOAL special variable:
.DEFAULT_GOAL := all

Why does my Makefile pattern rule run its recipe multiple times?

As per the gnu make documentation, a pattern rule's "...recipe is executed only once to make all the targets." However, I have the following Makefile
.PHONY: entrypoint
entrypoint: test_1.cpp test_2.cpp
test_%.cpp:
echo $#
And running make produces:
echo test_1.cpp
test_1.cpp
echo test_2.cpp
test_2.cpp
I'm new to make, and I'm probably misunderstanding something, but the documentation seems misleading if clear.
$ make -v
GNU Make 4.0
...
You're misreading the documentation. It means, the recipe is run only one time assuming that all the target patterns in that rule will be created.
Since you have only one target pattern in your rule (test_%.cpp`) make knows that each time it runs that recipe it will create one output file matching that pattern. To create different targets that match that pattern it will run multiple instances of the recipe.
If you had a rule like this:
%.x %.y %.z :
dothings
then make would expect that a single invocation of the recipe dothings would create all the targets matching this pattern (e.g., foo.x, foo.y, and foo.z).
Contrast this with an explicit rule like this:
foo.x foo.y foo.z :
dothings
Make here treats this exactly as if you'd written this:
foo.x :
dothings
foo.y :
dothings
foo.z :
dothings
That is, to build all three of these targets it would run the recipe three times.
There's no way to tell make "please run this recipe one time and it will produce every single target that could possibly match the pattern foo_%.cpp".
The following functions construct a dynamic list of dependencies of your multiple-target where the non-existent files are named last. This is more or less the method named "Another attempt" in the link you gave, except that it doesn't trip over missing files and is able to make a missing file by giving it as target on the command line. What it does not: execute the multitarget recipe if one of the multitargets is out of date relative to the others, but I think this is more of wanted side effect than a problem. The only drawback is the syntactic ugliness as you have to write it into an eval expression which forces you to quote all variables in the recipe which shall be evaluated at execution time.
define newline :=
endef
list2rules = $(firstword $1) $(if $(word 2,$1),: $(word 2,$1)$(newline)$(call list2rules,$(wordlist 2,1000,$1)))
multitarget = $(call list2rules,$(wildcard $1) $(filter-out $(wildcard $1),$1))
.PHONY: all
targets = test1 test2 footest3
#$(info $(call multitarget,$(targets)))
all: somefile
somefile: $(targets)
touch somefile
# here we generate the dependency list on the spot. Only one recipe to update all targets.
$(eval $(call multitarget,\
$(targets)) : ; \
touch $(targets) \
)

code management: generate source files with slight variations of various rules

I have a source file in a declarative language (twolc, actually) that I need to write many variations on: a normative version and many non-normative versions, each with one or more variations from the norm. For example, say the normative file has three rules:
Rule A:
Do something A-ish
Rule B:
Do something B-ish
Rule C:
Do something C-ish
Then one variation might have the exact same rules as the norm for A and C, but a different rule for B, which I will call B-1:
Rule A:
Do something A-ish
Rule B-1:
Do something B-ish, but with a flourish
Rule C:
Do something C-ish
Imagine that you have many different subtle variations on many different rules, and you have my situation. The problem I am worried about is code maintainability. If, later on, I decide that Rule A needs to be refactored somehow, then I will have 50+ files that need to have the exact same rule edited by hand.
My idea is to have separate files for each rule and concatenate them into variations using cat: cat A.twolc B.twolc C.twolc > norm.twolc, cat A.twolc B-1.twolc C.twolc > not-norm.twolc, etc.
Are there any tools designed to manage this kind of problem? Is there a better approach than the one I have in mind? Does my proposed solution have weaknesses I should watch out for?
As you added the makefile tag, here is a GNU-make-based (and Gnu make only) solution:
# Edit this
RULES := A B B-1 C
VARIATIONS := norm not-norm
norm-rules := A B C
not-norm-rules := A B-1 C
# Do not edit below this line
VARIATIONSTWOLC := $(patsubst %,%.twolc,$(VARIATIONS))
all: $(VARIATIONSTWOLC)
define GEN_rules
$(1).twolc: $$(patsubst %,%.twolc,$$($(1)-rules))
cat $$^ > $$#
endef
$(foreach v,$(VARIATIONS),$(eval $(call GEN_rules,$(v))))
clean:
rm -f $(VARIATIONSTWOLC)
patsubst is straightforward. The foreach-eval-call is a bit more tricky. Long story short: it loops over all variations (foreach). For each variation v, it expands (call) GEN_rules by replacing $(1) by $(v) (the current variation) and $$ by $. Each expansion result is then instantiated (eval) as a normal make rule. Example: for v=norm, the GEN_rules expansion produces:
norm.twolc: $(patsubst %,%.twolc,$(norm-rules))
cat $^ > $#
which is in turn expanded as (step-by-step):
step1:
norm.twolc: $(patsubst %,%.twolc,A B C)
cat $^ > $#
step2:
norm.twolc: A.twolc B.twolc C.twolc
cat $^ > $#
step3:
norm.twolc: A.twolc B.twolc C.twolc
cat A.twolc B.twolc C.twolc > norm.twolc
which does what you want: if norm.twolc does not exist or if any of A.twolc, B.twolc, C.twolc is more recent than norm.twolc, the recipe is executed.

Insert a new-line in a Makefile $(foreach ) loop

I'm compiling a large list of files in a Makefile.
my.list : ${deps}
rm -f $#
$(foreach F,$^,echo "${F}" >> $#;)
but ${deps} can be large and the generated command-line could be too large for one SHELL call. Is it possible to replace ';' by a newline '\n' ?
As already mentioned in Jonathan Wakely's answer, the straightforward answer is
define newline
endef
Interestingly enough, for all chars except newline there is an easier way to get non-printable characters into variables with some help from the shell, e.g.:
tab := $(shell printf '\011')
It won't work here, though, because the builtin shell function replaces all newlines by spaces.
The version above is still a bit fragile though, in particular when combining with automake which will silently remove the second consecutive newline from the input file, turning newline into an empty variable. To force it to keep the newline, we can extend the snippet:
blank :=
define newline
$(blank)
endef
Finally, to actually get a separate shell command for each dependency, you need to run the generated string through eval:
define generate-rule =
my.list : $(1)
rm -f $$#
$(foreach F,$$^,$(newline)$(tab)echo "${F}" >> $#)
endef
$(eval $(call generate-rule,$(deps)))
You can define a variable that expands to a newline like so:
define NEWLINE
endef
Now you can use $(NEWLINE) to expand to a newline.
This won't change anything though, the foreach will still generate a single command with embedded newlines between the statements, rather than a single command with ; between the statements.
Possibly the most straight-forward answer is to use:
#printf "\n"
You can also use this, as I commonly do, to show a description when each new make target executes. An example target:
.PHONY: clean-version
clean-version: ## Reset version back to committed version number
#printf "\n## Reset version to last committed version number ##\n\n"; \
$(GIT) checkout $(VERSION_FILE);
Replacing ';' by a carriage-return will produce a string of the same size, subject to the same problem.
"foreach" is simply a string expansion. If you want to execute a separate command for each item, you can use a for loop.
my.list : ${deps}
rm -f $#
(for F in $^; do echo $$F >> $# ; done)
Edit -- after some revisions, it looks like the only problem with my original was not due to the whitespaces, but with MAKE_O. I've fixed it in my version, but I'll mostly be removing them below.
By the way the original post was written, I'm not sure if my solution will be relevant. However, I found myself in the middle of a define already inside a foreach, and couldn't figure out how to insert a newline using any of the other answers as given.
Solution:
define BR
$(1)
endef
define MAKE_O
$(1): $(wildcard $(1:obj/%.o=src/%.cpp)); \
$(CXX) ... -c $$< \$(call BR)-o $1 ... \
\$(call BR)&& touch $$#
endef
$(foreach N,main.o,$(eval $(call MAKE_O,$(N))))
Desired output (compilation is truncated from auto-dependency generation, hence the touch):
> make obj/main.o
g++ ... -c src/main.cpp \
-o obj/main.o ... \
&& touch obj/main.o
I changed BR to perform the indentation but leave the end of the line up to you:
define BR
$(1)
$(1:%= ) #<remove this comment
endef
define MAKE_O
$(1): $(wildcard $(1:obj/%.o=src/%.cpp)); \
$(CXX) ... -c $$< $(call BR,\)-o $1 ... \
$(call BR,\)&& touch $$#
endef
$(foreach N,main.o,$(eval $(call MAKE_O,$(N))))
The markup won't help to show this, but line 2 of BR is $(1:%=_space_)_tab_ (the comment itself is not allowed.) Result:
> make -n obj/main.o
g++ obj/main.o -c \
-o obj/main.o && \
echo statement on new line
I used $(call BR,\) so that the newline was not parsed as an escape of the new line, and $(1:%=space)tab so that the tab is forced (many similar rules have been defined, like SPACE:=$(SPACE) $(SPACE) without a prior value.) The variable left of the whitespace must evaluate to something. To be clear, removing the whitespace before and after the call: ...lastword$(call BR,\)firstword... yields ...lastword\n\tfirstword..., or written out,
[^]...lastword \[$]
[^]$(call BR,\)firstword...[$]
...yields...
[^]...lastword \[$]
[^] firstword...[$]
to achieve the same (using ^,$ to denote the beginning and end of the line. Someone else will probably know how to format/annotate this better.)
My syntax highlighter is decidedly unimpressed with the 'escaped' parentheses and trailing whitespace, but the result is decent.

Resources