I have a rule from OP-TEE (uses large makefile include and make invocation trees to compile two operating system kernels and applications) with some immediately expanded variables -- as far as I can tell, there is no second expansion relevant here, since the double double dollar is in the rule and thus not subject to second expansion. It should always be expanded in the second phase:
$(link-out-dir$(sm))/$(user-ta-uuid).elf: $(objs) $(libdeps) \
$(link-script-pp$(sm))
#$(cmd-echo-silent) ' LD $$#'
echo Target '$#', '$$#', "$#", "$$#"
In the output, one can see that all occurrences of $#and $$# are expanded by make, not by sh, but the single dollar ones are not filled with the correct value, and the double ones did not escape themselves:
echo Target '', 'out/someuuid.elf', "", "out/someuuid.elf"
Target , out/someuuid.elf, , out/someuuid.elf
What i was expecting was:
echo Target 'out/someuuid.elf', '$#', "out/someuuid.elf", "$#"
Target out/someuuid.elf, $#, out/someuuid.elf,
Which mechanism of make could be the reason for the expansion to work differently here?
This target is nested two layers deep in other makefiles and four make invocations, thus it's a bit difficult to find out what the buildchain has configured differently here.
I cannot reproduce this behaviour with a minimal makefile:
all: target1 target2 target3
.PHONY: target1 target2 target3 all
.SUFFIXES:
target1:
echo Target1 '$#', '$$#', "$#", "$$#"
# should make no difference, since used in immediate context on rule specification
TARGET2 := target2
TARGET3 = target3
$(TARGET2):
echo Target2 '$#', '$$#', "$#", "$$#"
$(TARGET3):
echo Target3 '$#', '$$#', "$#", "$$#"
yielding the expected output with $# expanding to the target name and $$# expanding to a string or empty list of positional shell parameters:
echo Target1 'target1', '$#', "target1", "$#"
Target1 target1, $#, target1,
echo Target2 'target2', '$#', "target2", "$#"
Target2 target2, $#, target2,
echo Target3 'target3', '$#', "target3", "$#"
Target3 target3, $#, target3,
Specifying .SECONDEXPANSION does not change the results, as expected.
Syntax highlighting can be tricky -- I wasn't aware of the whole target being content of a variable itself -- some targets above there was a define gen-link-t beginning a multiline variable, ending some targets below.
The target was actually invoked after generation using $(eval $(call gen-link-t)) at the end of the file, instead of being read in line-by-line -- causing the first expansion of $$ to $ to happen when expanding the variable within the $(call), making the call return the $ as plain text. The second evaluation happened when make was reading this target again via $(eval)
Related
I have a Makefile with user-specified input files in the variable INPUT_FILES.
For each input file, I need to create an input file prime.
Some notes:
Each input file can have an arbitrary file location
It is reasonable to assume there aren't duplicate filenames
Each output file needs to go into $(OUTPUT_DIR)
My basic strategy has been to generate the set of targets based INPUT_FILES and then try to determine which input file is the actual dependency of the target.
A few variations I've tried:
# Create a list of targets
OUTPUT_FILES = $(foreach file,$(notdir $(INPUT_FILES)),$(OUTPUT_DIR)/$(file))
# This doesn't work, because all input files are dependencies of each output file
$(OUTPUT_FILES): $(INPUT FILES)
program --input $^ --output $#
# This doesn't work because $# hasn't been resolved yet
$(OUTPUT_FILES): $(filter,$(notdir $#),$(INPUT FILES))
program --input $^ --output $#
# This doesn't work, I think because $# is evaluated too late
.SECONDEXPANSION:
$(OUTPUT_FILES): $(filter,$(notdir $$#),$(INPUT FILES))
program --input $^ --output $#
# This doesn't work either
.SECONDEXPANSION:
$(OUTPUT_FILES): $$(filter,$(notdir $#),$(INPUT FILES))
program --input $^ --output $#
I've looked into static pattern rules as well, but I'm not sure if it can help with what I need.
In your case .SECONDEXPANSION: works because you can use make functions (filter) to compute the prerequisite of each output file. In other circumstances it could be impossible. But there is another GNU make feature that can be used in cases like yours: if you use GNU make you can programmatically instantiate make statements using foreach-eval-call. Just remember that the macro that is used as the statements pattern gets expanded twice, reason why you must double some $ signs (more on this later):
OUTPUT_DIR := dir
OUTPUT_FILES := $(addprefix $(OUTPUT_DIR)/,$(notdir $(INPUT_FILES)))
.PHONY: all
all: $(OUTPUT_FILES)
# The macro used as statements pattern where $(1) is the input file
define MY_RULE
$(1)-output-file := $(OUTPUT_DIR)/$$(notdir $(1))
$$($(1)-output-file): $(1)
#echo program --input $$^ --output $$#
endef
$(foreach i,$(INPUT_FILES),$(eval $(call MY_RULE,$(i))))
Demo:
$ mkdir -p a/a b
$ touch a/a/a b/b c
$ make INPUT_FILES="a/a/a b/b c"
program --input a/a/a --output dir/a
program --input b/b --output dir/b
program --input c --output dir/c
Explanation:
When make parses the Makefile it expands $(foreach ...): it iterates over all words of $(INPUT_FILES), for each it assigns the word to variable i and expands $(eval $(call MY_RULE,$(i))) in this context. So for word foo/bar/baz it expands $(eval $(call MY_RULE,$(i))) with i = foo/bar/baz.
$(eval PARAMETER) expands PARAMETER and instantiates the result as new make statements. So, for foo/bar/baz, make expands $(call MY_RULE,$(i)) with i = foo/bar/baz and considers the result as regular make statements. The expansion of $(eval ...) has no other effect, the result is the empty string. This is why in our case $(foreach ...) expands as the empty string. But it does something: create new make statements dynamically for each input file.
$(call NAME,PARAMETER) expands PARAMETER, assigns it to temporary variable 1 and expands the value of make variable NAME in this context. So, $(call MY_RULE,$(i)) with i = foo/bar/baz expands as the expanded value of variable MY_RULE with $(1) = foo/bar/baz:
foo/bar/baz-output-file := dir/$(notdir foo/bar/baz)
$(foo/bar/baz-output-file): foo/bar/baz
#echo program --input $^ --output $#
which is what is instantiated by eval as new make statements. Note that we had a first expansion here and the $$ became $. Note also that call can have more parameters: $(call NAME,P1,P2) will do the same with $(1) = P1 and $(2) = P2.
When make parses these new statements (as any other statements) it expands them (second expansion) and finally adds the following to its list of variables:
foo/bar/baz-output-file := dir/baz
and the following to its list of rules:
dir/baz: foo/bar/baz
#echo program --input $^ --output $#
This may look complicated but it is not if you remember that the make statements added by eval are expanded twice. First when $(eval ...) is parsed and expanded by make, and a second time when make parses and expands the added statements. This is why you frequently need to escape the first of these two expansions in your macro definition by using $$ instead of $.
And it is so powerful that it is good to know.
When asking for help please provide some kind of actual example names so we can understand more clearly what you have. It also helps us use terminology which is not confusing.
You really want to use $< in your recipes, not $^, I expect.
IF your "input files" are truly input-only (that is, they are not themselves generated by other make rules) then you can easily solve this problem with VPATH.
Just use this:
VPATH := $(sort $(dir $(INPUT_FILES)))
$(OUTPUT_DIR)/% : %
program --input $< --output $#
I finally found a permutation that works - I think the problem was forgetting that filter requires a % for matching patterns. The rule is:
.SECONDEXPANSION:
$(OUTPUT_FILES): $$(filter %$$(#F),$(INPUT_FILES))
program --input $^ --output $#
I also realized I can use #F (equivalent to $$(notdir $$#)) for cleaner syntax.
The rule gets the target's filename on its second expansion ($$(#F)) and then gets the input file (with path) that corresponds to it on second expansion ($$(filter %$$(#F),$(INPUT_FILES))).
Of course, the rule only works if filenames are unique. If someone has a cleaner solution, feel free to post.
In a Makefile with multiple targets, how can one prevent the prerequisites of targets that are not being used from expanding? See the following example:
thisexpands = $(warning Expanded!)
.PHONY: target1
target1: $(thisexpands)
#echo Target 1
.PHONY: target2
target2:
#echo Target 2
Calling target2 forces thisexpands to expand, even though it is lazily evaluated and it and target1 are never used.
In my real world case expanding thisexpands when calling target1 is an issue, because it is a shell command that prints errors when called out of context of target1 and it's parent targets(not shown here).
Makefiles are fully parsed before the first rule is run. As part of parsing, all the targets and prerequisites must be expanded. You can find details of when expansion happens for different parts of a makefile in How make Reads a Makefile in the GNU make manual.
One way is to use recursion:
thisexpands = $(warning Expanded!)
target1: ; $(MAKE) target1-recurse T1_DEPS='$(value thisexpands)'
T1_DEPS =
target1-recurse: $(T1_DEPS)
#echo Target 1
This doesn't work:
You can probably defer expansion by using secondary expansion, something like this:
.SECONDEXPANSION:
target1: $$(thisexpands)
Be very careful that you escape the prerequisite list appropriately.
There's no way to cancel the expansion completely. However, you can use the conditional assignment based on the value of $(MAKECMDGOALS):
thisexpands = $(if $(filter target1,$(MAKECMDGOALS)),$(warning Expanded!))
.PHONY: target1
target1: $(thisexpands)
#echo Target 1
.PHONY: target2
target2:
#echo Target 2
Note that it works if target1 is only built explicitly (make target1) and not by default, or as a part of building another target.
Consider the following simple makefile:
define cat_deps
$(shell cat $(1).dep)
endef
TARGET := main.o
all:$(TARGET)
#echo Done...!
%.o:$(call cat_deps,$*)
#echo $*
In target main.o variable $(1) in cat_desp must be main
but it is empty string. Also $(call cat_deps,%) does not work
and in this case variable $(1) becomes %. Note
that file main.dep exists. Please help me.
In target main.o variable $(1) in cat_desp must be main but it is empty string.
$1 in cat_deps actually expands to main, if you pass main to the function cat_deps. However, you are passing an empty string. In your rule:
%.o: $(call cat_deps,$*)
#echo $*
the $* above is expanded before the rule is being matched, and before that rule being matched, the automatic variable $* is actually empty.
You need to make use of secondary expansion in order to expand both the function call to cat_deps and the automatic variable $* after the rule is matched, i.e.:
.SECONDEXPANSION:
%.o: $$(call cat_deps,$$*)
#echo $*
Note the additional $ for both call and $*.
Note that you also need the following rule without recipe:
%: %.o
in order to suppress that implicit rule. Otherwise, a prerequisite of the new %.o rule above could match that (not-suppressed) implicit rule, which has a .o file as prerequisite, and will, in turn, match the new %.o rule again.
I have a setup where the files I want to process with make are dependent on the output of another program. Building the program and all its prerequisites
is also a complicated task so I would like to use make for this as well. Now my problem is, that it doesn't seem that one can generate rules and prerequisites in Makefile recipes. Consider the following code:
bar:
echo target1 target2 target3 > bar
foo: bar
$(eval BAR := $(shell cat bar))
define FUN
$(1):
touch a$(1)
endef
ifdef BAR
$(foreach i,$BAR,$(eval $(call FUN,$(i))))
endif
blub: foo $(BAR)
I replaced a big set of complicated recipes that lead to the generation of the list of files I want to have in the end by the bar recipe. In reality, producing the content of bar is very complicated and should be done by a set of Makefile recipes and cannot just be done by (as the above suggests):
BAR:=$(shell echo target1 target2 target3)
I would like to put the foreach loop into the recipe for foo but that fails with prerequisites cannot be defined in recipes which makes sense and is also explained in function define in makefile
But it seems that when I do make blub that at the time when foo eval's BAR to a different value, the prerequisites for blub are not re-evaluated.
So I think ultimately I'm looking for two things:
how do I generate recipes dynamically at runtime, based on (and dependent on) what another recipe (bar in this case) outputs?
how do I update the prerequisites of a target (blub in this case) dynamically at runtime, based on (and dependent on) what another recipe (bar in this case) outputs?
Thank you!
EDIT: SOLUTION
With the help of #user657267 the following seems to solve my problem:
.PHONY: all
all: blub
-include bar.make
.PHONY: blub
blub: $(BAR)
echo $^
bar.make: Makefile
printf 'BAR=target1 target2 target3\n' > $#
printf 'target1 target2 target3:\n' >>$#
printf '\ttouch $$#' >> $#
.PHONY: clean
clean:
rm -f target1 target2 target3 bar.make
Sounds like you should be using make's self-remaking features
-include bar.make
blub: $(BAR)
#echo $^
bar.make:
#echo BAR := target1 target2 target3 > $#
#echo target1 target2 target3: ; touch $$# >> $#
Obviously the recipes for bar.make are contrived, in the real world they'd probably invoke some kind of script that outputs a valid makefile.
I have the following makefile I use to generate files from some templates, the generated files have two possible extensions:
%.tex: %.tex*_tpl
./generate $#_tpl -o $#
%.xml: %.xml*_tpl
./generate $#_tpl -o $#
The dependency list will here match things like a.tex_tpl, a.tex-subpart1_tpl, a.tex-subpart2_tpl.
While this works, is there a way to avoid repetition? For example by matching *.{tex,xml} in the rule name and use the whole matched name in the dependency list? Something that would look like that:
%.{tex,xml}: $#_tpl
./generate $< -o $#
(Though I know %.{tex,xml} is not a valid rule name and you can't use $# in the dependency list)
Or any other (cleaner?) way.
Seems to me this does what you are looking for:
#
# I've assumed that files of the form:
#
# a.xml_tpl
# b.tex_tpl
#
# determine what targets you want to build
#
TARGETS:=$(patsubst %_tpl,%,$(wildcard *.xml_tpl *.tex_tpl))
.PHONY: all
all: $(TARGETS)
.SECONDEXPANSION:
$(TARGETS): %: $$(wildcard %*_tpl)
./generate $^ -o $#
The key is to use .SECONDEXPANSION to allow $$(wildcard %*_tpl) to be evaluated in a second expansion phase. The double $ is not a typo, by the way; it protects the expression from being evaluated at the time of the first expansion.
If I populate a directory with these files:
a.tex-subpart1_tpl
a.tex_tpl
a.xml-subpart1_tpl
a.xml-subpart2_tpl
a.xml_tpl
and run make -n, I get this on the console:
./generate a.xml_tpl a.xml-subpart1_tpl a.xml-subpart2_tpl -o a.xml
./generate a.tex_tpl a.tex-subpart1_tpl -o a.tex
Why the Second Expansion?
Without the second expansion, you'd have to have $(wildcard %*_tpl) in the dependency because with the $$ the wildcard function would never execute. Instead, make would treat $$(wildcard..) literally as the dependency, which is obviously wrong.
Ok, so $(wildcard %*_tpl) would be evaluated at the time make first runs across that line (this is the "first expansion"). At that time % has no value yet so wildcard would roughly be doing something like what would be ls %*_tpl at the command line.
For reasons of speed, make does not by default give you the opportunity to do any evaluation later than during the first expansion. If you want a later opportunity you have to specify .SECONDEXPANSION, which turns on the second expansion processing. Make still performs the firts expansion as usual. This is why you need to have $$(wildcard: it is transformed to $(wildcard during the first expansion. At the time of the second expansion make sees $(wildcard %*_tpl), replaces % with the actual stem and then executes the wildcard function with the actual stem rather than with a literal %.
Why $(TARGETS) in the Pattern Rule?
The pattern rule could be written:
%: $$(wildcard %*_tpl)
./generate $^ -o $#
without $(TARGETS). However, this rule would do nothing, as it would be a "match-anything rule". Basically, if make took such a rule at face value, then the computation cost would be significant, and most likely it is not the case that the author of the Makefile really means to apply this rule to any file whatsoever. So such a rule comes with restrictions, which in the Makefile here make it useless.
Adding $(TARGETS) makes it into a static pattern rule, which is not a match-anything rule. The addition of $(TARGETS) in front of the target pattern tells make that the rule applies only to these targets, and nothing else.