How to expand variables after entering custom define that wraps foreach? - makefile

Question

I have many very similar $(foreach..) loops in a makefile that work like in the target_old target below:
define NL
endef
extras=some/path/
vars=a b c
all: target_old target_new
target_old:
# foreach and some multiple evals inside and some multiple commands inside
$(foreach var, ${vars}, \
$(eval extra := ${extras}${var}) \
#echo var is ${var} and extras is ${extra}$(NL) \
)
# my try to wrap it in a function
define foreach_vars
$(foreach var, ${vars},
$(eval extra := ${extras}${var}) \
$1$(NL) \
)
endef
target_new:
#echo this is wrong:
$(call foreach_vars, \
#echo var is ${var} and extras is ${extra} \
)
I have many multiple such foreach loops with all the same evals inside the foreach. So I wanted to wrap the foreach loop with the evals inside my own function in foreach_vars. So I don't have to $(eval extra := ${extras}${var}) inside each foreach call. I created target_new target to test it. I would want the output from both targets to be the same, make target_old prints:
$ make target_old
var is a and extras is some/path/a
var is b and extras is some/path/b
var is c and extras is some/path/c
However target_new doesn't pick the ${var} from inside the loop, and ${var} just expands to nothing:
$ make target_new
this is wrong:
var is and extras is
var is and extras is
var is and extras is
I guess this is because the expansion happens before entering the $(call...). Is there any method I can use to "defer" the expansion of arguments inside the $(call...) call until inside foreach inside my function? Is is possible to write a custom foreach-like macro in make? Is there just other method used to implement such functionality?

Answer 1

Yes, your problem comes from the expansion(s) that do not happen when you would like and in the order you would like.
Your use of make is quite unusual because you are using make constructs (foreach, eval, call...) in recipes that are normally plain shell. I guess you have a very good reason but wouldn't it be much easier if you were separating the make world and the shell world? Like in the following, for instance:
extras := some/path/
vars := a b c
target_old:
#for var in $(vars); do \
extra=$(extras)$${var}; \
echo var is $${var} and extra is $${extra}; \
done
It uses make variables (vars, extras) and shell variables (extra, var). The recipe is plain shell. Note the $$ used to escape the first expansion by make such that the shell expansion ${xxx} is done by the shell. Note also the line continuations (\) that form a single line recipe, despite the look. As each line of a make recipe is executed by a separate shell, this is needed to pass shell variables between commands of the shell script.
If you wish, you can also wrap the shell for loop in a make recursively expanded variable:
for = for var in $(vars); do $(1); done
target_new:
#$(call for,extra=$(extras)$${var}; echo var is $${var} and extra is $${extra})

Answer 2

${var} gets immediately expanded, so it needs to be escaped as $${var}. This itself does not fix the issue, since now $1 contains a literal ${var}, which does NOT get expanded within foreach. I would make a simple subst though to get it fixed, e.g.:
$ cat Makefile
define NL
endef
extras=some/path/
vars=a b c
define foreach_vars
$(foreach var, ${vars},
$(eval extra := ${extras}${var}) \
$(subst $$(var),$(var), \
$(subst $$(extra),$(extra), \
$(1))) \
$(NL) \
)
endef
target_new:
$(call foreach_vars, \
#echo var is $$(var) and extras is $$(extra) \
)
Output:
$ make target_new
var is a and extras is some/path/a
var is b and extras is some/path/b
var is c and extras is some/path/c

Answer 3

When make comes to build target_new (like when you type make target_new for instance):
It expands the whole recipeImportant: The recipe is expanded before firing up the shell
For each line of the resulting expansion, it passes one at a time to a fresh invocation of the shell
It's worth showing the expansion make does in painful detail. We have as the recipe:
#echo this is wrong:
$(call foreach_vars, \
#echo var is ${var} and extras is ${extra} \
)
First off, ${var} becomes empty, as is ${extra}
make is left with $(call foreach_vars, #echo var is and extras is ). Now for the call:
1 is set to #echo var is and extras is
make expands $(foreach var, ${vars}, $(eval extra := ${extras}${var}) $1$(NL) )
${vars} is a b c
First iteration:
var is set to a
Make evals extra := some/path/a
The expansion of the eval is empty however, and we are left with $1$(NL) (modulo some whitespace), leaving #echo var is and extras is
Second iteration: ${extra} becomes some/path/b, and we are again left with #echo var is and extras is
Last iteration: ${extra} becomes some/path/c, and we are again left with #echo var is and extras is
The final recipe then:
#echo this is wrong:
#echo var is and extras is
#echo var is and extras is
#echo var is and extras is
which produces the output you described.
So how do we avoid the early expansion of parameters?
Once nice solution is to stick the command-line you want into a variable,
and pass the name of that variable instead.
define foreach_vars # 1: variable containing command-line
$(foreach var,${vars},
$(eval extra := ${extras}${var}) \
${$1}$(NL) \
)
endef
cmds<target_new> = #echo var is ${var} and extras is ${extra}
target_new:
#echo this is right:
$(call foreach_vars,cmds<$#>)
Why mangle the variable name with the name of the target? Lookup tables are nice, and you may find many targets ending up with the same recipe.
cmds<target_new> = #echo var is ${var} and extras is ${extra}
cmds<target_beta> = ${MAKE} ${var}-${extra}
cmds<target_release> = script ${var} | eat ${extra}
target_new target_beta target_release:
$(call foreach_vars,cmds<$#>)
etc.