How to expand variables after entering custom define that wraps foreach? - makefile

I have many very similar $(foreach..) loops in a makefile that work like in the target_old target below:
define NL
endef
extras=some/path/
vars=a b c
all: target_old target_new
target_old:
# foreach and some multiple evals inside and some multiple commands inside
$(foreach var, ${vars}, \
$(eval extra := ${extras}${var}) \
#echo var is ${var} and extras is ${extra}$(NL) \
)
# my try to wrap it in a function
define foreach_vars
$(foreach var, ${vars},
$(eval extra := ${extras}${var}) \
$1$(NL) \
)
endef
target_new:
#echo this is wrong:
$(call foreach_vars, \
#echo var is ${var} and extras is ${extra} \
)
I have many multiple such foreach loops with all the same evals inside the foreach. So I wanted to wrap the foreach loop with the evals inside my own function in foreach_vars. So I don't have to $(eval extra := ${extras}${var}) inside each foreach call. I created target_new target to test it. I would want the output from both targets to be the same, make target_old prints:
$ make target_old
var is a and extras is some/path/a
var is b and extras is some/path/b
var is c and extras is some/path/c
However target_new doesn't pick the ${var} from inside the loop, and ${var} just expands to nothing:
$ make target_new
this is wrong:
var is and extras is
var is and extras is
var is and extras is
I guess this is because the expansion happens before entering the $(call...). Is there any method I can use to "defer" the expansion of arguments inside the $(call...) call until inside foreach inside my function? Is is possible to write a custom foreach-like macro in make? Is there just other method used to implement such functionality?

Yes, your problem comes from the expansion(s) that do not happen when you would like and in the order you would like.
Your use of make is quite unusual because you are using make constructs (foreach, eval, call...) in recipes that are normally plain shell. I guess you have a very good reason but wouldn't it be much easier if you were separating the make world and the shell world? Like in the following, for instance:
extras := some/path/
vars := a b c
target_old:
#for var in $(vars); do \
extra=$(extras)$${var}; \
echo var is $${var} and extra is $${extra}; \
done
It uses make variables (vars, extras) and shell variables (extra, var). The recipe is plain shell. Note the $$ used to escape the first expansion by make such that the shell expansion ${xxx} is done by the shell. Note also the line continuations (\) that form a single line recipe, despite the look. As each line of a make recipe is executed by a separate shell, this is needed to pass shell variables between commands of the shell script.
If you wish, you can also wrap the shell for loop in a make recursively expanded variable:
for = for var in $(vars); do $(1); done
target_new:
#$(call for,extra=$(extras)$${var}; echo var is $${var} and extra is $${extra})

${var} gets immediately expanded, so it needs to be escaped as $${var}. This itself does not fix the issue, since now $1 contains a literal ${var}, which does NOT get expanded within foreach. I would make a simple subst though to get it fixed, e.g.:
$ cat Makefile
define NL
endef
extras=some/path/
vars=a b c
define foreach_vars
$(foreach var, ${vars},
$(eval extra := ${extras}${var}) \
$(subst $$(var),$(var), \
$(subst $$(extra),$(extra), \
$(1))) \
$(NL) \
)
endef
target_new:
$(call foreach_vars, \
#echo var is $$(var) and extras is $$(extra) \
)
Output:
$ make target_new
var is a and extras is some/path/a
var is b and extras is some/path/b
var is c and extras is some/path/c

When make comes to build target_new (like when you type make target_new for instance):
It expands the whole recipeImportant: The recipe is expanded before firing up the shell
For each line of the resulting expansion, it passes one at a time to a fresh invocation of the shell
It's worth showing the expansion make does in painful detail. We have as the recipe:
#echo this is wrong:
$(call foreach_vars, \
#echo var is ${var} and extras is ${extra} \
)
First off, ${var} becomes empty, as is ${extra}
make is left with $(call foreach_vars, #echo var is and extras is ). Now for the call:
1 is set to #echo var is and extras is
make expands $(foreach var, ${vars}, $(eval extra := ${extras}${var}) $1$(NL) )
${vars} is a b c
First iteration:
var is set to a
Make evals extra := some/path/a
The expansion of the eval is empty however, and we are left with $1$(NL) (modulo some whitespace), leaving #echo var is and extras is
Second iteration: ${extra} becomes some/path/b, and we are again left with #echo var is and extras is
Last iteration: ${extra} becomes some/path/c, and we are again left with #echo var is and extras is
The final recipe then:
#echo this is wrong:
#echo var is and extras is
#echo var is and extras is
#echo var is and extras is
which produces the output you described.
So how do we avoid the early expansion of parameters?
Once nice solution is to stick the command-line you want into a variable,
and pass the name of that variable instead.
define foreach_vars # 1: variable containing command-line
$(foreach var,${vars},
$(eval extra := ${extras}${var}) \
${$1}$(NL) \
)
endef
cmds<target_new> = #echo var is ${var} and extras is ${extra}
target_new:
#echo this is right:
$(call foreach_vars,cmds<$#>)
Why mangle the variable name with the name of the target? Lookup tables are nice, and you may find many targets ending up with the same recipe.
cmds<target_new> = #echo var is ${var} and extras is ${extra}
cmds<target_beta> = ${MAKE} ${var}-${extra}
cmds<target_release> = script ${var} | eat ${extra}
target_new target_beta target_release:
$(call foreach_vars,cmds<$#>)
etc.

Related

How does the makefile write user-defined variables to the yaml file to support vagrantfile fetch parameters

I have a project which use makefile to control vagrant, I want to put the vagrant parameter into the makefile, such as cpu, memory, ip, hostname, forwarded_port and the like. I find a way that vagrantfile read yaml file to parameterize vagrantfile. So makefile needs a target to read all the user option variables and write them to config.yaml as key-value pairs.
The sample is as follows
# === BEGIN USER OPTIONS ===
BOX_OS ?= fedora
# Box setup
#BOX_IMAGE
# Disk setup
DISK_COUNT ?= 1
DISK_SIZE_GB ?= 25
# VM Resources
MASTER_CPUS ?= 2
MASTER_MEMORY_SIZE_GB ?= 2
NODE_CPUS ?= 2
NODE_MEMORY_SIZE_GB ?= 2
NODE_COUNT ?= 2
# Network
MASTER_IP ?= 192.168.26.10
NODE_IP_NW ?= 192.168.26.
POD_NW_CIDR ?= 10.244.0.0/16
...
...
# === END USER OPTIONS ===
The echo command does achieve it
# Makefile
envInit:
#echo "POD_NW_CIDR : \"$(POD_NW_CIDR)\"" > ${FILECWD}/configs.yaml
But too many variables can be too complex.
Is there a way to bulk read variables and their values and write them to a yml file
I would very appreciate it if you guys can tell me how to achieve it that bulk read variables and their values and write them to a yml file.
Define all user options (along with the default values) as a list, so that they are iterable:
# list of user options with default values
userOptions = \
BOX_OS=2 \
DISK_COUNT=1 \
MASTER_IP=192.168.26.10
# replace each default value with the env value, if any
userOptionValues = $(foreach i, $(userOptions), \
$(word 1, $(subst =, ,$i))=$(or \
$($(word 1, $(subst =, ,$i))), $($(word 1, $(subst =, ,$i))), $(word 2, $(subst =, ,$i))))
# write the yaml file
envInit:
# empty the file
#printf "" > configs.yaml
# write a line for each option
#for i in $(userOptionValues); do \
printf "%s : %s\n" "$$(printf $$i | cut -d= -f1)" "$$(printf $$i | cut -d= -f2)" >> configs.yaml; \
done
#flyx Thank you for you answer, your code does work great. But I seem to have found a more convenient way, and I've partially modified it.
printvars:
#echo$(foreach V,$(sort $(.VARIABLES)), \
$(if $(filter-out environment% default automatic,$(origin $V)),$(info $V: $($V))))
But there is still a gap between achieving the goal.
# the Makefile test file
FILECWD = $(shell pwd)
# === BEGIN USER OPTIONS ===
CLOUD_IP ?= 192.168.79.222
CLOUD_NAME ?= cloud
CLOUD_CPU ?= 6
CLOUD_MEMORY ?= 8
# === END USER OPTIONS ===
printvars:
#echo$(foreach V,$(sort $(.VARIABLES)), \
$(if $(filter-out environment% default automatic,$(origin $V)),$(info $V: $($V))))
make printvars's output contains a number of other variables
$ make printvars
.DEFAULT_GOAL: printvars
CLOUD_IP: 192.168.79.222
CLOUD_MEMORY: 8
CLOUD_NAME: cloud
CURDIR: /testmakecreateyml0930
FILECWD: /testmakecreateyml0930
GNUMAKEFLAGS:
MAKEFILE_LIST: Makefile
MAKEFLAGS:
SHELL: /bin/sh
And it can only be printed and not exported to the yaml file.This is only one step away from success.
I would appreciate it if you could help me modify it to achieve my goal
You can write directly to a file with GNUmakes $(file) function:
define newline :=
$(strip)
$(strip)
endef
space := $(strip) $(strip)#
-never-matching := ¥# character 165, this is used as a list element that should never appear as a real element
option-names = $(subst $(-never-matching),,$(filter $(-never-matching)%,$(subst $(-never-matching)$(space),$(-never-matching),$(-never-matching)$(strip $(subst $(newline), $(-never-matching),$1)))))
# define your user options in as many separate parts as you like, spaces and empty lines included:
define USER_OPTIONS +=
a = spaces are no problem
b = "neither nearly all 'other' characters: 8&)("
endef
define USER_OPTIONS +=
c = baz baf
d = foobar
endef
# make all definition make variables verbatim
$(eval $(USER_OPTIONS))
YAML_FORMAT := $(foreach name,$(call option-names,$(USER_OPTIONS)),$(newline)$(name) : $($(name)))
# write the file. Warning: this happens before any rule is run!
$(file >test.yaml,$(YAML_FORMAT))
$(info $(foreach name,$(call option-names,$(USER_OPTIONS)),<$(name) : $($(name))> ))
The trick lies in the clustering of all relevant user option variables in one multi-line make variable. The function option-names pulls all identifiers from that variable into a separate list.
I took the newline etc. character definitions from the GNUmake table toolkit which has many functions for "programmatic" make.

Make args: is there a variable that contains all of them?

In make, I can define a file as follows:
.PHONY: echo-foo
echo-foo:
echo ${foo}
Usage:
make echo-foo foo=foo
Now suppose I would like to do the following:
make echo-vars foo=foo bar=bar
And I start by writing:
echo-vars:
echo ${???}
Is there a variable automatically defined which will output:
// either
foo=foo bar=bar
// or
foo bar
In other words: is there a MAKEARGS or some variable along those lines defined as part of the make env?
There's a special variable just for that:
echo-vars:
echo ${MAKEOVERRIDES}
.VARIABLES is somewhat close.
This prints only command-line variables:
test:
echo "$(foreach VAR_NAME,$(.VARIABLES),\
$(if $(filter command line,$(origin $(VAR_NAME))),$(VAR_NAME)))"
$(filter command line,$(origin VAR_NAME)) equals to "command line" if it's the value of $(origin VAR_NAME), and is empty otherwise. $(filter <needle>,<haystack>) is a make's boolean test for "<needle> is in or equals to <haystack> and is not empty".
$(if <condition>,<value>) returns <value> if <condition> is nonempty, otherwise an empty string.
$(foreach VAR_NAME,SET,EXPRESSION) returns a joined result of EXPRESSION applied to each element of a (space-separated) SET, where VAR_NAME is substituted with each element of the SET.
Add $(strip) to get rid of excess spaces.

How to concatenate strings in a Makefile?

I have a Makefile that runs pandoc. I want to turn a list of extensions:
PANDOC_EXTENSIONS = \
multiline_tables \
some_other_extension
into a string that looks like:
PANDOC_EXTENSION_LIST = +multiline_tables+some_other_extension
which will then be passed as a command line option to pandoc like this:
pandoc --from$(PANDOC_EXTENSION_LIST) ...
It's trivial in almost any programming language, but I can't figure out how to do this with the patsubst or subst functions, since make doesn't really have lists. Any ideas?
Here:
Makefile
PANDOC_EXTENSIONS = \
multiline_tables \
some_other_extension
$(foreach word,$(PANDOC_EXTENSIONS),$(eval PANDOC_EXTENSION_LIST := $(PANDOC_EXTENSION_LIST)+$(word)))
.PHONY: all
all:
echo $(PANDOC_EXTENSION_LIST)
Which runs like:
$ make
echo +multiline_tables+some_other_extension
+multiline_tables+some_other_extension
As this illustrates, GNU make really does have lists. A sequence of whitespace-separated words is a list.
Based on example in documentation:
empty:=
space:=$(empty) $(empty)
PANDOC_EXTENSIONS = \
multiline_tables \
some_other_extension
all:
#echo +$(subst ${space},+,${PANDOC_EXTENSIONS})
The result:
$ gmake
+multiline_tables+some_other_extension

How to declare a deferred variable that is computed only once for all?

I have a shell program that takes ages to complete. As written, executing make build takes 4 x 2 seconds to complete because $(value) is computed for each file.
A solution is to declare value a deferred variable by using := instead of =.
Unfortunately this is not a solution either because it slows down the execution of make clean and any other targets by 2 seconds because value is computed for nothing.
value = $(shell sleep 2 && echo 42)
in = a b c d
out = $(addsuffix .out,$(in))
build: $(out)
%.out: %
echo $(value) > $< || [ rm $# -a true ]
init:
touch $(in)
clean:
rm -vf $(out)
How can I set a variable what is assigned only if used, but only computed once ?
Said differently, I would like build to take 2 seconds to complete and clean to be immediate.
I am not interested to a solution that involves conditionals in order to bypass the assignment of value if the target is not build.
An alternative solution would be this. Unfortunately in this case I need to check whether or not the shelve file needs to be regenerated.
value = $(cat shelve)
shelve:
sleep 2 && echo 42 > $# || [ rm $# -a true ]
in = a b c d
out = $(addsuffix .out,$(in))
build: $(out)
%.out: %
echo $(value) > $< || [ rm $# -a true ]
init:
touch $(in)
clean:
rm -vf $(out)
Here's a trick you can play:
value = $(eval value := $(shell cat shelve))$(value)
How this works: value is first assigned using recursive assignment so the value on the RHS is not expanded.
The first time value is expanded the make parser will first run the $(eval ...) which starts up a "new parser" for makefiles. In that parser, the content value := $(cat shelve) is evaluated. Here, value is a simple variable assignment so the RHS is expanded immediately and the $(shell ...) is run and assigned to value.
Remember make doesn't really have a concept of variable scope, so this value is just the same global value variable that we are setting in the outer parser.
Then the eval completes and expands to the empty string, and make continues parsing things. Here it finds the value $(value) and expands that... value now has the result from the eval, not the eval text itself, so that's what will be expanded.
Maybe this will help:
value = $(eval value := $(shell cat shelve))$(value)
Here value contains the string $(eval value := $(shell cat shelve))$(value)
Now you expand it:
%.out: %
echo $(value) > $< ...
Make starts to expand this recipe. It gets to $(value) and sees it needs to expand the variable value: since it's recursive it expands the value:
$(eval value := $(shell cat shelve))$(value)
First it expands the eval, which parses this:
value := $(shell cat shelve)
That sets the value variable as a simply-expanded variable, so the RHS is expanded immediately. Say the results of cat shelve are "foo", so value is now set to foo (and it's marked simply expanded).
That's the end of the eval, so then make starts the next part which is $(value), so it looks up the variable value and discovers it's a simply-expanded variable with the value foo.
One solution would be to turn that value into a regular file target that gets updated only when its prerequisites change. If you insist on rebuilding that target for every build, mark it as phony.
When clean target does not depend on that file, then it won't be rebuilt when you invoke make clean.
In
%.out: %
echo $(value) > $< || [ rm $# -a true ]
echo $(value) > $< updates the prerequisite, whereas make expects it to update the target only. Updating a prerequisite must be done by a separate rule with that prerequisite being the target.
You can make the assignment depend on the target name in $(MAKECMDGOALS):
ifneq ($(MAKECMDGOALS),clean)
value := $(shell sleep 2 && echo 42)
endif
See also the docs for details.

gnu make: list the values of all variables (or "macros") in a particular run

How can I list the current value of all variables (also called macros) in a Makefile when running make?
E.g. if this is in the Makefile:
CUR-DIR := $(shell /bin/pwd)
LOG-DIR := $(CUR-DIR)/make-logs
Then I would like it to tell me:
CUR-DIR = /home/johv/src/test
LOG-DIR = /home/johv/src/test/make-logs
GNU make provides .VARIABLES
which holds all global variables' names.
However, this includes built-in variables(like MAKEFLAGS).
If you have to exclude built-in variables, some filtering like the following
might be needed.
The following makefile prints user-defined variables(CUR-DIR, LOG-DIR)
using info:
# Place this line at the top of your Makefile
VARS_OLD := $(.VARIABLES)
# Define your variables
CUR-DIR := $(shell pwd)
LOG-DIR := $(CUR-DIR)/make-logs
# Put this at the point where you want to see the variable values
$(foreach v, \
$(filter-out $(VARS_OLD) VARS_OLD,$(.VARIABLES)), \
$(info $(v) = $($(v))))
Thanks to #Ise Wisteria, condensed down, this shows all variables, useful for large projects with multiple makefiles (Buildroot).
$(foreach v, $(.VARIABLES), $(info $(v) = $($(v))))
output: BR2_GCC_TARGET_TUNE = "cortex-a8" ...
If you get an error like: insufficient number of arguments (1) to function 'addprefix' this project had some broken variables... I trimmed the list of variables to show, only with a prefix BR2_
$(foreach v, $(filter BR2_%,$(.VARIABLES)), $(info $(v) = $($(v))))
I ended up doing it like this:
gmake -pn | grep -A1 "^# makefile"| grep -v "^#\|^--" | sort | uniq > makevars.txt
which gives:
CUR-DIR := /home/johv/src/test
LOG-DIR := /home/johv/src/test/make-logs
MAKEFILE_LIST := Makefile
MAKEFLAGS = pn
SHELL = /bin/sh
VARS_OLD := [...]
gmake -pn is really verbose and looks kinda like this:
# environment
GNOME2_PATH = /usr/local:/opt/gnome:/usr:/usr/local:/opt/gnome:/usr
# automatic
#F = $(notdir $#)
# makefile
SHELL = /bin/sh
# default
RM = rm -f
It's also doable without saving all the .VARIABLES and filtering them out.
Moreover, if one of the original .VARIABLES was modified in your makefile, the two most voted answers won't catch it.
Check out $(origin) function. This target filters out and prints all the variables that were defined in a makefile:
print_file_vars:
$(foreach v, $(.VARIABLES), $(if $(filter file,$(origin $(v))), $(info $(v)=$($(v)))))
I get only a few excess variables this way: CURDIR SHELL MAKEFILE_LIST .DEFAULT_GOAL MAKEFLAGS.
One can replace file with environment or command line to print the respective kinds of variables.
There are a lot of good answers here, but you're going to have problems using $($(v)) if some of your variables are of the recursive flavor. This is why you should use $(value $(v)).
This variation cleans this up a little bit, sorts variables by name and makes the output a bit more readable.
dump:
$(foreach v, \
$(shell echo "$(filter-out .VARIABLES,$(.VARIABLES))" | tr ' ' '\n' | sort), \
$(info $(shell printf "%-20s" "$(v)")= $(value $(v))) \
)
Thanks to #kevinf for the great idea. I would suggest a minor change to prevent .VARIABLE itself from printing out in the variable list:
$(foreach v, $(filter-out .VARIABLES,$(.VARIABLES)), $(info $(v) = $($(v))))
Thanks to #kevinf for the foreach solution -- if one wants to export this list as a somewhat machine-readable file, one will have a hard time with uneven quotes or newlines when using echo or printf, since Make isn't able to quote the data correctly -- one needs to use the $(file ...) function to write the data to avoid sh/bash complaining about invalid syntax. For example, use this in your rule -- it prints variable name, definition and expanded value:
$(file > $(MAKEFILE_ENV_FILE),)
$(foreach v, $(.VARIABLES), \
$(file >> $(MAKEFILE_ENV_FILE),$(v)) \
$(file >> $(MAKEFILE_ENV_FILE), := $(value $(v))) \
$(file >> $(MAKEFILE_ENV_FILE), == $($(v))) \
$(file >> $(MAKEFILE_ENV_FILE),) \
)
(This will still not allow to always distinguish malicious variables with double newlines from two variables, for this one now add a sufficiently unique separator infront of each Makefile-generated newline just after each comma inside $(file >> NAME,TEXT))
Set MAKEFILE_ENV_FILE to some filename, e.g.:
MAKEFILE_ENV_FILE := $(abspath $(lastword $(MAKEFILE_LIST))).env

Resources