Is there a way to make use of exported variables in shell function without the need for sub-make?
Take the following example.
FOO := BAR
.EXPORT_ALL_VARIABLES:
.PHONY: buzz
buzz:
$(info buzz)
$(error finish)
.PHONY: fizz
fizz: $(if $(shell echo $$FOO),buzz,)
$(info fizz)
$(MAKE) fizz
If I run the fizz target like so I get the following output.
$ make fizz
fizz
make fizz
make[1]: Entering directory '/home/jshbrntt/test'
buzz
Makefile:8: *** finish. Stop.
make[1]: Leaving directory '/home/jshbrntt/test'
make: *** [Makefile:13: fizz] Error 2
As you can see only the second run of make fizz had the shell function expanded and cause the buzz target to also run.
Is there a way to make use of exported variables in shell function without the need for sub-make?
No.
Remember always that GNU make functions such as $(shell) are evaluated while the makefile is being parsed, not when make runs recipes (refer to section 3.7 of the manual), regardless of where in the makefile the $(shell) invocation appears. make determines which variables are exported based on its own environment and the combination of all rules and export / unexport directives in the makefile, potentially including rules and / or directives generated via $(shell) and other functions. In this way it ensures that it is consistent about the environment used to execute recipes.
Although it is conceivable that make would expose expose intermediate forms of its export list to the $(shell) function (and its documentation doesn't clearly specify whether it does so), as a practical matter it would be surprising for it to do so, and your experiment shows that it does not do. And although the manual does not explicitly speak directly to the question, it should be noted that its documentation of export, etc. is in a section entitled Communicating Variables to a Sub-make.
Personally, I recommend avoiding $(shell) (and $(wildcard)) altogether. If you do use $(shell) then I recommend reserving such use to outside recipes. Inside recipes, use shell code directly. This is clearer, certainly in terms of makefile semantics (what is evaluated when), but often in terms of the actual code, too. For your particular example, that might look like so:
FOO := BAR
.EXPORT_ALL_VARIABLES:
.PHONY: fizz
fizz:
#echo fizz
#if test -n "$$FOO"; then echo FOO found; else echo FOO missing; make fizz; fi
In all current versions of GNU make, exported variables are not sent to the shell function. There are some very nasty recursive behaviors that can happen (what if you write export BAR = $(shell echo $$FOO) ???)
In the next release of GNU make, make variables will be exported to the shell function.
However, there's never any good reason to use shell in a recipe. The recipe is running in a shell, so you can just write the commands that you want directly. So if your example is accurate in that you want to use this facility in a recipe, just take out the shell invocation:
.PHONY: fizz
fizz:
#echo fizz
#test -n $$FOO && echo FOO found || echo FOO missing
Related
Say in the working directory, I have:
$ find . | grep testfile
./testfile1
This is my Makefile:
list_files:
#echo "Show Files..."
#echo $(shell find . | grep testfile)
#touch testfile2
#echo $(shell find . | grep testfile)
#rm testfile2
With make, I got this:
$ make list_files
Show Files...
./testfile1
./testfile1
Why this happened? I expected it to be something like this:
Show Files...
./testfile1
./testfile1 ./testfile2
Then my question is:
Why all the variables/function in recipes inside a rule, are expanded likely simultaneously after target is invoked?
I have found an explanation from this answer that is pretty close to the truth:
The reason your attempt doesn't work is that make will evaluate all lines of the recipe before it starts the first line.
But there is no references provided there, I just cannot convinced myself of this working mechanism of GNU Make.
Could anyone give some clues? Thanks!
Why this happened?
Because, with one caveat that does not apply to your case, make functions such as $(shell ...) are evaluated when the makefile is parsed, not during the execution of recipes.
Why all the variables/function in recipes inside a rule, are expanded likely simultaneously after target is invoked?
They're not. They are expanded before the target's recipe runs. In fact, before make even determines whether the recipe should be run.
But there is no references provided
This is covered in the manual. See in particular section 8.14, The shell function:
The commands run by calls to the shell function are run when the function calls are expanded (see How make Reads a Makefile).
... which refers to section 3.7, How make Reads a Makefile, in particular:
GNU make does its work in two distinct phases. During the first phase it reads all the makefiles, included makefiles, etc. and internalizes all the variables and their values and implicit and explicit rules, and builds a dependency graph of all the targets and their prerequisites. During the second phase, make uses this internalized data to determine which targets need to be updated and run the recipes necessary to update them.
It also relies on section 8.1, Function Call Syntax:
A function call resembles a variable reference. It can appear anywhere a variable reference can appear, and it is expanded using the same rules as variable references.
"The same rules" of course includes the rules for when expansions are performed.
Your recipes should be written in the language of the target shell, usually /bin/sh. All make functions and variable references in each recipe will be expanded before any recipe runs, so their expansions cannot reflect the results of running any recipe during the current make run. It's particularly peculiar to try to use the $(shell ...) function to do that, because you can just use shell code directly in a recipe.
As explained in the comments, all $(shell ...) and other functions are executed (expanded) before executing any lines from the recipe. But you can delay the expansion:
list_files:
#echo "Show Files..."
#echo $$(shell find . | grep testfile)
#touch testfile2
#echo $$(shell find . | grep testfile)
#rm testfile2
This yields the expected output. Note the double $$. Make will still expand all variables/functions first before executing the recipe, and remove one of the dollar signs. The expressions will be expanded again, when executing the recipe lines.
Of course, you're better off without the $(shell) in this example. However, it's not inherently a bad idea. Sometimes you need to execute shell commands before expanding other variables, and then $(shell) is the trivial solution.
When I run
export PATH := mypath
$(error $(shell echo "$${PATH}"))
it seems my PATH isn't changed on the call to shell.
Why is this and how do I actually change the PATH for shell calls?
Is this with GNU make? There is a long-standing GNU make feature request to honor exported variables with $(shell …). This is not specific to PATH at all, it affects (or does not affect) all export variables.
According to the GNU make sources, this is tricky to implement:
/* Using a target environment for 'shell' loses in cases like:
export var = $(shell echo foobie)
bad := $(var)
because target_environment hits a loop trying to expand $(var) to put it
in the environment. This is even more confusing when 'var' was not
explicitly exported, but just appeared in the calling environment.
See Savannah bug #10593.
envp = target_environment (NULL);
*/
The solution is simple: never ever use $(shell) or export.
Environment variables should be part of the recipe that needs them.
For $(shell) invocations that are supposed to fill a makefile variable you can use instead.
it also has the advantage to be more flexible, because you can fill more than one variable with one recipe
you can also define proper dependencies, whereas $(shell) is always executed, either when the makefile is parsed or the recursively expanded variable gets expanded.
you get build errors and recipes are logged, whereas $(shell) can make the DevOp engineers life a living h...
PATH := mypath
Makefile.variables:
#PATH=$(PATH) echo "This my path '$${PATH}'"
echo >$# "MY_DYNAMIC_CONTENT := abcd"
include Makefile.variables
$(info MY_DYNAMIC_CONTENT '$(MY_DYNAMIC_CONTENT)')
Example run:
$ make
MY_DYNAMIC_CONTENT ''
This my path 'mypath'
echo >Makefile.variables "MY_DYNAMIC_CONTENT := abcd"
MY_DYNAMIC_CONTENT 'abcd'
make: 'Makefile.variables' is up to date.
I was looking at another stack overflow question, and they were using, what looks to be undocumented behavior for makefiles... If I have the following Makefile:
X=X
all:
#echo $#: $(X)
$(eval X=Y)
#echo $# part2: $(X)
all2:
#echo $#: $(X)
Then I run:
~> make all2
all2: X
~> make all all2
running all
all: X
all part2: Y
all2: Y
I would have expected the $(eval X=Y) to expand at Makefile parse time, and set the shell variable X to be Y for that line of the recipe (i.e. do nothing). Instead, it seems to be evaluated when the all recipe is run, plus, it seems to set the make variable. I've looked through the make man page, and for online manuals, but I can't find anything that describes this behavior (I'm using GNU Make 4.0). Can someone point me to some documentation describing this, or explain what's going on?
I'm not sure why you would expect either of the things you mention to be true. Maybe you're thinking that this is using the shell eval, somehow? That's not what it's doing, it's using the GNU make eval function, which is discussed here.
Just like any other variable or function, eval appearing in a recipe is not expanded until (and unless) the recipe is invoked because the target is out of date. See How make Reads a Makefile for full details on when variables and functions are expanded.
Second, all make variable assignments set make variables, and only if you export them will they be sent to the shell.
I have a rule something, that works on the variable VAR. I also have another rule something-all, that needs to run something, with VAR set to each value in vars.
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach VAR,$(vars),something)
This doesn't quite work, I get
noob#work:~/Desktop$ make something-all
something something
make: something: No such file or directory
make: *** [something-all] Error 1
It should probably print hello\nworld.
I used to do this with wildcard rules by retrieving VAR from %, but got the feeling that was the wrong way to do it. This looked like this:
vars = hello world
all: $(foreach VAR,$(vars),something-$(VAR))
something-%:
echo $*
The below should fix your problem
Using foreach (Tried on GNU Make 3.80 on sparc-solaris 2.8 and windows)
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach i, $(vars), $(MAKE) something VAR=$i || exit 1;)
Using shell for-loop (Tried on GNU Make 3.80 and cc make on sparc-solaris 2.8)
vars = hello world
something:
echo $(VAR)
something-all:
for i in $(vars); do $(MAKE) something VAR=$$i || exit 1; done
TL;DR: If you want to program make, drop GNU Make in favor of BSD Make.
This is a personal recommendation. While BSD Make seems more limited than GNU Make, as it offers less programming facilities, it is much easier to program and has a few unique killer features. This is why I propose a solution with GNU Make and another solution for BSD Make:
Doing it in GNU Make
Using GNU Make, you can write a macro to define a target. The canonical way to define a sequence in a Makefile is to add the steps of the sequence as dependencies to a target, as reflected by the snippet below:
vars= hello world
define something_t =
something: something-$(1)
something-$(1):
#echo $(1)
endef
$(foreach _,$(vars),$(eval $(call something_t,$_)))
It is recommended to use this organisation (rather than defining just one target), because you can work on it to make the task easily resumable if you interrupt the sequence. A Makefile describes a job whose advancement is entirely described by the state of the file system. A task is then easily resumable, if each step is associated to a file, usually a compilation object but sometimes also an empty file which is touch'ed to indicate that important checkpoints have been passed.
Using an auxiliary macro is a flexible solution that can be adapted to more complicated tasks than just echoing a name. Note that this does work with newest versions of GNU Make (4.1). On GNU Make 3.81, you should remove the equal sign from the macro definition.
Adapting your example for BSD Make
If this is an option for you, I recommand dropping the use of GNU Make and replace it by BSD Make, which is way easier to program: it has a short and to the point documentation, while the documentation of GNU Make is very verbose and somewhat unclear, BSD Make has industrial-strength examples of complex rulesets (FreeBSD Build system or BSD Owl), and it has a simple and predictable macro language.
vars= hello world
something:
.for _var in ${vars}
echo ${_var}
.endfor
This can evolve to support more complicated tasks, just by replacing the echo by the adapted commands, or using intermediary steps.
Allow the user to override some tasks, also in BSD Make
In this slightly more advanced variation, we allow the user to override our own recipes for building targets something-hello and something-world.
For each item in our list, a target something-* is created it if it does not already exist, and added to the dependencies of something. The whole operation of defining these targets only happens if something has been left undefined. Therefore, users of these macros can:
Override the recipes for something-hello and something-world
Override the full procedure bound to something.
Implementing such customisation possibilities is mandatory if we want to write useful, reusable, macros for Make. Unluckily, customisation of this sort is nearly impossible in GNU Make.
vars = hello world
.if!target(depend)
.for _var in ${vars}
.if!target(something-${_var})
something-${_var}:
echo ${_var}
.endif
something: something-${_var}
.endfor
.endif
Here's one way to do it:
VARS := hello world
THINGS := $(addprefix something-, $(VARS))
allthings: $(THINGS)
something-%:
echo $*
It should be no surprise that
vars := hello world
something-all:
$(foreach VAR,$(vars),something)
tries to run something something. That's exactly what the foreach expands to, since you don't reference VAR in the third expression.
All you need to do is reference VAR and use a command such as echo:
vars := hello world
something-all:
$(foreach VAR,$(vars),echo $(VAR);)
$ make
echo hello; echo world;
hello
world
Note how chaining the commands with a semicolon avoids forking several shells or -- GASP! -- recursive make invocations. It doesn't get more performant than that.
Alternatively, if your command accepts several somethings as arguments,
vars := hello world
something-all:
echo $(foreach VAR,$(vars),$(VAR))
$ make
echo hello world
hello world
But that is equivalent to the super simple echo $(vars). So it might pay off to think outside the box trying to change your requirements to make this simple solution work.
Given a Makefile:
ifeq "$(MAKELEVEL)" "0"
0 ::
#$(MAKE)
else
1 ::
#echo 'foo is: "$(foo)"'
endif
And executing, we get:
$ make foo='$#'
make[1]: Entering directory '/home/myname'
foo is: "1"
make[1]: Leaving directory '/home/myname'
$ make foo='$#' --environment-overrides
make[1]: Entering directory '/home/myname'
foo is: "0"
make[1]: Leaving directory '/home/myname'
So we have here a recursive variable foo with the value: $#, which - of course - expands to the name of the target. Now, we have two options here:
Either, Make expands first the variable, and then export the variable to a sub-make.
With this "logic", when Make runs the first makefile (MAKELEVEL = 0), it will build the target 0, Hence: expand the variable foo(and its value:$#) to 0, and then export - this already expanded value - to the sub-make.
This result, with the sub-make running its makefile, with a variable foo that has the simple value: 0.
This is in-fact the case, when we run make --environment-overrides, as you can see in the second run of the makefile, in the example above.
Another "logic" is, for Make to pass the value "verbatim". That means, with no expansion!
Hence all recursive variables will be still intact, when passed to the second Make.
For this logic, only the sub-make is allowed to expand its variables recursively, hence: any recursive-expansion will be done in the context of the sub-make.
In our example above, that we had foo with its value $#, if we are to follow this logic, Make will pass the value $# "verbatim", with no expansion at all, so the sub-make will effectively see a value $# for its foo variable, hence: when expanding in the context of its target, that happens to be 1, it will recursively expand foo(and its value: $#) to 1.
Actually, this is the "normal" behaviour that is evident in the first run of makefile, as evident in the example above.
So, for a lack of clear methodology, we are left to conclude, that this behaviour of either expand and then export or export and then expand is inconsistent.
That is, sometimes Make will choose the first method, where sometimes it will chose the second.
In our example, it was a command-line option (--environment--overrides), that acted as a deciding factor, as to what method Make has to choose.
But, can we really justify, that these - seemingly - unrelated features (i.e export/recursive vs. environment-overrides), will end up to have such a dramatic effect on each-other?
(Versions note: 4.0 and up).
make does export foo in expanded form, as can be seen here:
target:
#echo "'$$foo'"
Output:
$make foo='$#'
'target'
However, to pass its argument to sub-makes, it does not use the environment directly -- instead it stuffs everything into the variable MAKEFLAGS. And there it passes foo unexpanded:
target:
#echo "'$$MAKEFLAGS'"
Output:
$make foo='$#'
' -- foo=$$#'
To be clear: the sub-make does import foo from the environment, but that is then overridden by the setting from MAKEFLAGS.
When you specify --environment--overrides, that means that environment variables take precedence over settings in the makefile. The GNU make documentation does not explicitly specify what happens to variables passed in via MAKEFLAGS in the presence of --environment--overrides, but apparently they are overridden too.