From the docs:
...You can modify .SHELLFLAGS to add the -e option to the shell which will
cause any failure anywhere in the command line to cause the shell to
fail...
So, given the following makefile:
.SHELLFLAGS = -e -c
define cmd
echo true
true
endef
all::
$(shell $(cmd))
Running, I get:
truetrue
/bin/sh: 1: truetrue: not found
makefile:10: recipe for target 'all' failed
make: *** [all] Error 127
Did you see that (in the 1st line of output)? truetrue? Where did it come from?
Let's compare it with the following identical makefile, where -e was not added to .SHELLFLAGS.
The makefile, would be:
# Do you see the difference, from the previous Makefile?
# Here, the value is '-c' alone. No '-e' flag is present here!
.SHELLFLAGS = -c
define cmd
echo true
true
endef
all::
$(shell $(cmd))
Running, I get:
true true
Much better!
So, we have here 2 versions of makefile, both run identical commands, where the first version also added the -e flag to .SHELLFLAGS.
The results however - for the respective runs - were not consistent! We had:
Expands to the trivial command true true, which is analogous to the trivial command : true (i.e "ignore arguments..."), and therefore an exit-status: 0.
This was for the "simple" run, without the change in .SHELLFLAGS. (version 2).
As for the first run, Make went ahead, and condensed the command to truetrue (really?), hence: a meaningless command and a fatal error from the shell.
In summary: It seems that adding -e to .SHELLFLAGS is not that innocent as the documentation suggested above.
In fact, Make, then (when .SHELLFLAGS is modified to include -e), and only then - Make takes the liberty to do some more (unexpected?) modification to the command, resulting - for example - here with the command truetrue for a true true command.
Really?
(Versions-note: All versions supporting .SHELLFLAGS, which is 3.82 and up).
I’ve never understood using $(shell ...) in a recipe. There may be one or two
situations where it’s useful, but I see it constantly and I’ve yet to see one.
However, you’ll note that the documentation says that the value for
.SHELLFLAGS when you want to use the -e flag should be -ec, not -e -c.
Some versions of the Bourne shell do not accept multiple flags on the command
line: they must be a single set of flags. If you use -ec then your example
works.
The difference in behavior you’re seeing is in the treatment of newlines in
recipes: in some situations they’re removed along with the initial TAB, in other
situations they’re converted to a single space. I don’t know why adding a second
option to .SHELLFLAGS causes this but I agree with tripleee: you should
file a bug.
Related
Say in the working directory, I have:
$ find . | grep testfile
./testfile1
This is my Makefile:
list_files:
#echo "Show Files..."
#echo $(shell find . | grep testfile)
#touch testfile2
#echo $(shell find . | grep testfile)
#rm testfile2
With make, I got this:
$ make list_files
Show Files...
./testfile1
./testfile1
Why this happened? I expected it to be something like this:
Show Files...
./testfile1
./testfile1 ./testfile2
Then my question is:
Why all the variables/function in recipes inside a rule, are expanded likely simultaneously after target is invoked?
I have found an explanation from this answer that is pretty close to the truth:
The reason your attempt doesn't work is that make will evaluate all lines of the recipe before it starts the first line.
But there is no references provided there, I just cannot convinced myself of this working mechanism of GNU Make.
Could anyone give some clues? Thanks!
Why this happened?
Because, with one caveat that does not apply to your case, make functions such as $(shell ...) are evaluated when the makefile is parsed, not during the execution of recipes.
Why all the variables/function in recipes inside a rule, are expanded likely simultaneously after target is invoked?
They're not. They are expanded before the target's recipe runs. In fact, before make even determines whether the recipe should be run.
But there is no references provided
This is covered in the manual. See in particular section 8.14, The shell function:
The commands run by calls to the shell function are run when the function calls are expanded (see How make Reads a Makefile).
... which refers to section 3.7, How make Reads a Makefile, in particular:
GNU make does its work in two distinct phases. During the first phase it reads all the makefiles, included makefiles, etc. and internalizes all the variables and their values and implicit and explicit rules, and builds a dependency graph of all the targets and their prerequisites. During the second phase, make uses this internalized data to determine which targets need to be updated and run the recipes necessary to update them.
It also relies on section 8.1, Function Call Syntax:
A function call resembles a variable reference. It can appear anywhere a variable reference can appear, and it is expanded using the same rules as variable references.
"The same rules" of course includes the rules for when expansions are performed.
Your recipes should be written in the language of the target shell, usually /bin/sh. All make functions and variable references in each recipe will be expanded before any recipe runs, so their expansions cannot reflect the results of running any recipe during the current make run. It's particularly peculiar to try to use the $(shell ...) function to do that, because you can just use shell code directly in a recipe.
As explained in the comments, all $(shell ...) and other functions are executed (expanded) before executing any lines from the recipe. But you can delay the expansion:
list_files:
#echo "Show Files..."
#echo $$(shell find . | grep testfile)
#touch testfile2
#echo $$(shell find . | grep testfile)
#rm testfile2
This yields the expected output. Note the double $$. Make will still expand all variables/functions first before executing the recipe, and remove one of the dollar signs. The expressions will be expanded again, when executing the recipe lines.
Of course, you're better off without the $(shell) in this example. However, it's not inherently a bad idea. Sometimes you need to execute shell commands before expanding other variables, and then $(shell) is the trivial solution.
I've read in linux Makefile:
$(filter-out _all sub-make $(CURDIR)/Makefile, $(MAKECMDGOALS)) _all: sub-make
$(Q)#:
What is the meaning of $(Q)#:?
I'm trying to google it, but google always crappy if the search using some weird character. So in the end i can't found any Manual about it.
After looking in the code. Q is defined somewhere after those line. Since makefile have peculiar concept of variable (which is expandable), it can be implement in anywhere. Q is used to whether show message or not (Q maybe for Quiet).
ifeq ($(KBUILD_VERBOSE),1)
quiet =
Q =
else
quiet=quiet_
Q = #
endif
And for the last #: this means do-nothing-output-nothing.
So the conclusion $(Q)#: simply do-nothing-output-nothing.
To reinforce and expand on what nafsaka found:
Sometimes Makefiles are written like this:
target:
rm -rf $(DIRECTORY)
$(Q)$(MAKE) all
And Q will be defined as # or nothing for example:
V ?= 0
ifeq ($(V), 0)
Q = #
else
Q =
endif
If a target action is preceded by # then make won't display it when run. Here's the GNU make documentation on that subject: Recipe Echoing
In this case you need to define V=1 before running make to see commands as they're run (This is very common).
Another wrinkle: Look for "include file.mk" statements in your Makefile, which is where V and Q were defined in my case. Here's the GNU make documentation on include: Including Other Makefiles
From the Make manual:
5.2 Recipe Echoing
Normally make prints each line of the recipe before it is executed. We call this echoing because it gives the appearance that you are typing the lines yourself.
When a line starts with ‘#’, the echoing of that line is suppressed. The ‘#’ is discarded before the line is passed to the shell. Typically you would use this for a command whose only effect is to print something, such as an echo command to indicate progress through the makefile:
#echo About to make distribution files
When make is given the flag ‘-n’ or ‘--just-print’ it only echoes most recipes, without executing them. See Summary of Options. In this case even the recipe lines starting with ‘#’ are printed. This flag is useful for finding out which recipes make thinks are necessary without actually doing them.
The ‘-s’ or ‘--silent’ flag to make prevents all echoing, as if all recipes started with ‘#’. A rule in the makefile for the special target .SILENT without prerequisites has the same effect (see Special Built-in Target Names).
I am unable to prevent make from communicating any variables to a submake. I've read the manual and I've followed their advice (resetting MAKEOVERRIDES and MAKEFLAGS) but it's still not working has I think it should.
Consider the following prototype Makefile:
${warning $(MAKEOVERRIDES)}
${warning $(MAKEFLAGS)}
${warning $(VAR)}
none:
$(MAKE) -f Makefile MAKEOVERRIDES= MAKEFLAGS= all
all:
echo done!
If I make VAR=10 none, I get the following:
Makefile:2: VAR=10
Makefile:3:
Makefile:4: 10
make -f Makefile MAKEOVERRIDES= MAKEFLAGS= all
make[1]: Entering directory `/home/adriano/sandbox/makes'
Makefile:2:
Makefile:3:
Makefile:4: 10
echo done!
done!
make[1]: Leaving directory `/home/adriano/sandbox/makes'
Meaning that make is communication VAR to the submake. Is this the correct behaviour?
I've tried unexport VAR and bash -c make ... without any luck.
EDIT: I've modified none's recipe to: bash -c "echo $$MAKEOVERRIDES $$MAKEFLAGS $$VAR" ; make ...
This way I found out that VAR is actually being passed through the environment that make creates for the commands to be executed and not through the other variables (the other variables are also passed this way to make).
I think my question now is: how can I create a fresh shell/environment to run my sub make?
EDIT: Someone asked why am I trying to this; I'll try to answer to that here.
I have a "module" which uses a variable named CONFIG. In order to build this module I need to build another partially unrelated "module" which also uses CONFIG, but with a different value. The problem is that when I try to build the "sub-module" CONFIG contains the value of the "super-module." I could specify CONFIG when making the "sub-module" however both modules use many variables with the same name and trying to specify them all would make the modules tightly coupled which is something I cannot afford.
How can this be so difficult...
This is wrong:
none:
$(MAKE) -f Makefile MAKEOVERRIDES= MAKEFLAGS= all
These variables (MAKEOVERRIDES and MAKEFLAGS) are set in the environment by the parent make to be passed down to the sub-makes. Setting overrides on these values inside the recipe won't help, because make has to set the environment for the recipe before it actually starts the commands in the recipe (of course).
You have to override/remove these values in the parent makefile, so that those changes are seen by the parent make before it constructs the sub-make's environment:
MAKEOVERRIDES =
none:
$(MAKE) -f Makefile all
There's no perfect way to do this. However, you can play a trick that will work most of the time:
unexport $(shell echo '$(MAKEOVERRIDES)' | sed 's/=[^ ]*//g')
MAKEOVERRIDES =
The first line tries to unexport all the variables in MAKEOVERRIDES and the second line resets MAKEOVERRIDES. There are a few issues with this. One is that if MAKEOVERRIDES is empty, it will use "unexport" by itself which unexports everything. That can be easily worked around by sticking some bogus variable before the shell function. The other is that if any variable's value contains whitespace, the expansion will consider it a variable to be unexported. That's probably OK, but it's odd.
I can't think of any better way to do it.
You don't really say why you want to do this. Have you considered doing something different, such as running the commands where you want to have a "vanilla" environment using env; for example if you want to run a command with a limited and specific set of env vars, you can run:
test:
env -i PATH='$(PATH)' LANG='$(LANG)' runMyCommand --with --my arguments
Unfortunately some versions of env use - instead of -i; check your man page.
Alternatively, you can try to start a login shell which will re-read the user's shell setup environment from scratch:
test:
/bin/sh -lc 'runMyCommand --with --my arguments'
EDIT: It's difficult because what you're asking to do (restrict the environment of the sub-make) is tricky.
Luckily based on your description, it doesn't seem necessary. Make has a hierarchy of importance for finding variable values. The command line is the highest level (well, there's override but we'll ignore that). After that comes variables set in the makefile itself. And last and lowest comes variables imported from the environment (well, default variables are even lower but we'll ignore that too).
So if your goal is to allow the variables in the sub-makes to not be affected by command line variables given to the upper-level makes, then all this rigmarole of getting the variables out of the environment is not necessary. Variables set in the sub-makefiles will take precedence over the values in the environment. So all you have to do is get rid of the variables set on the command line, which I've already shown how to do above, by setting MAKEOVERRIDES.
Is it possible to include Makefiles dynamically? For example depending on some environment variable? I have the following Makefiles:
makefile
app1.1.mak
app1.2.mak
And there is an environment variable APP_VER which could be set to 1.1.0.1, 1.1.0.2, 1.2.0.1, 1.2.0.2.
But there will be only two different makefiles for 1.1 and 1.2 lines.
I have tried to write the following Makefile:
MAK_VER=$$(echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/')
include makefile$(MAK_VER).mak
all: PROD
echo MAK_VER=$(MAK_VER)
But it does not work:
$ make all
"makefile$(echo", line 0: make: Cannot open makefile$(echo
make: Fatal errors encountered -- cannot continue.
UPDATE:
As far as I understand make includes files before it calculates macros.
That's why it tries to execute the following statement
include makefile.mak
instead of
include makefile1.1.mak
You have two problems: your method of obtaining the version is too complicated, and your include line has a flaw. Try this:
include app$(APP_VER).mak
If APP_VER is an environmental variable, then this will work. If you also want to include the makefile called makefile (that is, if makefile is not the one we're writing), then try this:
include makefile app$(APP_VER).mak
Please note that this is considered a bad idea. If the makefile depends on environmental variables, it will work for some users and not others, which is considered bad behavior.
EDIT:
This should do it:
MAK_VER := $(subst ., ,$(APP_VER))
MAK_VER := $(word 1, $(MAK_VER)).$(word 2, $(MAK_VER))
include makefile app$(MAK_VER).mak
Try this:
MAK_VER=$(shell echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/')
MAK_FILE=makefile$(MAK_VER).mak
include $(MAK_FILE)
all:
echo $(MAK_VER)
echo $(MAK_FILE)
Modifying the outline solution
Have four makefiles:
makefile
app1.1.mak
app1.2.mak
appdummy.mak
The app.dummy.mak makefile can be empty - a symlink to /dev/null if you like. Both app.1.1.mak and app.1.2.mak are unchanged from their current content.
The main makefile changes a little:
MAK_VER = dummy
include makefile$(MAK_VER).mak
dummy:
${MAKE} MAK_VER=$$(echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/') all
all: PROD
...as now...
If you type make, it will read the (empty) dummy makefile, and then try to build the dummy target because it appears first. To build the dummy target, it will run make again, with APP_VER=1.1 or APP_VER=1.2 on the command line:
make APP_VER=1.1 all
Macros set on the command line cannot be changed within the makefile, so this overrides the line in the makefile. The second invocation of make, therefore, will read the correct version-specific makefile, and then build all.
This technique has limitations, most noticeably that it is fiddly to arrange for each and every target to be treated like this. There are ways around it, but usually not worth it.
Project organization
More seriously, I think you need to review what you're doing altogether. You are, presumably, using a version control system (VCS) to manage the source code. Also, presumably, there are some (significant) differences between the version 1.1 and 1.2 source code. So, to be able to do a build for version 1.1, you have to switch from the version 1.1 maintenance branch to the version 1.2 development branch, or something along those lines. So, why isn't the makefile just versioned for 1.1 or 1.2? If you switch between versions, you need to clean out all the derived files (object files, libraries, executables, etc) that may have been built with the wrong source. You have to change the source code over. So why not change the makefile too?
A build script to invoke make
I also observe that since you have the environment variable APP_VER driving your process, that you can finesse the problem by requiring a standardized 'make invoker' that sorts out the APP_VER value and invokes make correctly. Imagine that the script is called build:
#!/bin/sh
: ${APP_VER:=1.2.0.1} # Latest version is default
case $APP_VER in
[0-9].[0-9].*)
MAK_VER=`echo $APP_VER | sed -e 's/^\(...\).*/\1/'`
;;
*) echo "`basename $0 .sh`: APP_VER ($APP_VER) should start with two digits followed by dots" 1>&2;
exit 1;;
esac
exec make MAK_VER=$MAK_VER "$#"
This script validates that APP_VER is set, giving an appropriate default if it is not. It then processes that value to derive the MAK_VER (or errors out if it is incorrect). You'd need to modify that test after you reach version 10, of course, since you are planning to be so successful that you will reach double-digit version numbers in due course.
Given the correct version information, you can now invoke your makefile with any command line arguments.
The makefile can be quite simple:
MAK_VER = dummy
include app$(MAK_VER).mak
all: PROD
...as now...
The appdummy.mak file now contains a rule:
error:
echo "You must invoke this makefile via the build script" 1>&2
exit 1
It simply points out the correct way to do the build.
Note that you can avoid the APP_VER environment variable if you keep the product version number under the VCS in a file, and the script then reads the version number from the file. And there could be all sorts of other work done by the script, ensuring that correct tools are installed, other environment variables are set, and so on.
I need to execute some make rules conditionally, only if the Python installed is greater than a certain version (say 2.5).
I thought I could do something like executing:
python -c 'import sys; print int(sys.version_info >= (2,5))'
and then using the output ('1' if ok, '0' otherwise) in a ifeq make statement.
In a simple bash shell script it's just:
MY_VAR=`python -c 'import sys; print int(sys.version_info >= (2,5))'`
but that doesn't work in a Makefile.
Any suggestions? I could use any other sensible workaround to achieve this.
Use the Make shell builtin like in MY_VAR=$(shell echo whatever)
me#Zack:~$make
MY_VAR IS whatever
me#Zack:~$ cat Makefile
MY_VAR := $(shell echo whatever)
all:
#echo MY_VAR IS $(MY_VAR)
Beware of recipes like this
target:
MY_ID=$(GENERATE_ID);
echo $MY_ID;
It does two things wrong. The first line in the recipe is executed in a separate shell instance from the second line. The variable is lost in the meantime. Second thing wrong is that the $ is not escaped.
target:
MY_ID=$(GENERATE_ID); \
echo $$MY_ID;
Both problems have been fixed and the variable is useable. The backslash combines both lines to run in one single shell, hence the setting of the variable and the reading of the variable afterwords, works.
I realize the original post said how to get the results of a shell command into a MAKE variable, and this answer shows how to get it into a shell variable. But other readers may benefit.
One final improvement, if the consumer expects an "environment variable" to be set, then you have to export it.
my_shell_script
echo $MY_ID
would need this in the makefile
target:
export MY_ID=$(GENERATE_ID); \
./my_shell_script;
Hope that helps someone. In general, one should avoid doing any real work outside of recipes, because if someone use the makefile with '--dry-run' option, to only SEE what it will do, it won't have any undesirable side effects. Every $(shell) call is evaluated at compile time and some real work could accidentally be done. Better to leave the real work, like generating ids, to the inside of the recipes when possible.
Wrapping the assignment in an eval is working for me.
# dependency on .PHONY prevents Make from
# thinking there's `nothing to be done`
set_opts: .PHONY
$(eval DOCKER_OPTS = -v $(shell mktemp -d -p /scratch):/output)
With GNU Make, you can use shell and eval to store, run, and assign output from arbitrary command line invocations. The difference between the example below and those which use := is the := assignment happens once (when it is encountered) and for all. Recursively expanded variables set with = are a bit more "lazy"; references to other variables remain until the variable itself is referenced, and the subsequent recursive expansion takes place each time the variable is referenced, which is desirable for making "consistent, callable, snippets". See the manual on setting variables for more info.
# Generate a random number.
# This is not run initially.
GENERATE_ID = $(shell od -vAn -N2 -tu2 < /dev/urandom)
# Generate a random number, and assign it to MY_ID
# This is not run initially.
SET_ID = $(eval MY_ID=$(GENERATE_ID))
# You can use .PHONY to tell make that we aren't building a target output file
.PHONY: mytarget
mytarget:
# This is empty when we begin
#echo $(MY_ID)
# This recursively expands SET_ID, which calls the shell command and sets MY_ID
$(SET_ID)
# This will now be a random number
#echo $(MY_ID)
# Recursively expand SET_ID again, which calls the shell command (again) and sets MY_ID (again)
$(SET_ID)
# This will now be a different random number
#echo $(MY_ID)
Here's a bit more complicated example with piping and variable assignment inside recipe:
getpodname:
# Getting pod name
#eval $$(minikube docker-env) ;\
$(eval PODNAME=$(shell sh -c "kubectl get pods | grep profile-posts-api | grep Running" | awk '{print $$1}'))
echo $(PODNAME)
I'm writing an answer to increase visibility to the actual syntax that solves the problem. Unfortunately, what someone might see as trivial can become a very significant headache to someone looking for a simple answer to a reasonable question.
Put the following into the file "Makefile".
MY_VAR := $(shell python -c 'import sys; print int(sys.version_info >= (2,5))')
all:
#echo MY_VAR IS $(MY_VAR)
The behavior you would like to see is the following (assuming you have recent python installed).
make
MY_VAR IS 1
If you copy and paste the above text into the Makefile, will you get this? Probably not. You will probably get an error like what is reported here:
makefile:4: *** missing separator. Stop
Why: Because although I personally used a genuine tab, Stack Overflow (attempting to be helpful) converts my tab into a number of spaces. You, frustrated internet citizen, now copy this, thinking that you now have the same text that I used. The make command, now reads the spaces and finds that the "all" command is incorrectly formatted. So copy the above text, paste it, and then convert the whitespace before "#echo" to a tab, and this example should, at last, hopefully, work for you.
In the below example, I have stored the Makefile folder path to LOCAL_PKG_DIR and then use LOCAL_PKG_DIR variable in targets.
Makefile:
LOCAL_PKG_DIR := $(shell eval pwd)
.PHONY: print
print:
#echo $(LOCAL_PKG_DIR)
Terminal output:
$ make print
/home/amrit/folder
From the make manual
The shell assignment operator ‘!=’ can be used to execute a shell script and set a >variable to its output. This operator first evaluates the right-hand side, then passes >that result to the shell for execution. If the result of the execution ends in a >newline, that one newline is removed; all other newlines are replaced by spaces. The >resulting string is then placed into the named recursively-expanded variable. For >example:
hash != printf '\043'
file_list != find . -name '*.c'
source