Say in the working directory, I have:
$ find . | grep testfile
./testfile1
This is my Makefile:
list_files:
#echo "Show Files..."
#echo $(shell find . | grep testfile)
#touch testfile2
#echo $(shell find . | grep testfile)
#rm testfile2
With make, I got this:
$ make list_files
Show Files...
./testfile1
./testfile1
Why this happened? I expected it to be something like this:
Show Files...
./testfile1
./testfile1 ./testfile2
Then my question is:
Why all the variables/function in recipes inside a rule, are expanded likely simultaneously after target is invoked?
I have found an explanation from this answer that is pretty close to the truth:
The reason your attempt doesn't work is that make will evaluate all lines of the recipe before it starts the first line.
But there is no references provided there, I just cannot convinced myself of this working mechanism of GNU Make.
Could anyone give some clues? Thanks!
Why this happened?
Because, with one caveat that does not apply to your case, make functions such as $(shell ...) are evaluated when the makefile is parsed, not during the execution of recipes.
Why all the variables/function in recipes inside a rule, are expanded likely simultaneously after target is invoked?
They're not. They are expanded before the target's recipe runs. In fact, before make even determines whether the recipe should be run.
But there is no references provided
This is covered in the manual. See in particular section 8.14, The shell function:
The commands run by calls to the shell function are run when the function calls are expanded (see How make Reads a Makefile).
... which refers to section 3.7, How make Reads a Makefile, in particular:
GNU make does its work in two distinct phases. During the first phase it reads all the makefiles, included makefiles, etc. and internalizes all the variables and their values and implicit and explicit rules, and builds a dependency graph of all the targets and their prerequisites. During the second phase, make uses this internalized data to determine which targets need to be updated and run the recipes necessary to update them.
It also relies on section 8.1, Function Call Syntax:
A function call resembles a variable reference. It can appear anywhere a variable reference can appear, and it is expanded using the same rules as variable references.
"The same rules" of course includes the rules for when expansions are performed.
Your recipes should be written in the language of the target shell, usually /bin/sh. All make functions and variable references in each recipe will be expanded before any recipe runs, so their expansions cannot reflect the results of running any recipe during the current make run. It's particularly peculiar to try to use the $(shell ...) function to do that, because you can just use shell code directly in a recipe.
As explained in the comments, all $(shell ...) and other functions are executed (expanded) before executing any lines from the recipe. But you can delay the expansion:
list_files:
#echo "Show Files..."
#echo $$(shell find . | grep testfile)
#touch testfile2
#echo $$(shell find . | grep testfile)
#rm testfile2
This yields the expected output. Note the double $$. Make will still expand all variables/functions first before executing the recipe, and remove one of the dollar signs. The expressions will be expanded again, when executing the recipe lines.
Of course, you're better off without the $(shell) in this example. However, it's not inherently a bad idea. Sometimes you need to execute shell commands before expanding other variables, and then $(shell) is the trivial solution.
Related
I am trying to remove the path prefix. Here is a small example showing just the issue.
Makefile
dist_directory = ./dist
default: build
build: $(patsubst %.md, $(dist_directory)/%.html, $(wildcard *.md))
$(dist_directory)/%.html: %.md
#echo start
#echo $#
#echo ${$#//$(dist_directory)/}
#echo end
Create a file: touch stuff.md
Then build: make
The output is:
start
dist/stuff.html
end
The expected output is:
start
dist/stuff.html
/stuff.html
end
There are similar posts on Stack Exchange. However, they have not worked for me in a Makefile for some reason. I'm probably doing something wrong.
https://unix.stackexchange.com/questions/311758/remove-specific-word-in-variable
Remove a fixed prefix/suffix from a string in Bash
Remove substring matching pattern both in the beginning and the end of the variable
You have many issues here. The most fundamental one is that if you want to use shell variables you have to escape the dollar sign so that make doesn't interpret it. And, you can only use shell variable substitutions on shell variables, while $# is a make variable, so you need:
#foo='$#' ; echo $${foo//$(dist_directory)/}
The more subtle one is that make always uses /bin/sh (POSIX standard shell) when it invokes recipes, and the above syntax is specific to bash. One way around that would be to explicitly set SHELL := /bin/bash in your makefile to force make to use bash. Luckily that is not necessary because POSIX sh can also do this, as mentioned by Reda in another answer:
#foo='$#' ; echo $${###*/}
But even more, you don't need any of this because make sets the automatic variable $* to the part of the target that matches the stem (the %):
#echo $*.html
It also sets $(#F) to the filename part of the $# variable:
#echo $(#F)
ETA
If you want to do something very similar to your shell variable expansion using GNU make you can use:
#echo $(patsubst $(dist_directory)/%,%,$#)
Haven't been using make for a while. But just got a project from a 10 years old compiler using Ubuntu.
I am looking at the makefile and trying to find out which compiler it is using.
${MAKE} is used in the file.
But where can I find out the definition of MAKE.
Thanks
You could simply use both the info and value built-in functions inside your makefile:
$(info MAKE: $(value MAKE))
This will work if MAKE is a recursively expanded variable, which it is by default. Otherwise, if MAKE were a simply expanded variable, you will see the expansion that was done at the moment of evaluating MAKE's definition (i.e., the same as $(MAKE)).
A better approach, which is independent of the flavour of the variable, would be to run make with the option -p and look at the definition of MAKE, e.g.:
make -p | grep 'MAKE ='
You will probably find out that MAKE is defined as:
MAKE = $(MAKE_COMMAND)
and MAKE_COMMAND, which is another variable (this time, a simply expanded one), may be in turn defined as:
MAKE_COMMAND := make
I have a few software projects which are distributed as RPMs. They are versioned using semantic versioning to which we affix a release number. Using the regular conventions, this is MAJOR.MINOR.PATCH-REL_NUM. Though beyond the scope of this article, the release numbers are stored in git. The release target in the makefile looks something like this:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
# Although the third step, this was re-ordered to step 1
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
While debugging, I eventually discovered that, although the call to eval was the third step in the recipe, it was actually being evaluated first! This is why the RPM always had a release number one less than the number I was watching get pushed to the remote.
I have done much googling on this and I haven't found any hits that explain the order of evaluation with regard to eval when used in recipes. Perhaps it isn't even with respect to eval but functions in general. Furthermore, I haven't found verbiage on this in the GNU manuals for make either (if it's there, kindly point out what chapter). I've worked around the problem so it's not a bother, I'm just wondering, is this expected and if so, why?
The missing bit, that no one above is getting, is simple: when make is going to run a recipe it expands all lines of the recipe first, before it starts the first line. So:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
# Although the third step, this was re-ordered to step 1
$(eval RELEASE_NUMBER=$(shell $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
when make decides to run the release target it first expands all the lines in the recipe, which means the eval is expanded, then it runs the resulting lines. That's why you're getting the behavior you're seeing.
I don't really see why you need to use eval here at all; why not just use:
release:
$(MAKE) clean
$(BLD_ROOT)/tools/incr_rel_num
$(MAKE) rpm RPM_RELEASE_NUM="$$(cat $(BLD_ROOT)/path/to/rel_num.txt))"
(BTW, you should never use bare make inside your makefiles; you should always use $(MAKE) (or ${MAKE}, same thing).
The $(eval ...) function
generates a fragment of make-sytax which becomes part of the parsed makefile.
The makefile is parsed entirely before any recipes are executed and when recipes
are executed all make-statements, make-expressions and make-variables have been
evaluated away.
So it does not make sense to consider an $(eval ...) call as being one
of the lines of a recipe. It might generate values that are used in the make-expansion
of the recipe, but if so then this happens when the makefile is parsed, before the recipe is run.
Thus in your example, the line:
$(eval RELEASE_NUMBER=$(shell $(BLD_ROOT)/path/to/rel_num.txt))
which I assume should really be:
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
is evaluated when the makefile is parsed, and let's say it results in the
make-variable RELEASE_NUMBER acquiring the value 1.0, because, when the
makefile is parsed, the file $(BLD_ROOT)/path/to/rel_num.txt) contains
1.0. In that case your recipe:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
will resolve to the like of:
release:
make clean
some_build_dir/tools/incr_rel_num
make rpm RPM_RELEASE_NUM=1.0
You will observe when make runs the recipe that it prints no line that
is "the expansion of" $(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt)),
because there is no such thing in the recipe. It doesn't matter that:
some_build_dir/tools/incr_rel_num
is presumably a command that writes, say, 1.1 or 2.0 in the file some_build_dir/path/to/rel_num.txt.
That action simply has no effect on the recipe. Nothing that executed in the recipe
can change the recipe.
$(eval ...) has no business in your recipe. What you want to achieve is simply:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
RELEASE_NUMBER=$$(cat $(BLD_ROOT)/path/to/rel_num.txt) && \
make rpm RPM_RELEASE_NUM=$$RELEASE_NUMBER
where $$ is what you do in a makefile to escape $ and, in this case,
leave it for the shell when the recipe is executed.
This recipe expands to 3 shell commands executed in sequence:
$ make clean
$ some_build_dir/tools/incr_rel_num
$ RELEASE_NUMBER=$(cat some_build_dir/path/to/rel_num.txt) && \
make rpm RPM_RELEASE_NUM=$RELEASE_NUMBER
and might as well be simplified further to:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
make rpm RPM_RELEASE_NUM=$$(cat $(BLD_ROOT)/path/to/rel_num.txt)
You are correct, there are multiple levels of evaluation. The content on what is inside eval is evaluated a first time before that the function is actually called. If you want the content of eval to be evaluated at the time eval is called, you have to escape the $ sign by putting it twice, like this :
$(eval RELEASE_NUMBER=$$(shell $(BLD_ROOT)/path/to/rel_num.txt))
To view what is really inside eval at the time it's called you can use the same syntax with info instead of eval :
$(info RELEASE_NUMBER=$$(shell $(BLD_ROOT)/path/to/rel_num.txt))
Now I'm not sure about the part which is evaluated too soon so the $ symbols that I doubled may not be the good one(s), but using the info function will help you to find the correct command.
I was looking at another stack overflow question, and they were using, what looks to be undocumented behavior for makefiles... If I have the following Makefile:
X=X
all:
#echo $#: $(X)
$(eval X=Y)
#echo $# part2: $(X)
all2:
#echo $#: $(X)
Then I run:
~> make all2
all2: X
~> make all all2
running all
all: X
all part2: Y
all2: Y
I would have expected the $(eval X=Y) to expand at Makefile parse time, and set the shell variable X to be Y for that line of the recipe (i.e. do nothing). Instead, it seems to be evaluated when the all recipe is run, plus, it seems to set the make variable. I've looked through the make man page, and for online manuals, but I can't find anything that describes this behavior (I'm using GNU Make 4.0). Can someone point me to some documentation describing this, or explain what's going on?
I'm not sure why you would expect either of the things you mention to be true. Maybe you're thinking that this is using the shell eval, somehow? That's not what it's doing, it's using the GNU make eval function, which is discussed here.
Just like any other variable or function, eval appearing in a recipe is not expanded until (and unless) the recipe is invoked because the target is out of date. See How make Reads a Makefile for full details on when variables and functions are expanded.
Second, all make variable assignments set make variables, and only if you export them will they be sent to the shell.
I have a project that involves sub-directories with sub-makefiles. I'm aware that I can pass variables from a parent makefile to a sub-makefile through the environment using the export command. Is there a way to pass variables from a sub-makefile to its calling makefile? I.e. can export work in the reverse? I've attempted this with no success. I'm guessing once the sub-make finishes its shell is destroyed along with its environment variables. Is there another standard way of passing variables upward?
The short answer to your question is: no, you can't [directly] do what you want for a recursive build (see below for a non-recursive build).
Make executes a sub-make process as a recipe line like any other command. Its stdout/stderr get printed to the terminal like any other process. In general, a sub-process cannot affect the parent's environment (obviously we're not talking about environment here, but the same principle applies) -- unless you intentionally build something like that into the parent process, but then you'd be using IPC mechanisms to pull it off.
There are a number of ways I could imagine for pulling this off, all of which sound like an awful thing to do. For example you could write to a file and source it with an include directive (note: untested) inside an eval:
some_target:
${MAKE} ${MFLAGS} -f /path/to/makefile
some_other_target : some_target
$(eval include /path/to/new/file)
... though it has to be in a separate target as written above because all $(macro statements) are evaluated before the recipe begins execution, even if the macro is on a later line of the recipe.
gmake v4.x has a new feature that allows you to write out to a file directly from a makefile directive. An example from the documentation:
If the command required each argument to be on a separate line of the
input file, you might write your recipe like this:
program: $(OBJECTS)
$(file >$#.in) $(foreach O,$^,$(file >>$#.in,$O))
$(CMD) $(CMDFLAGS) #$#.in
#rm $#.in
(gnu.org)
... but you'd still need an $(eval include ...) macro in a separate recipe to consume the file contents.
I'm very leery of using $(eval include ...) in a recipe; in a parallel build, the included file can affect make variables and the timing for when the inclusion occurs could be non-deterministic w/respect to other targets being built in parallel.
You'd be much better off finding a more natural solution to your problem. I would start by taking a step back and asking yourself "what problem am I trying to solve, and how have other people solved that problem?" If you aren't finding people trying to solve that problem, there's a good chance it's because they didn't start down a path you're on.
edit You can do what you want for a non-recursive build. For example:
# makefile1
include makefile2
my_tool: ${OBJS}
# makefile2
OBJS := some.o list.o of.o objects.o
... though I caution you to be very careful with this. The build I maintain is extremely large (around 250 makefiles). Each level includes with a statement like the following:
include ${SOME_DIRECTORY}/*/makefile
The danger here is you don't want people in one tree depending on variables from another tree. There are a few spots where for the short term I've had to do something like what you want: sub-makefiles append to a variable, then that variable gets used in the parent makefile. In the long term that's going away because it's brittle/unsafe, but for the time being I've had to use it.
I suggest you read the paper Recursive Make Considered Harmful (if that link doesn't work, just google the name of the paper).
Your directory structure probably looks like this:
my_proj
|-- Makefile
|-- dir1
| `-- Makefile
`-- dir2
`-- Makefile
And what you are doing in your parent Makefile is probably this:
make -C ./dir1
make -C ./dir2
This actually spawns/forks a new child process for every make call.
You are asking for updating the environment of the parent process from its children, but that's not possible by design (1, 2).
You still could work around this by:
using a file as shared memory between two processes (see Brian's answer)
using the child's exit error code as a trigger for different actions [ugly trick]
I think the simplest solution is using standard out from a sub Makefile.
Parent Makefile
VAR := $(shell $(MAKE) -s -C child-directory)
all:
echo $(VAR)
Child Makefile
all:
#echo "MessageToTheParent"