I have a file, tmp containing:
{
"VolumeId": "vol--22222222",
}
and a makefile, containing only:
MYFILE =latest
x:
\cp tmp $(MYFILE)
grep VolumeId $(MYFILE)
#echo $(shell grep VolumeId $(MYFILE))
If I run make x, I get:
\cp tmp latest
grep VolumeId latest
"VolumeId": "vol--22222222",
VolumeId: vol--22222222,
as expected. If I modify the file tmp, replacing 2's wit 4's, I get:
\cp tmp latest
grep VolumeId latest
"VolumeId": "vol--44444444444",
VolumeId: vol--22222222,
...huh? The greps return different results! The second contains data from the file before the copy.
I run
rm latest ; make x
I get:
\cp tmp latest
grep VolumeId latest
"VolumeId": "vol--4444444444",
What's going on here?
GNU Make 3.81. Ubuntu 12.04 on VMWare 5.
Update 1
Here's a more explicit example
CMD0 =$(shell date +"%s.%N")
CMD1 =date +"%s.%N"
CMD2 =date +"%s.%N"
y:
#sleep 2
#date +"%s.%N" # 2nd
#echo $(CMD0) # 1st A
#sleep 2
#date +"%s.%N" # 3rd
#sleep 2
#$(CMD1) # 4th
#sleep 2
#echo $(shell $(CMD2) ) # 1st B
Output:
1381596581.761093768
1381596579.743610973
1381596583.769058027
1381596585.774766561
1381596579.751625601
Its look like all the $(shell ... ) commands get evaluated together before any line of the y recipe.
This is a little surprising (and counter-intuitive - what's the rationale for this odd design?). It has important consequences if the shell commands have side effects. Also, I can't find any documentation in make manual that describes this order of evaluation.
Update 2
What's especially odd about the above is that its hard to come up with a mental model for the order of evaluation. Is it that any rule of the form, $(function ... ) is executed before each line of the recipe? If so, why is $(CMD0) evaluated before the recipe, but not $(CMD1). Its true that CMD0 contains a $(f ... ) - does make look at this, even though CMD0 is declared as a delayed evaluated variable (i.e.declared with = not :=)?
Update 3
Reducing it down to essential components:
notSafe:
backup-everything # Format executed even if backup fails
echo $(shell format-disk) | tee log.txt #
CMDX =$(shell format-disk)
alsoNotSafe:
backup-everything # Format executed even if backup fails
echo $(CMDX) | tee log.txt # Even though CMDX is a delayed evaluation variable
CMDZ =format-disk
safe:
backup-everything # Works as expected.
$(CMDZ) | tee log.txt # Evaluation of CMDZ is delayed until line is executed.
Mad. Who designed this?
You're not telling us everything: when you run it the 1st time, you echo an empty line. Instead you get a grep file not found error.
And that's the decisive hint: $-expressions are evaluated by any variant of make before running the commands of the rule. I.e. in each later run you see the file contents of the previous run.
Btw. your leading backslash is spurious and causes the command to be passed to the Shell, instead of being directly executed by make. With makepp's builtin commands (prefixed by &) and fixed dependencies that would be:
latest: tmp
&cp $(input) $(output)
&grep VolumeId $(output)
#&echo $(&grep VolumeId $(output))
That doesn't change the problem, but by using the proper rule variables, makes it obvious what's wrong.
Related
Consider the following Makefile.
$(shell touch /tmp/example.txt)
FILE := /tmp/example.txt
CONTENTS = $(shell cat $(FILE); bash -c 'echo [debugging id: $$RANDOM]')
.PHONY: all
all:
#cat $(FILE)
#echo '$$(CONTENTS):' $(CONTENTS)
bash -c 'echo file-contents-$$RANDOM' > $(FILE)
#cat $(FILE)
#echo '$$(CONTENTS):' $(CONTENTS) # This line outputs the old contents. Why?
It prints the contents of the file, overwrites with new contents and prints the contents again. It shows as (after second shots of make):
file-contents-1543
$(CONTENTS): file-contents-1543 [debugging id: 15172]
bash -c 'echo file-contents-$RANDOM' > /tmp/example.txt
file-contents-22441
$(CONTENTS): file-contents-1543 [debugging id: 151]
The old content is file-contents-1543 and new content is file-contents-22441 (the numbers are random), but the last line echo $(CONTENTS) does not print the new contents.
I think the command is actually called twice as debugging ids show but shell function in the lazy variable seems to be executed before writing the new contents to the file.
I expect that lazy variable in Makefile is evaluated every time it is referred, the echo $(CONTENTS) command always prints the latest file contents. What am I wrong?
By the way, I found that using CONTENTS = $$(cat $(FILE)) works as I expect. I will using this instead of shell function but is it ok?
I expect that lazy variable in Makefile is evaluated every time it is referred, the echo $(CONTENTS) command always prints the latest file contents. What am I wrong?
First of all, in make's slang these variables are called recursive, not lazy. And, yes, they get expanded (i.e. recursively substituted) each time they are referred with $(CONTENTS). Considering that $(eval...) and $(shell...) (as pretty much anything looking as $(...)) also went through the same (recursive) expansion procedure (albeit, with some "side-effects"), each expansion of such variable could also result in some sort of "evaluation" or "execution".
Next, the order of expansion in make is a bit specific. In particular, the recipes (i.e. the lines starting with [tab]) are expanded after the whole makefile was (pre-)processed, but before the first line of the recipe gets executed by shell. Which is the main source of your confusion, I suppose.
I found that using CONTENTS = $$(cat $(FILE)) works as I expect
$$ is a way to get a single literal $ after an expansion procedure. So $$(cat $(FILE)) when expanded becomes $(cat /tmp/example.txt) which is a legal syntax for command substitution in bash. This means it will work only as part of a bash command (recipe line). If that is what you want then it's okay.
I am trying to follow the meaty skeleton tutorial on osdev. The Makefile is not running one of the shell scripts. I have set all of the permissions on each of the files to be executable.
In lib/Makefile, I have the below few lines set:
$(info DEFAULT_HOST!=../default-host.sh)
$(info HOST?=DEFAULT_HOST)
$(info HOSTARCH!=../target-triplet-to-arch.sh $(HOST))
after these lines have executed, neither DEFAULT_HOST nor HOSTARCH get set.
default-host.sh:
#!/bin/sh
echo i686-elf
arget-triplet-to-arch.sh:
#!/bin/sh
if echo "$1" | grep -Eq 'i[[:digit:]]86-'; then
touch here.txt
echo i386
else
touch there.txt
echo "$1" | grep -Eo '^[[:alnum:]_]*'
fi
Note, I added the touch statements in arget-triplet-to-arch.sh. When run from the shell, one or other of those files is created, but not when the Makefile is run. This means that make seems to not be running the shell commands. How can I get make to run the shell commands?
As Beta says, info doesn't "allow you to see the value of that line being evaluated". info expands its argument then prints it to stdout. "Expands" means it resolves any variable references, it doesn't mean interpreting it as a makefile command. So if you run $(info hi) it prints "hi". If you run $(info foo = bar) if prints foo = bar but does not set the value of the variable foo to bar.
For using !=, note that this feature was added to GNU make 4.0. If your version is older than that then this assignment doesn't do what you expect. In particular, a line like FOO!=echo bar will be interpreted as if it were FOO! = echo bar... in other words it sets the make variable named FOO!.
Personally I always put whitespace around the assignment statements in my makefiles... this makes it clear that they are make assignments, not shell variable assignments (not that it shouldn't be clear anyway for anyone who knows makefile syntax, but...). In newer versions of GNU make, variable names cannot contain whitespace.
I am using make's shell command to populate some variables, and on output it indicates that they are being set to the values I expect. However when I pass them to my make recipes they all show up empty. I suspect make is doing some odd parsing on the results. Eg:
MyTarget: $(MySources)
LINE='$(shell cat $< | grep GIMME_THE_LINE_I_WANT)'
CH9=$(shell echo $(LINE) | cut -b9)
echo $(CH9) # doesn't print anything
I checked my generator commands manually by setting SHELL=sh -XV and when I run identical commands I get the right values, it just looks like bash is 'zeroing' my variables. Any idea what's wrong?
There are several things going on here. The first is that when you have:
MyTarget: $(MySources)
LINE='$(shell cat $< | grep GIMME_THE_LINE_I_WANT)'
You are setting a shell variable called LINE, not a make variable. The build instructions for a target are all shell commands. So after the first line, the shell variable $LINE contains GIMME_THE_LINE_I_WANT. However...
...each line in the build instructions for a target runs in a separate shell process. So if you have:
mytarget:
MYVAR=foo
echo $$MYVAR
You'll see no output, because $MYVAR isn't set in the context of the second command. Also note the use of $$ here, because otherwise the $ would be interpreted by Make (that is, writing $MYVAR would actually be the make expression $M followed by the text YVAR). You can resolve this by logically joining your lines into a single shell script, like this:
mytarget:
MYVAR=foo; \
echo $$MYVAR
The \ is Makefile syntax that extends a single logical line over multiple physical lines, and of course ; is simply shell syntax for combining multiple commands on one line.
With all this in mind, we could rewrite your target like this:
MyTarget: $(MySources)
LINE=$$(cat $< | grep GIMME_THE_LINE_I_WANT); \
CH9=$$(echo $$LINE | cut -b9); \
echo $$CH9
Notice that since we are already running a shell script I'm not using Make's $(shell ...) construct, and that I'm making sure to escape all of the $ characters to ensure that the shell, not Make, is handling variable expansion.
Taking it just a little further, you don't need to use cat in that script; you could simply write:
MyTarget: $(MySources)
LINE=$$(grep GIMME_THE_LINE_I_WANT $<); \
CH9=$$(echo $$LINE | cut -b9); \
echo $$CH9
Or:
MyTarget: $(MySources)
CH9=$$(grep GIMME_THE_LINE_I_WANT $< | cut -b9); \
echo $$CH9
(NB: While not germane to this solution, it's although worth noting that each invocation of $(shell ...) is also run in a separate process, so a variable set in one won't be available in another.)
The make runs every command in its separate shell. So, the values are not carried over.
When in doubt, you could always debug it with -d option. Also, a site note, the debug option is very useful when you are trying to figure out why a rule did not fire the way you had intended it.
~> cat Makefile
MyTarget:
LINE="somevalue"
echo ${LINE}
~>
~>
~> make -d MyTarget | tail -10
LINE="somevalue"
Putting child 0x2004fbb0 (MyTarget) PID 4052 on the chain.
Live child 0x2004fbb0 (MyTarget) PID 4052
Reaping winning child 0x2004fbb0 PID 4052
echo
Live child 0x2004fbb0 (MyTarget) PID 4192
Reaping winning child 0x2004fbb0 PID 4192
Removing child 0x2004fbb0 PID 4192 from chain.
Successfully remade target file 'MyTarget'.
~>
Just clarifying for the people who downvoted and commented that my above solution doesn't work: First, I apologize for my laziness. May be I should have been clear. Let me try again.
The "make -d" is not a solution to OP's problem.
I tried to show OP how he/she could use debug option to solve a variety of problems that people come across while using makefiles (which, I admit, goes in a slight tangent than just solving the OP's problem at hand).
The above debug shows that the first command was executed in a shell with PID=4052 and the second command was executed in another shell with PID=4192 (which doesn't carry the value of that variable). Also it shows that using a variable with single dollar (${LINE}) just gives you a blank (because the makefile doesn't interpret it as a shell variable).
Again, to be clear: "make -d" is not a solution. Just combine the commands in one line, separated by commas, use double dollars; if the line is long, escape the new lines.
MyTarget:
LINE="somevalue"; \
echo $${LINE}
I have a rule like this in a makefile:
target : dependencies
rm -rd somedirectory
runcodecoverage.exe # this generates somefile
$(eval COVERAGE=$(shell grep "blah" somefile))
#echo $(COVERAGE)
When I run this file the first time (after make cleaning) the echo doesn't print anything. But the second and times after that, it prints the correct result. If I replace the $(eval ..) line with just grep "blah" somefile I get the result I want, so the problem must be in the use of $(eval) and $(shell). Why is this happening?
Edit: I solved this by adding a new dependency, so it now looks like this:
generatesomefile :
runcodecoverage.exe # this generates somefile
target : dependencies generatesomefile
rm -rd somedirectory
$(eval COVERAGE=$(shell grep "blah" somefile))
#echo $(COVERAGE)
It seems like $(eval) was being substituted with the result of grep as soon as the "target" target was entered, even though I wanted it to be run after runcodecoverage.exe had run. In this sense the answer I accepted as correct wasn't quite right - the docs say this on variable expansion:
Rule Definition
A rule is always expanded the same way, regardless of the form:
immediate : immediate ; deferred
deferred
The eval function doesn't work the way you think it does.
The first time you run Make, it expands the eval function and carries out the variable assignment before executing any rule. And since there is no somefile, the grep returns nothing and COVERAGE stays empty. When Make executes the rule, it passes the empty variable to the echo which duly reports nothing.
The second time you run Make, once again it expands the eval function, executes the grep on somefile (which was built in the first run), and stores the result in COVERAGE. Then it executes the rule and passes COVERAGE to the echo, which puts it up on your screen.
The $(eval ...) runs when your Makefile is parsed, not when the recipe is executed.
It's not clear to me what you expect to happen; setting a Make variable from within a recipe does not seem like a sustainable approach, but it depends a lot on where and how you need to use this variable. If all you need is to output the coverage from the same recipe, replace the last couple of lines with just
grep "blah" somefile
I was thinking about using Make for small checks for my dev setup. One thing I want is to check that a particular string exists in a file before doing some action. If I wanted to create the entire file it would be trivial
action: filename
...
filename:
echo 'blah' >> filename
But how can this logic be applied to actions, like grep? My dependency isn't that a file exists, it's that the file has correct content.
I'm asking specifically about Make and not other solutions like chef/puppet
You can run any shell commands you want in a make recipe. As many of them as you want also.
So if you need to run grep before doing something else just do that.
Just remember that every line in a recipe is run in its own shell session so they don't share state.
So this:
action: filename
...
filename:
grep -qw blah $# || echo 'blah' > $#
runs grep on filename (via the automatic variable for the current target $#) looking for whole words and quitting on the first match (-q).
If grep finds blah then it will return success and the || will short-circuit and the recipe is done. If grep fails then the || will trigger and the echo will run.
You might be tempted to do things that require the inverse logic do X only if Y is true:
filename:
grep -qw blah $# && echo blah2 > $#
but that doesn't work correctly. When grep fails the && short-circuits and make sees a recipe failure and bails the make process out with an error.
You need this instead.
filename:
! grep -qw blah $# || echo blah2 > $#
to invert the logic and ensure that the "failure" from grep is seen as success as far as make is concerned.
That all being said in this specific example if filename exists at all then that recipe won't ever run as it has no prerequisites so make will always consider it up to date. To work around that you need to give the file a prerequisite that will force it to be considered out of date. Specifically a force target.
Don't follow the advice about .PHONY for this case though. .PHONY targets should never be prerequisites of non-.PHONY targets.
Expanding on what #john wrote I got the following to work:
TEST_FILE=filename
.PHONY: ${TEST_FILE}
string=testing
filecheck=$(shell grep -qw ${string} ${TEST_FILE} || echo ${TEST_FILE})
all: ${filecheck}
${TEST_FILE}:
echo 'changing the file'
echo ${string} >> ${TEST_FILE}
Here the file on which I'm operating is a .PHONY target. I think that's ok because I'm actually not creating the file, just modifying it. This will work if the file does not exist, or exists without the needed string.
You could add a test in the target's recipe (As Etan posted before I could complete this answer...). If you do want to do this using just make logic, you could do something along the lines of:
actions: $(if $(shell grep -q $$string filename && echo y),filename,)
filename:
echo blah >> $#
If filename contains the string, then there will be an actions: filename dependency, and filename will be built when you build actions. Notice, though that this will check whether the string exists in filename at the time the makefile is parsed -- if filename is generated, or modified in this makefile, then it would not effect whether the action is run. If you want to test right before overwriting the file, then you would use a bash if statement in the recipe itself.