I am trying to create a file and set the contents of a file to a variable but the variable is always seen as empty.
An example:
define FILE_VAR
cat /output/1.txt >> /output/2.txt
$(eval s := $(shell cat /output/2.txt))
$(eval FILENAME_BIN=$(word 1, $(s)).$(word 2, $(s)).$(word 3, $(s)).bin)
$(eval FILENAME_JFFS2=$(word 1, $(s)).$(word 2, $(s)).$(word 3, $(s)).jffs2)
endef
If 2.txt exists before running this the variables will be set to the data prior to running make (not the new redirected data), if 2.txt doesn't exist then the variables are not set. It looks like it is evaluating what the file is like when make is first run, which is not what I want...
You are unclear as to what is done by GNU make, and when, and what is done by the shell,
and when, to execute a recipe.
Anything anywhere in a makefile of the form $(...) is evaluated by make
unless it escaped as $$(...) in a shell command.
A shell command, unless in the context of the make-function $(shell ...) or
back-ticks can only be a command within a recipe:
target: [prerequisite...]
command
...
The commands composing a recipe are executed in sequence, each in a distinct
shell, to execute the recipe.
Nothing of the unescaped form $(...) is a command in the command-sequence
of a recipe unless it is the expansion of a variable or macro you have defined that expands to a command.
A line in the scope of a target need not be a command or expand to a command. It
may also consist of an $(...) expression that expands to nothing,
and simply instructs make to do something, e.g.
$(info ...)
expands to nothing and tells make to print ... on the standard output.
$(eval ...)
expands to nothing and tells make to evaluate ...
This recipe has just two commands, not four:
target: prereq...
$(eval TARGET=$#)
command1
$(info $(TARGET))
command2
Any make-expressions, $(...) in the scope of a recipe are evaluated in
sequence when make decides to run the recipe and the command-sequence is what
is left after they have all been evaluated. Then
the command-sequence is run. For example:
Makefile
target:
echo Hello > hello.txt
$(info Begin {)
$(eval s := $(shell cat hello.txt))
$(eval WORD=$(word 1, $(s)))
$(info [$(WORD)])
$(info } End)
Run that:
$ make
Begin {
cat: hello.txt: No such file or directory
[]
} End
echo Hello > hello.txt
Observe that all of the make-expressions are evaluated before the
commands of the target are run.
Your FILE_VAR macro is evidently intended to be expanded in recipe scope, since the first line is a shell command.
The remaining lines must therefore achieve their purposes by shell commands if they depend upon the
the effects of running the first line. None of them does so. The remaining 3 lines
are evaluated before 2.txt exists, if it doesn't already exist at make-time.
Related
new_contents = "\$$(cooly)"
all:
mkdir -p subdir
echo $(new_contents) | sed -e 's/^ //' > subdir/makefile
#echo "---MAKEFILE CONTENTS---"
#cd subdir && cat makefile
#echo "---END MAKEFILE CONTENTS---"
#cd subdir && $(MAKE)
# Note that variables and exports. They are set/affected globally.
cooly = "The subdirectory can see me!"
export cooly
# This would nullify the line above: unexport cooly
clean:
rm -rf subdir
What I want is a "$(cooly)" string, not the variable value.
I tried several combinations:
new_contents = "$(cooly)", gives the variable value, The subdirectory can see me!
new_contents = "$$", gives $
new_contents = "\$(cooly)", gives Syntax error: Unterminated quoted string error
Why new_contents = "$$(cooly)" doesn't give "$(cooly)" but result in nothing?
"$$" -> "$", so why isn't "$$(cooly)" ---> "$(cooly)"?
You have to understand both how make expansion works, and how shell expansion works, in order to write more complicated recipes in make. That's because, make recipes are passed to the shell after make is done expanding them.
Make passes recipe lines to the shell virtually verbatim: there is only one character that's special (not counting backslash/newlines at the end) and that's $. If make sees a $ it will try to expand it as a variable reference. To avoid that, you have to escape it as $$ to hide it from make.
So let's look at your makefile:
cooly = "The subdirectory can see me!"
echo $(new_contents) ...
If new_contents is "$(cooly)", make sees the $(cooly) as a variable reference and expands it before it even invokes the shell. So first make expands $(new_contents) to "$(cooly)", then it expands that to ""The subdirectory can see me!"" (because the quotes are in both variables, and quotes are not special to make: they're just like any other character like a or b). The result will be:
echo ""The subdirectory can see me!""
The shell will toss the quotes since they're no-ops and echo that value (into the pipe).
If new_contents is "\$(cooly)", that backslash doesn't mean anything to make. Just like quotes, backslashes (unless they are at the end of a line) are not special to make. So make expands just as before, but this time the command it passes to the shell is this:
echo "\"The subdirectory can see me!""
backslashes are not special to make, but they are special to the shell. Here you've escaped the second quote so the shell doesn't treat it as a quote character, which means you have an odd number of quotes in your command, which is why you get an error from the shell about non-terminated quotes.
If new_contents is "$$(cooly)", make doesn't expand the variable, it is passed along to the shell like this:
echo "$(cooly)"
However, $ is also special to the shell. Putting it in double quotes doesn't prevent the shell from trying to expand it. This tells the shell to run the command cooly and substitute the output. Almost certainly there is no command named cooly and so you'll get an error message to stderr (maybe you didn't notice it) and the shell will replace it with nothing because it didn't print anything to stdout.
If new_contents is "\$$(cooly)" then make will not expand, and run this shell command:
echo "\$(cooly)"
The shell sees the backslash and doesn't expand the $ but instead uses it literally, and you get the result you want.
Here are some hints:
First, do not include quotes in your make variables (unless the variable contains an entire shell command and you need quotes inside it). Make doesn't care about quotes and having them embedded in variables makes it very difficult to reason about what the shell will see.
Include the quotes only in the recipe.
Second, remember that since make doesn't care about quotes, it doesn't have the same behavior as the shell WRT single vs. double quotes. You can use single quotes around make variables to reduce the need to escape things from the shell, without hiding them from make.
So, I would write this:
new_contents = $$(cooly)
cooly = The subdirectory can see me!
all:
mkdir -p subdir
echo '$(new_contents)' | sed -e 's/^ //' > subdir/makefile
...
BTW, it's never a good idea to add # values to your makefile until it's completely done and working. Seeing the output make prints (which is what it's sending to the shell) is a great help in figuring out whether your recipes are right, and whether the problem is with your make constructs or shell constructs.
Makefile:
.PHONY: all
SHELL:=/usr/bin/env bash
all:
$(eval x=$(shell cat file))
#echo "$x"
File:
foo
bar
Output:
foo bar
How do I get the contents of the file into the make variable without losing the newlines?
You can't do this with shell, as described in its documentation.
If you have a sufficiently new version of GNU make, you can use the file function however.
Make converts newlines from shell outputs to spaces (see here):
The shell function performs the same function that backquotes (‘`’)
perform in most shells: it does command expansion. This means that it
takes as an argument a shell command and evaluates to the output of
the command. The only processing make does on the result is to convert
each newline (or carriage-return / newline pair) to a single space. If
there is a trailing (carriage-return and) newline it will simply be
removed.
So, you cannot preserve spaces from the $(shell) command directly. That being said, make does allow multiline variables using define -- but beware, attempting to use such variables is problematic. Consider:
define x
foo
bar
endef
all:
#echo "$x"
Make expands the $x in place, and you end up with:
all:
#echo " foo
bar"
(where the newline is considered the end of the recipe line..).
Depending on what you want this for, you may be able to get around this is using a bash variable:
all:
#x=$$(cat file); \
echo $$x
Or potentially storing your output in a file, and referencing that when necessary.
all:
eval (cat file >> output.txt)
cat output.txt
(and yes, the last one is convoluted as written, but I'm not sure what you're trying to do, and this allows the output of your command to be persistent across recipe lines).
If the file contents are ensured not to contain any binary data, and if you're willing to to extra processing each time you access the variable, then you could:
foo:=$(shell cat file | tr '\n' '\1')
all:
#echo "$(shell echo "$(foo)" | tr '\1' '\n')"
Note that you cannot use nulls \0, and I suspect that probably means there's a buffer overflow bug in my copy of Make.
The GNU make manual says that the eval function expands the arguments and then feeds the results of the expansion to make parser. The following is quoted from GNU make manual.
The argument to the eval function is expanded, then the results of that expansion are parsed as makefile syntax.
I don't quite understand how the make parser process the text fed by eval, so
I write following makefile to test.
define myprint
echo "this is a line"
endef
goal:
$(eval $(call myprint))
gcc -o goal test.c
I know that the correct invocation of myprint should be only use the call function: $(call myprint) and delete the 'Tab' character before echo. I write the makefile in this form just to test the eval function.
My expectation: first the eval function expands myprint, which is an echo command preceded by a 'Tab', and the 'Tab' is used to make the expanded text to be a legal recipe. Then the eval feeds the expanded text to maker parser, who will identify the text to be a recipe, and run it. As the command is legal, the makefile should run properly.
However, I meet such an error:
Makefile:6: *** recipe commences before first target. Stop.
Could somebody explain why make produce such an error?
the results of that expansion are parsed as makefile syntax
Your use of eval is different: you would like it to be parsed as shell syntax. You can write:
define myprint
echo "this is a line"
endef
goal:
$(myprint)
gcc -o goal test.c
or:
define myprint
echo "this is a $(1)"
endef
goal:
$(call myprint,line)
gcc -o goal test.c
Because after make expansion the recipes are valid shell syntax. But not what you wrote because the expansion of eval is still interpreted as make syntax, not shell. To illustrate a typical use of eval and call, consider this:
define myprint
echo "this is a $(1)"
endef
define mygoal
$(1):
$$(call myprint,line)
gcc -o $(1) $(2).c
endef
$(eval $(call mygoal,goal,test))
It is a bit more tricky than two first examples (without eval) but it illustrates the real purpose of eval: programmatically instantiate make constructs. Here is how it works, step by step:
During the first phase of its 2-phases algorithm, make expands the $(eval... function call, that is:
Expand the parameter of the $(eval... function (the $(call... function):
Expand the parameters of the $(call... function (goal and test). No effect in our case.
Assign the result to the temporary variables $(1) and $(2).
Expand the mygoal variable in this context, which replaces $(1), $(2) and $$(call... by goal, test and $(call..., respectively.
Instantiates (in memory) the result as a make construct, a complete rule in this case:
goal:
$(call myprint,line)
gcc -o goal test.c
The first phase continues but it has no effect on this instantiated rule because the recipes are expanded by make during the second phase.
During the second phase, when the time comes to build the goal target, make expands the recipe before executing it, that is:
Expand the $(call myprint... parameter (line, no effect).
Assign the result to temporary parameter $(1).
Expand variable myprint in this context, which produces:
echo "this is a line"
All this is thus the same as if we had written the rule:
goal:
echo "this is a line"
gcc -o goal test.c
Note the double $$ in the initial definition of mygoal:
It’s important to realize that the eval argument is expanded twice;
first by the eval function, then the results of that expansion are
expanded again when they are parsed as makefile syntax. This means you
may need to provide extra levels of escaping for “$” characters when
using eval.
$(eval …) needs a syntactically complete makefile fragment. It cannot be used to paste tokens into other makefile constructs. Perhaps the manual does not explain this clearly enough, but it's implemented by reading its argument as if it were an included makefile.
#RenaudPacalet, I write following makefile to test whether the expansion of call 'eats' one dollar.
define myprint
echo "In my print $$(ls)"
endef
goal:
$(call myprint)
$(info $(call myprint))
gcc -o goal test.c
Its output is:
echo "In my print $(ls)"
echo "In my print $(ls)"
In my print call.mk ... (files list)
As $(call myprint) outputs "In my print $(ls)" correctly, it must be expanded to echo "In my print $$(ls)" first, then it will be expanded to the correct shell command echo "In my print $(ls)". So I think the call function does not 'eats' one dollar.
Another evidence is the output of info function. The GNU make manual says:
$(info text…)
This function does nothing more than print its (expanded) argument(s) to standard output.
From the manual we can infer that make will expand the arguments of the info function. As the output of the info is echo "In my print $(ls)", so the arguments before expansion should be echo "In my print $$(ls)". So we can conclude that the call function doesn't 'eat' one dollar.
Does anyone know how to use a here-document redirection on a recipe?
test:
sh <<EOF
echo I Need This
echo To Work
ls
EOF
I can't find any solution trying the usual backslash method (which basically ends with a command in a single line).
Rationale:
I have a set of multi-line recipes that I want to proxy through another command (e.g., sh, docker).
onelinerecipe := echo l1
define twolinerecipe :=
echo l1
echo l2
endef
define threelinerecipe :=
echo l1
echo l2
echo l3
endef
# sh as proxy command and proof of concept
proxy := sh
test1:
$(proxy) <<EOF
$(onelinerecipe)
EOF
test2:
$(proxy) <<EOF
$(twolinerecipe)
EOF
test3:
$(proxy) <<EOF
$(threelinerecipe)
EOF
The solution I would love to avoid: transform multiline macros into single lines.
define threelinerecipe :=
echo l1;
echo l2;
echo l3
endef
test3:
$(proxy) <<< "$(strip $(threelinerecipe))"
This works (I use gmake 4.0 and bash as make's shell) but it requires changing my recipes and I have a lot.
Strip removes the newlines, from the macro, then everything is written in a single line.
My end goal is: proxy := docker run ...
Using the line .ONESHELL: somewhere in your Makefile will send all recipe lines to a single shell invocation, you should find your original Makefile works as expected.
When make sees a multi-line block in a recipe
(i.e., a block of lines all ending in \, apart from the last),
it passes that block un-modifed to the shell.
This generally works in bash,
apart from here docs.
One way around this is to strip any trailing \s,
then pass the resulting string to bash's eval.
You do this in make by playing with ${.SHELLFLAGS} and ${SHELL}.
You can use both of these in target-specific form if you only want it to kick in for a few targets.
.PHONY: heredoc
heredoc: .SHELLFLAGS = -c eval
heredoc: SHELL = bash -c 'eval "$${#//\\\\/}"'
heredoc:
#echo First
#cat <<-there \
here line1 \
here anotherline \
there
#echo Last
giving
$ make
First
here line1
here anotherline
Last
Careful with that quoting, Eugene.
Note the cheat here:
I am removing all backslashes,
not just the ones at the ends of the line.
YMMV.
With GNU make, you can combine multi-line variables with the export directive to use a multi-line command without having to turn on .ONESHELL globally:
define script
cat <<'EOF'
here document in multi-line shell snippet
called from the "$#" target
EOF
endef
export script
run:; # eval "$$script"
will give
here document in multi-line shell snippet
called from the "run" target
You can also combine it with the value function to prevent its value from being expanded by make:
define _script
cat <<EOF
SHELL var expanded by the shell to $SHELL, pid is $$
EOF
endef
export script = $(value _script)
run:; # eval "$$script"
will give
SHELL var expanded by the shell to /bin/sh, pid is 12712
Not a here doc but this might be a useful workaround.
And it doesn’t require any GNU Make’isms.
Put the lines in a subshell with parens, prepend each line with echo.
You’ll need trailing sloshes and semi-colon and slosh where appropriate.
test:
( \
echo echo I Need This ;\
echo echo To Work ;\
echo ls \
) \
| sh
I'm trying to check whether a variable is defined using ifndef/ifdef, but I keep getting a not found error from the execution. I'm using GNU Make 3.81, and here is a snippet of what I have:
all: _images
$(call clean, .)
$(call compile, .)
#$(OPENER) *.pdf &
_images:
$(call clean, "images")
$(call compile, "images")
define clean
#rm -f ${1}/*.log ${1}/*.aux ${1}/*.pdf
endef
define compile
ifdef ${1}
dir = ${1}
else
dir = .
endif
ifdef ${2}
outdir = ${2}
else
outdir = ${1}
endif
#$(COMPILER) -output-directory ${outdir} ${dir}/*.tex
endef
And the exact error:
$ make
ifdef "images"
/bin/sh: 1: ifdef: not found
make: *** [_images] Error 127
Edit:
Considering Barmar comments, here goes the conclusions:
The contents of a define are shell command lines, not make directives;
to break lines inside commands within a define block, the linebreak must be escaped -- with \;
also, each block corresponding to one-liner commands is executed separately, each in a different shell execution, which means that, defining local variables won't work if the intention is to access the variable value in the next one-liner block.
Thanks tripleee for the nice work around.
You can combine the shell's facilities with Make's to get a fairly succinct definition.
define compile
#dir="${1}"; outdir="${2}"; outdir=$${outdir:-"$dir"}; \
$(COMPILER) -output-directory "$${outdir}" "$${dir:-.}/*.tex
The double-dollar is an escape which passes a single dollar sign to the shell. The construct ${variable:-value} returns the value of $variable unless it is unset or empty, in which case it returns value. Because ${1} and ${2} are replaced by static strings before the shell evaluates this expression, we have to take the roundabout route of assigning them to variables before examining them.
This also demonstrates how to combine two "one-liners" into a single shell invocation. The semicolon is a statement terminator (basically equivalent to a newline) and the sequence of a backslash and a newline causes the next line to be merged with the current line into a single "logical line".
This is complex enough that I would recommend you omit the leading # but I left it in just to show where it belongs. If you want silent operation, once you have it properly debugged, run with make -s.