Makefile:
.PHONY: all
SHELL:=/usr/bin/env bash
all:
$(eval x=$(shell cat file))
#echo "$x"
File:
foo
bar
Output:
foo bar
How do I get the contents of the file into the make variable without losing the newlines?
You can't do this with shell, as described in its documentation.
If you have a sufficiently new version of GNU make, you can use the file function however.
Make converts newlines from shell outputs to spaces (see here):
The shell function performs the same function that backquotes (‘`’)
perform in most shells: it does command expansion. This means that it
takes as an argument a shell command and evaluates to the output of
the command. The only processing make does on the result is to convert
each newline (or carriage-return / newline pair) to a single space. If
there is a trailing (carriage-return and) newline it will simply be
removed.
So, you cannot preserve spaces from the $(shell) command directly. That being said, make does allow multiline variables using define -- but beware, attempting to use such variables is problematic. Consider:
define x
foo
bar
endef
all:
#echo "$x"
Make expands the $x in place, and you end up with:
all:
#echo " foo
bar"
(where the newline is considered the end of the recipe line..).
Depending on what you want this for, you may be able to get around this is using a bash variable:
all:
#x=$$(cat file); \
echo $$x
Or potentially storing your output in a file, and referencing that when necessary.
all:
eval (cat file >> output.txt)
cat output.txt
(and yes, the last one is convoluted as written, but I'm not sure what you're trying to do, and this allows the output of your command to be persistent across recipe lines).
If the file contents are ensured not to contain any binary data, and if you're willing to to extra processing each time you access the variable, then you could:
foo:=$(shell cat file | tr '\n' '\1')
all:
#echo "$(shell echo "$(foo)" | tr '\1' '\n')"
Note that you cannot use nulls \0, and I suspect that probably means there's a buffer overflow bug in my copy of Make.
Related
I wrote in Makefile
define deploy_meta
$(shell git log -n 2 --oneline | awk '{print "commit"NR ": " $0}')
commit: nogit-$(timestamp)
tag: nogit-$(timestamp)
deployed-from: $(shell hostname)
deployed-by: $(USER)
deploy-date: $(shell date -u '+%Y%m%d%H%M%S')
endef
but if gives me
$cat .deploy
commit1: commit2:
commit: nogit-1669806282
tag: nogit-1669806282
...
Command itself
git log -n 2 --oneline | awk '{print "commit"NR ": " $0}'
works fine and gives two lines. It is evident, that make feels it, since it prints two "commit#" words. But it doesn't print content. Why?
The main problem is simple enough: Make expands $0 here (to nothing). You need $$0.
If you want two lines, though, $(shell) is also part of your problem here:
The shell function provides for make the same facility that backquotes ('`') provide in most shells: it does command expansion. This means that it takes as an argument a shell command and expands to the output of the command. The only processing make does on the result is to convert each newline (or carriage-return / newline pair) to a single space. If there is a trailing (carriage-return and) newline it will simply be removed.
(emphasis mine). You'll need a completely different approach, e.g., use $(shell) to produce a file, then include that file, for instance.
new_contents = "\$$(cooly)"
all:
mkdir -p subdir
echo $(new_contents) | sed -e 's/^ //' > subdir/makefile
#echo "---MAKEFILE CONTENTS---"
#cd subdir && cat makefile
#echo "---END MAKEFILE CONTENTS---"
#cd subdir && $(MAKE)
# Note that variables and exports. They are set/affected globally.
cooly = "The subdirectory can see me!"
export cooly
# This would nullify the line above: unexport cooly
clean:
rm -rf subdir
What I want is a "$(cooly)" string, not the variable value.
I tried several combinations:
new_contents = "$(cooly)", gives the variable value, The subdirectory can see me!
new_contents = "$$", gives $
new_contents = "\$(cooly)", gives Syntax error: Unterminated quoted string error
Why new_contents = "$$(cooly)" doesn't give "$(cooly)" but result in nothing?
"$$" -> "$", so why isn't "$$(cooly)" ---> "$(cooly)"?
You have to understand both how make expansion works, and how shell expansion works, in order to write more complicated recipes in make. That's because, make recipes are passed to the shell after make is done expanding them.
Make passes recipe lines to the shell virtually verbatim: there is only one character that's special (not counting backslash/newlines at the end) and that's $. If make sees a $ it will try to expand it as a variable reference. To avoid that, you have to escape it as $$ to hide it from make.
So let's look at your makefile:
cooly = "The subdirectory can see me!"
echo $(new_contents) ...
If new_contents is "$(cooly)", make sees the $(cooly) as a variable reference and expands it before it even invokes the shell. So first make expands $(new_contents) to "$(cooly)", then it expands that to ""The subdirectory can see me!"" (because the quotes are in both variables, and quotes are not special to make: they're just like any other character like a or b). The result will be:
echo ""The subdirectory can see me!""
The shell will toss the quotes since they're no-ops and echo that value (into the pipe).
If new_contents is "\$(cooly)", that backslash doesn't mean anything to make. Just like quotes, backslashes (unless they are at the end of a line) are not special to make. So make expands just as before, but this time the command it passes to the shell is this:
echo "\"The subdirectory can see me!""
backslashes are not special to make, but they are special to the shell. Here you've escaped the second quote so the shell doesn't treat it as a quote character, which means you have an odd number of quotes in your command, which is why you get an error from the shell about non-terminated quotes.
If new_contents is "$$(cooly)", make doesn't expand the variable, it is passed along to the shell like this:
echo "$(cooly)"
However, $ is also special to the shell. Putting it in double quotes doesn't prevent the shell from trying to expand it. This tells the shell to run the command cooly and substitute the output. Almost certainly there is no command named cooly and so you'll get an error message to stderr (maybe you didn't notice it) and the shell will replace it with nothing because it didn't print anything to stdout.
If new_contents is "\$$(cooly)" then make will not expand, and run this shell command:
echo "\$(cooly)"
The shell sees the backslash and doesn't expand the $ but instead uses it literally, and you get the result you want.
Here are some hints:
First, do not include quotes in your make variables (unless the variable contains an entire shell command and you need quotes inside it). Make doesn't care about quotes and having them embedded in variables makes it very difficult to reason about what the shell will see.
Include the quotes only in the recipe.
Second, remember that since make doesn't care about quotes, it doesn't have the same behavior as the shell WRT single vs. double quotes. You can use single quotes around make variables to reduce the need to escape things from the shell, without hiding them from make.
So, I would write this:
new_contents = $$(cooly)
cooly = The subdirectory can see me!
all:
mkdir -p subdir
echo '$(new_contents)' | sed -e 's/^ //' > subdir/makefile
...
BTW, it's never a good idea to add # values to your makefile until it's completely done and working. Seeing the output make prints (which is what it's sending to the shell) is a great help in figuring out whether your recipes are right, and whether the problem is with your make constructs or shell constructs.
Does anyone know how to use a here-document redirection on a recipe?
test:
sh <<EOF
echo I Need This
echo To Work
ls
EOF
I can't find any solution trying the usual backslash method (which basically ends with a command in a single line).
Rationale:
I have a set of multi-line recipes that I want to proxy through another command (e.g., sh, docker).
onelinerecipe := echo l1
define twolinerecipe :=
echo l1
echo l2
endef
define threelinerecipe :=
echo l1
echo l2
echo l3
endef
# sh as proxy command and proof of concept
proxy := sh
test1:
$(proxy) <<EOF
$(onelinerecipe)
EOF
test2:
$(proxy) <<EOF
$(twolinerecipe)
EOF
test3:
$(proxy) <<EOF
$(threelinerecipe)
EOF
The solution I would love to avoid: transform multiline macros into single lines.
define threelinerecipe :=
echo l1;
echo l2;
echo l3
endef
test3:
$(proxy) <<< "$(strip $(threelinerecipe))"
This works (I use gmake 4.0 and bash as make's shell) but it requires changing my recipes and I have a lot.
Strip removes the newlines, from the macro, then everything is written in a single line.
My end goal is: proxy := docker run ...
Using the line .ONESHELL: somewhere in your Makefile will send all recipe lines to a single shell invocation, you should find your original Makefile works as expected.
When make sees a multi-line block in a recipe
(i.e., a block of lines all ending in \, apart from the last),
it passes that block un-modifed to the shell.
This generally works in bash,
apart from here docs.
One way around this is to strip any trailing \s,
then pass the resulting string to bash's eval.
You do this in make by playing with ${.SHELLFLAGS} and ${SHELL}.
You can use both of these in target-specific form if you only want it to kick in for a few targets.
.PHONY: heredoc
heredoc: .SHELLFLAGS = -c eval
heredoc: SHELL = bash -c 'eval "$${#//\\\\/}"'
heredoc:
#echo First
#cat <<-there \
here line1 \
here anotherline \
there
#echo Last
giving
$ make
First
here line1
here anotherline
Last
Careful with that quoting, Eugene.
Note the cheat here:
I am removing all backslashes,
not just the ones at the ends of the line.
YMMV.
With GNU make, you can combine multi-line variables with the export directive to use a multi-line command without having to turn on .ONESHELL globally:
define script
cat <<'EOF'
here document in multi-line shell snippet
called from the "$#" target
EOF
endef
export script
run:; # eval "$$script"
will give
here document in multi-line shell snippet
called from the "run" target
You can also combine it with the value function to prevent its value from being expanded by make:
define _script
cat <<EOF
SHELL var expanded by the shell to $SHELL, pid is $$
EOF
endef
export script = $(value _script)
run:; # eval "$$script"
will give
SHELL var expanded by the shell to /bin/sh, pid is 12712
Not a here doc but this might be a useful workaround.
And it doesn’t require any GNU Make’isms.
Put the lines in a subshell with parens, prepend each line with echo.
You’ll need trailing sloshes and semi-colon and slosh where appropriate.
test:
( \
echo echo I Need This ;\
echo echo To Work ;\
echo ls \
) \
| sh
I want to perform a string substitution in my Makefile. I can easily do this with a string literal like so:
foo:
echo $(subst /,-,"hello/world")
Which yields the expected:
hello-world
But when I switch to using a variable, I can't seem to get the substitution to stick:
foo:
x="hello/world" ; \
echo $(subst /,-,$$x)
Instead of replacing the slash with a dash, I still get the original string printed back. Can someone explain what is going on here? Does the variable need to be explicitly converted to a string literal or something?
UPDATE:
The fix based on MadScientist's answer--this will allow me to reference the modified string as a variable.
foo:
x="hello/world" ; \
y=`echo $$x | tr / -` ; \
echo $$y
But instead of echo $$y this could be something more useful.
You can't combine make functions with shell variables... all make functions are expanded first, then the resulting script is passed to the shell to be run. When the shell gets the script there are no more make functions in it (and if there were, the shell wouldn't know what to do with them!)
Your subst is running on the literal string $x, which has no / so nothing to replace and results in $x, which the shell expands to the string hello/world.
If you have to work on a shell variable value you must use shell commands such as sed or tr, not make's subst function:
foo:
x="hello/world" ; \
echo $$x | tr / -
You could define x as a make variable:
Makefile:
x = foo bar baz
t:
#echo $(subst bar,qux,$(x))
Output:
make
foo qux baz
Version:
make --version
GNU Make 3.81
The following command prints the absolute path of a particular C++ header, according to where g++ believes it to be.
echo \#include\<ham/hamsterdb.h\> | g++ -M -x c++-header - | grep hamsterdb.hpp | sed -e 's/-: //' -e 's/ \\//'
On my system, this outputs: /usr/local/include/ham/hamsterdb.hpp
I run into a problem when trying to run this inside a Makefile in order to set a variable:
FILE=$(shell echo \#include\<ham/hamsterdb.h\> | g++ -M -x c++-header - | grep hamsterdb.hpp | sed -e 's/-: //' -e 's/ \\//')
.PHONY spec
spec:
#echo $(FILE)
This outputs a new line. I think it's the hash ('#') character which is messing with make; if I rewrite the FILE=... line like this:
FILE=$(shell echo \#include\<ham/hamsterdb.h\>)
the output is still nothing.
You have to escape the hash twice in order to use it inside functions: once for Make and once again for the shell.
That is,
FILE = $(shell echo \\\#include\ \<ham/hamsterdb.h\>)
Notice three backslashes instead of two as one could expect. The third backslash is needed because otherwise the first one would escape the second and the hash would be still unescaped.
UPD.
Another possible solution is to only escape the hash for Make and use Bash single quotes to prevent interpreting the hash as a shell comment. This also eliminates the need of escaping spaces, < and >:
FILE = $(shell echo '\#include <ham/hamsterdb.h>')
You need just a bit more quoting:
FILE=$(shell echo \\\#include\<ham/hamsterdb.h\> ...
^^^
Quote once for make itself, a second time for the shell (the shell needs to 'see' \#).