Related
Makefile:
.PHONY: all
SHELL:=/usr/bin/env bash
all:
$(eval x=$(shell cat file))
#echo "$x"
File:
foo
bar
Output:
foo bar
How do I get the contents of the file into the make variable without losing the newlines?
You can't do this with shell, as described in its documentation.
If you have a sufficiently new version of GNU make, you can use the file function however.
Make converts newlines from shell outputs to spaces (see here):
The shell function performs the same function that backquotes (‘`’)
perform in most shells: it does command expansion. This means that it
takes as an argument a shell command and evaluates to the output of
the command. The only processing make does on the result is to convert
each newline (or carriage-return / newline pair) to a single space. If
there is a trailing (carriage-return and) newline it will simply be
removed.
So, you cannot preserve spaces from the $(shell) command directly. That being said, make does allow multiline variables using define -- but beware, attempting to use such variables is problematic. Consider:
define x
foo
bar
endef
all:
#echo "$x"
Make expands the $x in place, and you end up with:
all:
#echo " foo
bar"
(where the newline is considered the end of the recipe line..).
Depending on what you want this for, you may be able to get around this is using a bash variable:
all:
#x=$$(cat file); \
echo $$x
Or potentially storing your output in a file, and referencing that when necessary.
all:
eval (cat file >> output.txt)
cat output.txt
(and yes, the last one is convoluted as written, but I'm not sure what you're trying to do, and this allows the output of your command to be persistent across recipe lines).
If the file contents are ensured not to contain any binary data, and if you're willing to to extra processing each time you access the variable, then you could:
foo:=$(shell cat file | tr '\n' '\1')
all:
#echo "$(shell echo "$(foo)" | tr '\1' '\n')"
Note that you cannot use nulls \0, and I suspect that probably means there's a buffer overflow bug in my copy of Make.
Does anyone know how to use a here-document redirection on a recipe?
test:
sh <<EOF
echo I Need This
echo To Work
ls
EOF
I can't find any solution trying the usual backslash method (which basically ends with a command in a single line).
Rationale:
I have a set of multi-line recipes that I want to proxy through another command (e.g., sh, docker).
onelinerecipe := echo l1
define twolinerecipe :=
echo l1
echo l2
endef
define threelinerecipe :=
echo l1
echo l2
echo l3
endef
# sh as proxy command and proof of concept
proxy := sh
test1:
$(proxy) <<EOF
$(onelinerecipe)
EOF
test2:
$(proxy) <<EOF
$(twolinerecipe)
EOF
test3:
$(proxy) <<EOF
$(threelinerecipe)
EOF
The solution I would love to avoid: transform multiline macros into single lines.
define threelinerecipe :=
echo l1;
echo l2;
echo l3
endef
test3:
$(proxy) <<< "$(strip $(threelinerecipe))"
This works (I use gmake 4.0 and bash as make's shell) but it requires changing my recipes and I have a lot.
Strip removes the newlines, from the macro, then everything is written in a single line.
My end goal is: proxy := docker run ...
Using the line .ONESHELL: somewhere in your Makefile will send all recipe lines to a single shell invocation, you should find your original Makefile works as expected.
When make sees a multi-line block in a recipe
(i.e., a block of lines all ending in \, apart from the last),
it passes that block un-modifed to the shell.
This generally works in bash,
apart from here docs.
One way around this is to strip any trailing \s,
then pass the resulting string to bash's eval.
You do this in make by playing with ${.SHELLFLAGS} and ${SHELL}.
You can use both of these in target-specific form if you only want it to kick in for a few targets.
.PHONY: heredoc
heredoc: .SHELLFLAGS = -c eval
heredoc: SHELL = bash -c 'eval "$${#//\\\\/}"'
heredoc:
#echo First
#cat <<-there \
here line1 \
here anotherline \
there
#echo Last
giving
$ make
First
here line1
here anotherline
Last
Careful with that quoting, Eugene.
Note the cheat here:
I am removing all backslashes,
not just the ones at the ends of the line.
YMMV.
With GNU make, you can combine multi-line variables with the export directive to use a multi-line command without having to turn on .ONESHELL globally:
define script
cat <<'EOF'
here document in multi-line shell snippet
called from the "$#" target
EOF
endef
export script
run:; # eval "$$script"
will give
here document in multi-line shell snippet
called from the "run" target
You can also combine it with the value function to prevent its value from being expanded by make:
define _script
cat <<EOF
SHELL var expanded by the shell to $SHELL, pid is $$
EOF
endef
export script = $(value _script)
run:; # eval "$$script"
will give
SHELL var expanded by the shell to /bin/sh, pid is 12712
Not a here doc but this might be a useful workaround.
And it doesn’t require any GNU Make’isms.
Put the lines in a subshell with parens, prepend each line with echo.
You’ll need trailing sloshes and semi-colon and slosh where appropriate.
test:
( \
echo echo I Need This ;\
echo echo To Work ;\
echo ls \
) \
| sh
I have a directory containing several files, some of which have spaces in their names:
Test workspace/
Another directory/
file1.ext
file2.ext
demo 2012-03-23.odp
I use GNU's $(wildcard) command on this directory, and then iterate over the result using $(foreach), printing everything out. Here's the code:
FOO := $(wildcard *)
$(info FOO = $(FOO))
$(foreach PLACE,$(FOO),$(info PLACE = $(PLACE)))
Here's what I would expect to see printed out:
Test workspace
Another directory
file1.ext
file2.ext
demo 2012-03-23.odp
Here's what I would actually get:
Test
workspace
Another
directory
file1.ext
file2.ext
demo
2012-03-23.odp
The latter is obviously of no use to me. The documentation for $(wildcard) flat-out states that it returns a "space-separated list of names" but completely fails to acknowledge the huge problems this raises. Nor does the documentation for $(foreach).
Is it possible to work around this? If so, how? Renaming every file and directory to remove the spaces is not an option.
The bug #712 suggests that make does not handle names with spaces. Nowhere, never.
I found a blog post saying it's partially implemented by escaping the spaces with \ (\\ seems to be typo or formatting artefact), but:
It does not work in any functions except $(wildcard).
It does not work when expanding lists of names from variables, which includes the special variables $?, $^ and $+ as well as any user-defined variable. Which in turn means that while $(wildcard) will match correct files, you won't be able to interpret the result anyway.
So with explicit or very simple pattern rules you can get it to work, but beyond that you are out of luck. You'll have to look for some other build system that does support spaces. I am not sure whether jam/bjam does, scons, waf, ant, nant and msbuild all should work.
GNU Make does very poorly with space-separated filenames.
Spaces are used as delimiters in word list all over the place.
This blog post summarizes the situation well, but WARNING: it incorrectly uses \\ rather than \
target: some\ file some\ other\ file
some\ file some\ other\ file:
echo done
You can also use variables, so this would also work
VAR := some\ file some\ other\ file
target: $(VAR)
$(VAR):
echo done
Only the wildcard function recognizes the escaping, so you can't do anything fancy without lots of pain.
But don't forget that your shell uses spaces as delimiters too.
If I wanted to change the echo done to touch $#, I'd have to add slash to escape it for my shell.
VAR := some\ file
target: $(VAR)
$(VAR):
touch $(subst \,\\,$#)
or, more likely, use quotes
VAR := some\ file some\ other\ file
target: $(VAR)
$(VAR):
touch '$#'
In the end, if you want to avoid a lot of pain, both in GNU make, and in your shell, don't put spaces in your filenames. If you do, hopefully the limited capabilities of Make will be sufficient.
This method will also allow use of listed file names such as $? and user variables that are lists of files.
The best way to deal with spaces in Make is to substitute spaces for other characters.
s+ = $(subst \ ,+,$1)
+s = $(subst +,\ ,$1)
$(call s+,foo bar): $(call s+,bar baz) $(call s+,bar\ baz2)
# Will also shows list of dependencies with spaces.
#echo Making $(call +s,$#) from $(call +s,$?)
$(call s+,bar\ baz):
#echo Making $(call +s,$#)
$(call s+,bar\ baz2):
#echo Making $(call +s,$#)
Outputs
Making bar baz
Making bar baz2
Making foo bar from bar baz bar baz2
You can then safely manipulate lists of file names using all the GNU Make
functions. Just be sure to remove the +'s before using these names in a rule.
SRCS := a\ b.c c\ d.c e\ f.c
SRCS := $(call s+,$(SRCS))
# Can manipulate list with substituted spaces
OBJS := $(SRCS:.c=.o)
# Rule that has object files as dependencies.
exampleRule:$(call +s,$(OBJS))
# You can now use the list of OBJS (spaces are converted back).
#echo Object files: $(call +s,$(OBJS))
a\ b.o:
# a b.o rule commands go here...
#echo in rule: a b.o
c\ d.o:
e\ f.o:
Outputs
in rule: a b.o
Object files: a b.o c d.o e f.o
This info is all from the blog that everyone else was posting.
Most people seem to be recommending using no spaces in paths or using Windows 8.3 paths, but if you must use spaces, escaping spaces and substitution works.
If you are willing to rely on your shell a bit more, this gives a list which can hold names with spaces just fine:
$(shell find | sed 's: :\\ :g')
The original question said that "renaming is not an option", yet many commenters have pointed out that renaming is pretty much the only way Make can handle spaces. I suggest a middle way: Use Make to temporarily rename the files and then rename them back. This gives you all the power of Make with implicit rules and other goodness, but doesn't mess up your file naming scheme.
# Make cannot handle spaces in filenames, so temporarily rename them
nospaces:
rename -v 's/ /%20/g' *\ *
# After Make is done, rename files back to having spaces
yesspaces:
rename -v 's/%20/ /g' *%20*
You could call these targets by hand with make nospaces and make yesspaces, or you can have other targets depends on them. For example, you might want to have a "push" target which makes sure to put the spaces back in filenames before syncing files back with a server:
# Put spaces back in filenames before uploading
push: yesspaces
git push
[Sidenote: I tried the answer which suggested using +s and s+ but it made my Makefile harder to read and debug. I gave up on it when it gave me guff over implicit rules likes: %.wav : %.ogg ; oggdec "$<".]
package_version := $(version)x0d$(date)
what is the x0d part between version and date vars? is it just string?
What $(dotin_files:.in=) does below
code
dotin_files := $(shell find . -type f -name \*.in)
dotin_files := $(dotin_files:.in=)
what this means $(dotin_files:=.in)
code
$(dotin_files): $(dotin_files:=.in)
$(substitute) $#.in > $#
can target contain multiple files?
what is the meaning of declaring target variable as PHONY?
code
.PHONY: $(dotin_files)
In the regex replacement code below
code
substitute := perl -p -e 's/#([^#]+)#/defined $$ENV{$$1} ? $$ENV{$$1} : $$&/ge'
what are $$ENV{$$1} and $$&? I guess it's Perl scope...
thanks for your time
Variable Expansion
$() is variable expansion in make, this should just be string substitution - if your makefile is
version=1
date=1.1.10
package_version:=$(version)x0d$(date)
then the variable package_version will expand to 1x0d1.1.10.
Substitution
The syntax $(var:a=b) is a substitution reference and will expand to var with a suffix a substituted with b.
For example, in
foobar:= foo bar
faabar:=$(foobar:oo=aa)
$(faabar) will expand to the string faa bar.
Multiple Targets
Multiple targets in a make rule is equivalent to having n rules with a single target, eg
foo bar:foo.c bar.c
$(CC) -o $# $^
is equivalent to
foo:foo.c bar.c
$(CC) -o $# $^
bar:foo.c bar.c
$(CC) -o $# $^
remember that any variables here are expanded.
Phony Targets
The .PHONY target declares that a rule doesn't produce an actual file, so it will always be built. As always, variables are expanded first. In your case this will expand to something like
.PHONY: foo bar
Escaping
A dollar sign is an escape character in makefiles, the $$ in your perl example is a literal $, eg substitute will be the string
perl -p -e 's/#([^#]+)#/defined $ENV{$1} ? $ENV{$1} : $&/ge'
The dollar signs here are processed by perl, and probably give environment variables (I don't know perl).
x0d part between version and date vars, is it just string?
Yes.
What $(dotin_files:.in=) does below
Removes the .in from the filenames found with the shell find.
what this means $(dotin_files:=.in)
I think you meant $(dotin_files:.in=). As already answered, within the variable dotin_files it replaces any occurrence of ".in" with an empty string(the part between the "=" and ")".
can target contain multiple files?
Yes
what is the meaning of declaring target variable as PHONY?
make will ignore targets time-stamp and consider them as new
thus rebuilding them each time.
In the regex replacement code below what are $$ENV{$$1} and $$&?
To avoid expansion of $ENV, the $ is doubled, think of '%' in C format strings, thus the string
perl -p -e 's/#([^#]+)#/defined $$ENV{$$1} ? $$ENV{$$1} : $$&/ge'
when called as a shell command will become:
perl -p -e 's/#([^#]+)#/defined $ENV{$1} ? $ENV{$1} : $&/ge'
$ENV is the perl Environment hash, $1 I think it's a backreference in the s/// regexp group.
Michael, you've been asking a lot of basic Makefile questions, and the ones you're asking now are ones you should be able to answer for yourself by experiment.
can target contain multiple files?
Try it:
dotin_files := foo.in bar.in
$(dotin_files):
#echo $#
Now try make foo.in and make bar.in. What happens?
What $(dotin_files:.in=) does
It's a substitution reference. Try it yourself and see what happens, like this:
dotin_files := foo.in bar.in
dotin_files := $(dotin_files:.in=)
all:
#echo $(dotin_files)
What did it do?
substitute := perl -p -e 's/#([^#]+)#/defined $$ENV{$$1} ? $$ENV{$$1} : $$&/ge'
what are $$ENV{$$1} and $$&? I guess it's Perl scope...
Let's take a look:
all:
#echo $(substitute)
If you want to know more about Perl, or regexs, or find, or make, or whatever, feel free to ask here, but please take a little time to try to figure it out first.
I am reading the document of GNU Make. Here is an example
%.d: %.c
#set -e; rm -f $#; \
$(CC) -M $(CPPFLAGS) $< > $#.$$$$; \
sed ’s,\($*\)\.o[ :]*,\1.o $# : ,g’ < $#.$$$$ > $#; \
rm -f $#.$$$$
I tried this on a C++ program, and got the list of files
init3d.d init3d.d.18449 input.d input.d.18444 main.d main.d.18439
Here is what I found but don't understand in the same document
If you have enabled secondary expansion and you want a literal dollar sign in the prerequisites list, you must actually write four dollar signs (‘$$$$’).
I wonder what the four dollar signs "$$$$" mean actually? How do they 18449, 18444 or 18439?
Thanks and regards!
If make "secondary expansion" is enabled, $$$$ is required in order to generate a single $ in the actual output. $ is normally used to expand variables, call make functions, etc. $$ with secondary expansion enabled does something else, but otherwise it generates an actual $ in the output.
The shell that make uses to execute command-lines on Unix-like systems normally interprets $$ as expand to shell process ID. So, without secondary expansion enabled, $$$$ will turn into $$ in the output, which the shell will expand to the process ID.
(Using the shell process ID as a suffix is a simple way of trying to guarantee uniqueness of file name for a temporary file.)
$$ will be converted to $, but in Makefile rules (which are shell expressions) you'll have to also escape the resulting $ using a \ or by using single quotes ' around your expression.
Here is an example that demonstrates it:
DOLLAR:=$$
dollar:
echo '$$' > $#
echo "\$$" >> $#
echo '$(DOLLAR)' >> $#
echo "\$(DOLLAR)" >> $#
cat dollar
18449, 18444 or 18439 look like process ids so maybe a process ID?