I'm looking to record the exact commands used to build artifacts within a makefile. I'd like this to be stored in a file for later consumption. I am running into issues due to quotes. Basically, what I want is:
define record_and_run_recipe
#echo '$(2)' > $1
$2
endef
all:
$(call record_and_run_recipe,out.cmd,\
#echo 'hello world "$$1"' )
cat out.cmd
I would like this to output (exactly)
#echo 'hello world "$1"'
Of course, the quotes end up matching with the quotes in the expansion of the variable, and this messes everything up. (I get #echo hello world instead). Bash doesn't like '\'' either, so I can't simply do $(2:'=\'). I also seem to have issues with , characters...
I'm not looking to debug the entire makefile, just dump a couple of recipes. I'm wondering if anyone has a robust way of accomplishing this.
As I said in my comment above, you can use GNU make's $(info ...) function. It's not exactly clear from your example above what you want to do; why are you trying to put the output into a file, then cat it? Is that important?
If you can't use info, the canonical way to handle quoting in shell is to surround the string with single quotes, then replace every single quote with '\''. You say "bash doesn't like" that, but I don't know what that means. Normally you'd do something like:
define record_and_run_recipe
#echo '$(subst ','\'',$2)' > $1
$2
endef
As far as commas you will absolutely have a problem with commas if you want to use the $(call ...) function. The only way to avoid that is to put the string into a variable, like:
output = foo, bar
... $(call blah,$(output))
to "hide" the comma from call.
Related
I am learning Gnu Make and don't understand the construction. Actually wondering what parameters are in quotes? This entry is the first in the Makefile.
LIBLORAGW_VERSION := `cat ../VERSION`
Backticks ` ` are shell syntax to return enclosed command output as a string.
So that, this line executes cat ../VERSION, grabs its output and assigns it to LIBLORAGW_VERSION
Upd: Yes, the explanation above is a simplified "it looks like" but is not correct in details. See MadScientist's comment and try this
DATE=`date`
all:
$(info DATE is $(DATE))
#echo DATE is $(DATE)
new_contents = "\$$(cooly)"
all:
mkdir -p subdir
echo $(new_contents) | sed -e 's/^ //' > subdir/makefile
#echo "---MAKEFILE CONTENTS---"
#cd subdir && cat makefile
#echo "---END MAKEFILE CONTENTS---"
#cd subdir && $(MAKE)
# Note that variables and exports. They are set/affected globally.
cooly = "The subdirectory can see me!"
export cooly
# This would nullify the line above: unexport cooly
clean:
rm -rf subdir
What I want is a "$(cooly)" string, not the variable value.
I tried several combinations:
new_contents = "$(cooly)", gives the variable value, The subdirectory can see me!
new_contents = "$$", gives $
new_contents = "\$(cooly)", gives Syntax error: Unterminated quoted string error
Why new_contents = "$$(cooly)" doesn't give "$(cooly)" but result in nothing?
"$$" -> "$", so why isn't "$$(cooly)" ---> "$(cooly)"?
You have to understand both how make expansion works, and how shell expansion works, in order to write more complicated recipes in make. That's because, make recipes are passed to the shell after make is done expanding them.
Make passes recipe lines to the shell virtually verbatim: there is only one character that's special (not counting backslash/newlines at the end) and that's $. If make sees a $ it will try to expand it as a variable reference. To avoid that, you have to escape it as $$ to hide it from make.
So let's look at your makefile:
cooly = "The subdirectory can see me!"
echo $(new_contents) ...
If new_contents is "$(cooly)", make sees the $(cooly) as a variable reference and expands it before it even invokes the shell. So first make expands $(new_contents) to "$(cooly)", then it expands that to ""The subdirectory can see me!"" (because the quotes are in both variables, and quotes are not special to make: they're just like any other character like a or b). The result will be:
echo ""The subdirectory can see me!""
The shell will toss the quotes since they're no-ops and echo that value (into the pipe).
If new_contents is "\$(cooly)", that backslash doesn't mean anything to make. Just like quotes, backslashes (unless they are at the end of a line) are not special to make. So make expands just as before, but this time the command it passes to the shell is this:
echo "\"The subdirectory can see me!""
backslashes are not special to make, but they are special to the shell. Here you've escaped the second quote so the shell doesn't treat it as a quote character, which means you have an odd number of quotes in your command, which is why you get an error from the shell about non-terminated quotes.
If new_contents is "$$(cooly)", make doesn't expand the variable, it is passed along to the shell like this:
echo "$(cooly)"
However, $ is also special to the shell. Putting it in double quotes doesn't prevent the shell from trying to expand it. This tells the shell to run the command cooly and substitute the output. Almost certainly there is no command named cooly and so you'll get an error message to stderr (maybe you didn't notice it) and the shell will replace it with nothing because it didn't print anything to stdout.
If new_contents is "\$$(cooly)" then make will not expand, and run this shell command:
echo "\$(cooly)"
The shell sees the backslash and doesn't expand the $ but instead uses it literally, and you get the result you want.
Here are some hints:
First, do not include quotes in your make variables (unless the variable contains an entire shell command and you need quotes inside it). Make doesn't care about quotes and having them embedded in variables makes it very difficult to reason about what the shell will see.
Include the quotes only in the recipe.
Second, remember that since make doesn't care about quotes, it doesn't have the same behavior as the shell WRT single vs. double quotes. You can use single quotes around make variables to reduce the need to escape things from the shell, without hiding them from make.
So, I would write this:
new_contents = $$(cooly)
cooly = The subdirectory can see me!
all:
mkdir -p subdir
echo '$(new_contents)' | sed -e 's/^ //' > subdir/makefile
...
BTW, it's never a good idea to add # values to your makefile until it's completely done and working. Seeing the output make prints (which is what it's sending to the shell) is a great help in figuring out whether your recipes are right, and whether the problem is with your make constructs or shell constructs.
Makefile:
.PHONY: all
SHELL:=/usr/bin/env bash
all:
$(eval x=$(shell cat file))
#echo "$x"
File:
foo
bar
Output:
foo bar
How do I get the contents of the file into the make variable without losing the newlines?
You can't do this with shell, as described in its documentation.
If you have a sufficiently new version of GNU make, you can use the file function however.
Make converts newlines from shell outputs to spaces (see here):
The shell function performs the same function that backquotes (‘`’)
perform in most shells: it does command expansion. This means that it
takes as an argument a shell command and evaluates to the output of
the command. The only processing make does on the result is to convert
each newline (or carriage-return / newline pair) to a single space. If
there is a trailing (carriage-return and) newline it will simply be
removed.
So, you cannot preserve spaces from the $(shell) command directly. That being said, make does allow multiline variables using define -- but beware, attempting to use such variables is problematic. Consider:
define x
foo
bar
endef
all:
#echo "$x"
Make expands the $x in place, and you end up with:
all:
#echo " foo
bar"
(where the newline is considered the end of the recipe line..).
Depending on what you want this for, you may be able to get around this is using a bash variable:
all:
#x=$$(cat file); \
echo $$x
Or potentially storing your output in a file, and referencing that when necessary.
all:
eval (cat file >> output.txt)
cat output.txt
(and yes, the last one is convoluted as written, but I'm not sure what you're trying to do, and this allows the output of your command to be persistent across recipe lines).
If the file contents are ensured not to contain any binary data, and if you're willing to to extra processing each time you access the variable, then you could:
foo:=$(shell cat file | tr '\n' '\1')
all:
#echo "$(shell echo "$(foo)" | tr '\1' '\n')"
Note that you cannot use nulls \0, and I suspect that probably means there's a buffer overflow bug in my copy of Make.
I have a directory containing several files, some of which have spaces in their names:
Test workspace/
Another directory/
file1.ext
file2.ext
demo 2012-03-23.odp
I use GNU's $(wildcard) command on this directory, and then iterate over the result using $(foreach), printing everything out. Here's the code:
FOO := $(wildcard *)
$(info FOO = $(FOO))
$(foreach PLACE,$(FOO),$(info PLACE = $(PLACE)))
Here's what I would expect to see printed out:
Test workspace
Another directory
file1.ext
file2.ext
demo 2012-03-23.odp
Here's what I would actually get:
Test
workspace
Another
directory
file1.ext
file2.ext
demo
2012-03-23.odp
The latter is obviously of no use to me. The documentation for $(wildcard) flat-out states that it returns a "space-separated list of names" but completely fails to acknowledge the huge problems this raises. Nor does the documentation for $(foreach).
Is it possible to work around this? If so, how? Renaming every file and directory to remove the spaces is not an option.
The bug #712 suggests that make does not handle names with spaces. Nowhere, never.
I found a blog post saying it's partially implemented by escaping the spaces with \ (\\ seems to be typo or formatting artefact), but:
It does not work in any functions except $(wildcard).
It does not work when expanding lists of names from variables, which includes the special variables $?, $^ and $+ as well as any user-defined variable. Which in turn means that while $(wildcard) will match correct files, you won't be able to interpret the result anyway.
So with explicit or very simple pattern rules you can get it to work, but beyond that you are out of luck. You'll have to look for some other build system that does support spaces. I am not sure whether jam/bjam does, scons, waf, ant, nant and msbuild all should work.
GNU Make does very poorly with space-separated filenames.
Spaces are used as delimiters in word list all over the place.
This blog post summarizes the situation well, but WARNING: it incorrectly uses \\ rather than \
target: some\ file some\ other\ file
some\ file some\ other\ file:
echo done
You can also use variables, so this would also work
VAR := some\ file some\ other\ file
target: $(VAR)
$(VAR):
echo done
Only the wildcard function recognizes the escaping, so you can't do anything fancy without lots of pain.
But don't forget that your shell uses spaces as delimiters too.
If I wanted to change the echo done to touch $#, I'd have to add slash to escape it for my shell.
VAR := some\ file
target: $(VAR)
$(VAR):
touch $(subst \,\\,$#)
or, more likely, use quotes
VAR := some\ file some\ other\ file
target: $(VAR)
$(VAR):
touch '$#'
In the end, if you want to avoid a lot of pain, both in GNU make, and in your shell, don't put spaces in your filenames. If you do, hopefully the limited capabilities of Make will be sufficient.
This method will also allow use of listed file names such as $? and user variables that are lists of files.
The best way to deal with spaces in Make is to substitute spaces for other characters.
s+ = $(subst \ ,+,$1)
+s = $(subst +,\ ,$1)
$(call s+,foo bar): $(call s+,bar baz) $(call s+,bar\ baz2)
# Will also shows list of dependencies with spaces.
#echo Making $(call +s,$#) from $(call +s,$?)
$(call s+,bar\ baz):
#echo Making $(call +s,$#)
$(call s+,bar\ baz2):
#echo Making $(call +s,$#)
Outputs
Making bar baz
Making bar baz2
Making foo bar from bar baz bar baz2
You can then safely manipulate lists of file names using all the GNU Make
functions. Just be sure to remove the +'s before using these names in a rule.
SRCS := a\ b.c c\ d.c e\ f.c
SRCS := $(call s+,$(SRCS))
# Can manipulate list with substituted spaces
OBJS := $(SRCS:.c=.o)
# Rule that has object files as dependencies.
exampleRule:$(call +s,$(OBJS))
# You can now use the list of OBJS (spaces are converted back).
#echo Object files: $(call +s,$(OBJS))
a\ b.o:
# a b.o rule commands go here...
#echo in rule: a b.o
c\ d.o:
e\ f.o:
Outputs
in rule: a b.o
Object files: a b.o c d.o e f.o
This info is all from the blog that everyone else was posting.
Most people seem to be recommending using no spaces in paths or using Windows 8.3 paths, but if you must use spaces, escaping spaces and substitution works.
If you are willing to rely on your shell a bit more, this gives a list which can hold names with spaces just fine:
$(shell find | sed 's: :\\ :g')
The original question said that "renaming is not an option", yet many commenters have pointed out that renaming is pretty much the only way Make can handle spaces. I suggest a middle way: Use Make to temporarily rename the files and then rename them back. This gives you all the power of Make with implicit rules and other goodness, but doesn't mess up your file naming scheme.
# Make cannot handle spaces in filenames, so temporarily rename them
nospaces:
rename -v 's/ /%20/g' *\ *
# After Make is done, rename files back to having spaces
yesspaces:
rename -v 's/%20/ /g' *%20*
You could call these targets by hand with make nospaces and make yesspaces, or you can have other targets depends on them. For example, you might want to have a "push" target which makes sure to put the spaces back in filenames before syncing files back with a server:
# Put spaces back in filenames before uploading
push: yesspaces
git push
[Sidenote: I tried the answer which suggested using +s and s+ but it made my Makefile harder to read and debug. I gave up on it when it gave me guff over implicit rules likes: %.wav : %.ogg ; oggdec "$<".]
I'm dynamically generating config.mk with a bash script which will be used by a Makefile. The file is constructed with:
cat > config.mk <<CFG
SOMEVAR := $value_from_bash1
ANOTHER := $value_from_bash2
CFG
How do I ensure that the generated file really contains the contents of $value_from_bash*, and not something expanded / interpreted? I probably need to escape $ to $$ and \ to \\, but are there other characters that needs to be escaped? Perhaps there is a special literal assignment I've not heard of?
Spaces seems to be troublesome too:
$ ls -1
a b
a
$ cat Makefile
f := a b
default_target:
echo "$(firstword $(wildcard ${f}))"
$ make
a
If I use f := a\ b it works (using quotes like f := 'a b' did not work either, makefile just treats it as a regular character)
Okay, it turned out that Makefiles need little escaping for itself, but the commands which are executed by the shell interpreter need to be escaped.
Characters which have a special meaning in Makefile and that need to be escaped are:
sharp (#, comment) becomes \#
dollar ($, begin of variable) becomes $$
Newlines cannot be inserted in a variable, but to avoid breaking the rest of the Makefile, prepend it with a backslash so the line break will be ignored.
Too bad a backslash itself cannot be escaped (\\ will still be \\ and not \ as you might expect). This makes it not possible to put a literal slash on the end of a string as it will either eat the newline or the hash of a following comment. A space can be put on the end of the line, but that'll also be put in the variable itself.
The recipe itself is interpreted as a shell command, without any fancy escaping, so you've to escape data yourself, just imagine that you're writing a shellscript and inserting the variables from other files. The strategy here would be putting the variables between single quotes and escape only ' with '\'' (close the string, insert a literal ' and start a new string). Example: mornin' all becomes 'morning'\'' all' which is equivalent to "morning' all".
The firstword+wildcard issue is caused by the fact that filenames with spaces in them are treated as separate filenames by firstword. Furthermore, wildcard expands escapes using \ so x\ y is matches as one word, x y and not two words.
It seems that the full answer to this question is found nowhere on the internet, so I finally sat down and figured it out for the Windows case.
Specifically, the "Windows case" refers to file names that are valid in Windows, meaning that they do not contain the characters \, /, *, ?, ", ^, <, >, |, or line breaks. It also means \ and / are both considered valid directory separators for the purposes of Make.
An example will clear it up better than I can explain. Basically, if you are trying to match this file path:
Child\a$b {'}(a.o#$#,&+=~`),[].c
Then you have to write these rules:
all: Child\\a$$b\\\ \\\ {'}(a.o\#$$#,&+=~`),[].o
%.o: %.c
$(CC) '$(subst ','"'"',$(subst \,,$(subst \\,/,$+)))'
Stare at it for a long time and it'll sort of start making some remote sense.
This works in my MSYS2 environment, so I presume it is correct.
I don't see how that makefile can work as you say. A pattern rule cannot be the default.
You're missing a `$` in `$(wildcard ...)`, so I think you haven't posted what you're really testing.
You should escape newlines too.