I have a Makefile which has this line :
$(CROSS_COMPILE)$(CC) $(DLFLAGS) $(LIBPATH) -o $(OUTP)/$(SONAME) *.o $(LIBINCL)
This *.o expansion sometimes looks like 1.0 2.o 3.o 4.o. However, other times it can be 2.o 1.o 4.o 3.o (and other combinations). This causes the resulting shared object to have different checksums.
As a limited workaround, in some cases we change the line above to this :
$(CROSS_COMPILE)$(CC) $(DLFLAGS) $(LIBPATH) -o $(OUTP)/$(SONAME) $(sort $(wildcard *.o)) $(LIBINCL)
However, I can't do this fix for every source package. What I would like is for the shell (bash) to perform the glob substitution based on a sort of the filename, so that the '*.o' glob substitution above is consistent between machines and builds.
Any ideas ? Can I tell bash to (by default) change how it does globbing to accomplish what I want ?
I don't think it is a good idea to count on wildcard expansion to return filenames in a specific order -- but if you know that the filenames will be in a certain numeric interval, why not use brace expansion (https://www.gnu.org/software/bash/manual/bashref.html#Brace-Expansion)?
$ echo {1..4}.o
1.o 2.o 3.o 4.o
Related
Some make variables have an effect on the generation of the targets and thus one may want to rebuild if their value changes, for instance because they were specified explicitly on the command line. With GNU Make, it is relatively easy to do. For instance one can do this:
CHECK_CFLAGS:=.last-cflags.$(shell echo $(CFLAGS) | md5sum | awk '{ print $$1 }')
foo.o: foo.c $(CHECK_CFLAGS)
.last-cflags.%:
-rm .last-cflags.*
touch $#
(Obviously if you are using target specific value, things become more complex). Is there a way to achieve the same desired effect with standard (POSIX) Make? If not, with the BSD variant of Make?
If you can assume GNU make 4.0 or above, you can use the != assignment operator which was implemented in GNU make for portability with BSD make.
From the GNU make manual:
The shell assignment operator '!=' can be used to execute a shell
script and set a variable to its output. This operator first evaluates
the right-hand side, then passes that result to the shell for execution.
If the result of the execution ends in a newline, that one newline is
removed; all other newlines are replaced by spaces. The resulting
string is then placed into the named recursively-expanded variable.
Note there is one subtle difference between != and := $(shell ...): in the former the resulting variable is a recursive variable which means it will be re-expanded on use. So you need to be careful if your script might emit characters that are special to make (the $ basically).
I want to encode the the rule "to make <name>.done, you need all files of the pattern <name>.needed.*. I've attempted to write this with this Makefile:
%.done: $(wildcard %.needed.*)
cat $^ > $#
Yet when I run touch foo.needed.bar && make foo.done, all I get is
cat > foo.done
It appears the % inside $(wildcard) is being interpreted as a literal "%". How can I get it expanded to the right value ("foo" in this case)?
The % is just a placeholder for "any string" in pattern matching. It has no special meaning in the wildcard function and is interpreted literally.
You might attempt using $* instead (which would expand to the stem of the filename), but unfortunately it won't work either:
%.done: $(wildcard $*.needed.*)
The reason it doesn't work is that the automatic variables ($* is one of them) are not available for use in the dependency list.
The workaround is to request a secondary expansion for the target:
.SECONDEXPANSION:
%.done: $$(wildcard $$*.needed.*)
This will prompt GNU Make to go over the rule a second time after processing the Makefile as usual, expanding any escaped variables that weren't expanded the first time around. The second time around, the automatic variable have their appropriate values.
Update: GNU Make 3.81, Ubuntu 12.04
I have a set of markdown files that I want to compile to (say) html files, so this is my rule:
%.html: %.md
pandoc $< -o $#
So make foo.html would convert foo.md into foo.html.
However, there are spaces in the source markdown filenames and I do not have the ability to control these, that is I can't change a setting to remove the spaces.
This means if I make foo\ bar.html, I get
make: *** No rule to make target `foo bar.html'. Stop.
How can I write a generic rule %.html: %.md where the prerequisite filename has spaces?
I can get around it by using:
foo\ bar.html: foo\ bar.md
pandoc $< -o $#
But then I must manually write out this rule for every such source file that I have, when I'd rather use the % construct. Is my only hope to do some sort of $(foreach f,$(get list of *.md files),$(eval $(call function_to_generate_rule)))?
It seems from what #binki says that GNU make 3.82 might not have this issue, but unfortunately I do not have the option to update from v3.81 that is on my Ubuntu 12.04 machine.
I managed to "solve" it like so by using SECONDEXPANSION to substitute spaces with backslash-space in the prerequisite (so a prerequisite of foo bar.md becomes foo\ bar.md).
# define a variable with a single space
space:=
space+=
.SECONDEXPANSION:
%.html: $$(subst $$(space),\$$(space),%).md
pandoc "$<" -o "$#"
Here is the log. Again, works on Ubuntu 12.04/GNU Make 3.81, perhaps if you have 3.82 you can use #binki's solution which seems more elegant.
Edit
Apparently make’s support for whitespace in inference rules depends on what variant of GNU Make you are using. It just magically works fine with Gentoo’s patched sys-devel/make-3.82-r4 (and fails with Gentoo’s make 3.81-r2). I did not notice any explanation in make-3.82’s ChangeLog or NEWS or the Gentoo patches when quickly checking them. So implicit rules working with whitespace could just be a fluke in make-3.82 itself or even from Gentoo’s patchset. Official GNU support for whitespace in targets is tracked in the still-open GNU Make bug #712.
Original misguided answer
You can use any quoting characters that your shell supports. make ignores them when performing macro substitution and passes them directly to the shell. For example,
.SUFFIXES: .md .html
.md.html:
pandoc "$(<)" > "$(#)"
results in $ make foo\ bar.html passing the shell pandoc "foo bar.md" > "foo bar.html". I decided to use the traditional style of specifying generic make rules instead of the GNU Make extension involving %, but you can do this with GNU Make’s %-style rules too, I assume.
This does not solve the potential problem of the filenames containing quote characters in them. I think that, simply, most people just avoid putting " or ' in filenames because of the likelihood of causing issues with Makefiles or other scripts. Or you could use a GNU Makefile extension to replace the " characdter with \", something that makes sh happy (we’re going to just ignore cmd for now because I don’t even…):
.SUFFIXES: .md .html
.md.html:
pandoc "$(subst ",\",$(<))" > "$(subst ",\",$(#))"
This was tested with a file called a"b"c.md which succeeded in creating a"b"c.html (disclaimer: I used discount’s markdown command instead of pandoc).
I have a directory containing several files, some of which have spaces in their names:
Test workspace/
Another directory/
file1.ext
file2.ext
demo 2012-03-23.odp
I use GNU's $(wildcard) command on this directory, and then iterate over the result using $(foreach), printing everything out. Here's the code:
FOO := $(wildcard *)
$(info FOO = $(FOO))
$(foreach PLACE,$(FOO),$(info PLACE = $(PLACE)))
Here's what I would expect to see printed out:
Test workspace
Another directory
file1.ext
file2.ext
demo 2012-03-23.odp
Here's what I would actually get:
Test
workspace
Another
directory
file1.ext
file2.ext
demo
2012-03-23.odp
The latter is obviously of no use to me. The documentation for $(wildcard) flat-out states that it returns a "space-separated list of names" but completely fails to acknowledge the huge problems this raises. Nor does the documentation for $(foreach).
Is it possible to work around this? If so, how? Renaming every file and directory to remove the spaces is not an option.
The bug #712 suggests that make does not handle names with spaces. Nowhere, never.
I found a blog post saying it's partially implemented by escaping the spaces with \ (\\ seems to be typo or formatting artefact), but:
It does not work in any functions except $(wildcard).
It does not work when expanding lists of names from variables, which includes the special variables $?, $^ and $+ as well as any user-defined variable. Which in turn means that while $(wildcard) will match correct files, you won't be able to interpret the result anyway.
So with explicit or very simple pattern rules you can get it to work, but beyond that you are out of luck. You'll have to look for some other build system that does support spaces. I am not sure whether jam/bjam does, scons, waf, ant, nant and msbuild all should work.
GNU Make does very poorly with space-separated filenames.
Spaces are used as delimiters in word list all over the place.
This blog post summarizes the situation well, but WARNING: it incorrectly uses \\ rather than \
target: some\ file some\ other\ file
some\ file some\ other\ file:
echo done
You can also use variables, so this would also work
VAR := some\ file some\ other\ file
target: $(VAR)
$(VAR):
echo done
Only the wildcard function recognizes the escaping, so you can't do anything fancy without lots of pain.
But don't forget that your shell uses spaces as delimiters too.
If I wanted to change the echo done to touch $#, I'd have to add slash to escape it for my shell.
VAR := some\ file
target: $(VAR)
$(VAR):
touch $(subst \,\\,$#)
or, more likely, use quotes
VAR := some\ file some\ other\ file
target: $(VAR)
$(VAR):
touch '$#'
In the end, if you want to avoid a lot of pain, both in GNU make, and in your shell, don't put spaces in your filenames. If you do, hopefully the limited capabilities of Make will be sufficient.
This method will also allow use of listed file names such as $? and user variables that are lists of files.
The best way to deal with spaces in Make is to substitute spaces for other characters.
s+ = $(subst \ ,+,$1)
+s = $(subst +,\ ,$1)
$(call s+,foo bar): $(call s+,bar baz) $(call s+,bar\ baz2)
# Will also shows list of dependencies with spaces.
#echo Making $(call +s,$#) from $(call +s,$?)
$(call s+,bar\ baz):
#echo Making $(call +s,$#)
$(call s+,bar\ baz2):
#echo Making $(call +s,$#)
Outputs
Making bar baz
Making bar baz2
Making foo bar from bar baz bar baz2
You can then safely manipulate lists of file names using all the GNU Make
functions. Just be sure to remove the +'s before using these names in a rule.
SRCS := a\ b.c c\ d.c e\ f.c
SRCS := $(call s+,$(SRCS))
# Can manipulate list with substituted spaces
OBJS := $(SRCS:.c=.o)
# Rule that has object files as dependencies.
exampleRule:$(call +s,$(OBJS))
# You can now use the list of OBJS (spaces are converted back).
#echo Object files: $(call +s,$(OBJS))
a\ b.o:
# a b.o rule commands go here...
#echo in rule: a b.o
c\ d.o:
e\ f.o:
Outputs
in rule: a b.o
Object files: a b.o c d.o e f.o
This info is all from the blog that everyone else was posting.
Most people seem to be recommending using no spaces in paths or using Windows 8.3 paths, but if you must use spaces, escaping spaces and substitution works.
If you are willing to rely on your shell a bit more, this gives a list which can hold names with spaces just fine:
$(shell find | sed 's: :\\ :g')
The original question said that "renaming is not an option", yet many commenters have pointed out that renaming is pretty much the only way Make can handle spaces. I suggest a middle way: Use Make to temporarily rename the files and then rename them back. This gives you all the power of Make with implicit rules and other goodness, but doesn't mess up your file naming scheme.
# Make cannot handle spaces in filenames, so temporarily rename them
nospaces:
rename -v 's/ /%20/g' *\ *
# After Make is done, rename files back to having spaces
yesspaces:
rename -v 's/%20/ /g' *%20*
You could call these targets by hand with make nospaces and make yesspaces, or you can have other targets depends on them. For example, you might want to have a "push" target which makes sure to put the spaces back in filenames before syncing files back with a server:
# Put spaces back in filenames before uploading
push: yesspaces
git push
[Sidenote: I tried the answer which suggested using +s and s+ but it made my Makefile harder to read and debug. I gave up on it when it gave me guff over implicit rules likes: %.wav : %.ogg ; oggdec "$<".]
package_version := $(version)x0d$(date)
what is the x0d part between version and date vars? is it just string?
What $(dotin_files:.in=) does below
code
dotin_files := $(shell find . -type f -name \*.in)
dotin_files := $(dotin_files:.in=)
what this means $(dotin_files:=.in)
code
$(dotin_files): $(dotin_files:=.in)
$(substitute) $#.in > $#
can target contain multiple files?
what is the meaning of declaring target variable as PHONY?
code
.PHONY: $(dotin_files)
In the regex replacement code below
code
substitute := perl -p -e 's/#([^#]+)#/defined $$ENV{$$1} ? $$ENV{$$1} : $$&/ge'
what are $$ENV{$$1} and $$&? I guess it's Perl scope...
thanks for your time
Variable Expansion
$() is variable expansion in make, this should just be string substitution - if your makefile is
version=1
date=1.1.10
package_version:=$(version)x0d$(date)
then the variable package_version will expand to 1x0d1.1.10.
Substitution
The syntax $(var:a=b) is a substitution reference and will expand to var with a suffix a substituted with b.
For example, in
foobar:= foo bar
faabar:=$(foobar:oo=aa)
$(faabar) will expand to the string faa bar.
Multiple Targets
Multiple targets in a make rule is equivalent to having n rules with a single target, eg
foo bar:foo.c bar.c
$(CC) -o $# $^
is equivalent to
foo:foo.c bar.c
$(CC) -o $# $^
bar:foo.c bar.c
$(CC) -o $# $^
remember that any variables here are expanded.
Phony Targets
The .PHONY target declares that a rule doesn't produce an actual file, so it will always be built. As always, variables are expanded first. In your case this will expand to something like
.PHONY: foo bar
Escaping
A dollar sign is an escape character in makefiles, the $$ in your perl example is a literal $, eg substitute will be the string
perl -p -e 's/#([^#]+)#/defined $ENV{$1} ? $ENV{$1} : $&/ge'
The dollar signs here are processed by perl, and probably give environment variables (I don't know perl).
x0d part between version and date vars, is it just string?
Yes.
What $(dotin_files:.in=) does below
Removes the .in from the filenames found with the shell find.
what this means $(dotin_files:=.in)
I think you meant $(dotin_files:.in=). As already answered, within the variable dotin_files it replaces any occurrence of ".in" with an empty string(the part between the "=" and ")".
can target contain multiple files?
Yes
what is the meaning of declaring target variable as PHONY?
make will ignore targets time-stamp and consider them as new
thus rebuilding them each time.
In the regex replacement code below what are $$ENV{$$1} and $$&?
To avoid expansion of $ENV, the $ is doubled, think of '%' in C format strings, thus the string
perl -p -e 's/#([^#]+)#/defined $$ENV{$$1} ? $$ENV{$$1} : $$&/ge'
when called as a shell command will become:
perl -p -e 's/#([^#]+)#/defined $ENV{$1} ? $ENV{$1} : $&/ge'
$ENV is the perl Environment hash, $1 I think it's a backreference in the s/// regexp group.
Michael, you've been asking a lot of basic Makefile questions, and the ones you're asking now are ones you should be able to answer for yourself by experiment.
can target contain multiple files?
Try it:
dotin_files := foo.in bar.in
$(dotin_files):
#echo $#
Now try make foo.in and make bar.in. What happens?
What $(dotin_files:.in=) does
It's a substitution reference. Try it yourself and see what happens, like this:
dotin_files := foo.in bar.in
dotin_files := $(dotin_files:.in=)
all:
#echo $(dotin_files)
What did it do?
substitute := perl -p -e 's/#([^#]+)#/defined $$ENV{$$1} ? $$ENV{$$1} : $$&/ge'
what are $$ENV{$$1} and $$&? I guess it's Perl scope...
Let's take a look:
all:
#echo $(substitute)
If you want to know more about Perl, or regexs, or find, or make, or whatever, feel free to ask here, but please take a little time to try to figure it out first.