I have the following project structure:
+-Makefile
+-src/
+-a/
| +-foo.py
+-b/
| +-foo.py
+-c/
| +-foo.py
Each foo.py file is a different file with exactly the same name (ie they have different inodes but are all literally called 'foo.py' - although of course in reality the name is not foo.py, which is just an example).
I wish to create a GNU Makefile rule which, when run, creates the following structure:
+-Makefile
+-src/
+-a/
| +-foo.py
| +-a.zip
+-b/
| +-foo.py
| +-b.zip
+-c/
| +-foo.py
| +-c.zip
This is the closest I have been able to figure out, although of course it fails due to using a target variable in the prereqs which doesn't seem to be allowed:
SRCDIR = src/
PRJ_DIRS = a b c
SRC_FILE = foo.py
# This next rather nasty line turns PRJ_DIRS in to, e.g., src/a/a.zip etc.
ZIP_FILES = $(addprefix $(SRCDIR),$(join $(PRJ_DIRS),$(PRJ_DIRS:%=/%.zip)))'
build: $(ZIP_FILES)
# Next line crashes because we can't use $# in the prereq
$(ZIP_FILES): $(addprefix $(dir $#), $(SRC_FILE))
touch $#
So one way to put the question:
How can I write a rule the uses each corresponding foo.py file as a prereq to build the appropriate .zip file?
Or, a more direct question that might actually be looking at this from the wrong angle:
How can I refer to the specific target that's being built in the prereqs?
Okay, since you consider secondary expansion a kludge, it's $(foreach ...) and $(eval ...) to the rescue again (a very powerful combination).
Replace your rule with the following and you should get what you want.
define rule
$(SRCDIR)$(1)/$(1).zip: $(SRCDIR)$(1)/$(SRC_FILE)
touch $$#
endef
$(foreach dir, $(PRJ_DIRS), $(eval $(call rule,$(dir))))
Related
Some code from Makefile:
tempDir := ...
javaSources := $(wildcard src/java/**/%.java)
javaClasses := $(subst src/java, $(tempDir)/java/classes, $(subst .java,.class, $(javaSources)))
$(javaClasses): $(javaSources)
mkdir -p $(tempDir)/java/classes || true
javac \
-d $(tempDir)/java/classes \
-cp $(tempDir)/java/classes \
$?
How to create a pattern rule (like here) to preserve in / out order?
#MadScientist
First, your wildcard won't work. GNU make uses only basic shell globbing, which means it can't understand advanced globbing like ** meaning "search all subdirectories". Second, % is not a shell globbing character at all so you're just looking for files that are literally named %.java.
Instead you probably want something like this:
javaSources := $(shell find src/java -name '*.java')
Next, to create the javaClasses content you really don't want to use subst because it substitutes everywhere which can give false matches (e.g., $(subst .x,.y,foo.xbar) will yield foo.ybar which is probably not what you want).
Something like this is simpler to understand:
javaClasses := $(patsubst src/java/%.java,$(tempdir)/java/classes/%.class,$(javaSources))
Finally, you are repeating exactly the same error you made in the previous question, where you tried to list all the targets and all the prerequisites in the same rule. Just as I said for that question, that is not right.
The answer is exactly the same as in the previous question: you should write a pattern rule that describes how to build one single target from one single source file.
And again you need an all target or similar which depends on all the outputs.
In complement to MadScientist answer, you should probably use a pattern rule like:
$(tempDir)/java/classes/%.class: src/java/%.java
mkdir -p $(dir $#)
javac -d $(dir $#) -cp $(dir $#) $<
(not sure what -cp should be in this case, it depends on your specific project). And as MadScientist also suggested, you will need:
.PHONY: all
all: $(javaClasses)
such that you can call make all to compile all the source files that need to be. Put it before any other explicit target if you want all to be the default goal (the goal make selects if you just call make), or use the .DEFAULT_GOAL special variable:
.DEFAULT_GOAL := all
I have a bunch of different source files in my static HTML blog. The outermost extensions explain the format to be processed next.
Example: Source file article.html.md.gz (with target article.html) should be processed by gunzip, then by my markdown processor.
Further details:
The order of the extensions may vary
Sometimes an extension is not used (article.html.gz)
I know how to process all different extensions
I know that the final form is always article.html
Ideally I would have liked to just write rules as follows:
...
all-articles: $(ALL_HTML_FILES)
%: %.gz
gunzip ...
%: %.md
markdown ...
%: %.zip
unzip ...
And let make figure out the path to take based on the sequence of extensions.
From the documentation however, I understand that there are constraints on match-all rules, and the above is not possible.
What's the best way forward? Can make handle this situation at all?
Extensions are made up examples. My actual source files make more sense :-)
I'm on holiday so I'll bite.
I'm not a fan of pattern rules, they are too restricted and yet too arbitrary at the same time for my tastes. You can achieve what you want quite nicely in pure make:
.DELETE_ON_ERROR:
all: # Default target
files := a.html.md.gz b.html.gz
cmds<.gz> = gzip -d <$< >$#
cmds<.md> = mdtool $< -o $#
define rule-text # 1:suffix 2:basename
$(if $(filter undefined,$(flavor cmds<$1>)),$(error Cannot handle $1 files: [$2$1]))
$2: $2$1 ; $(value cmds<$1>)
all: $2
endef
emit-rule = $(eval $(call rule-text,$1,$2))# 1:suffix 2:basename
emit-hierachy = $(if $(suffix $2),$(call emit-rule,$1,$2)$(call emit-hierachy,$(suffix $2),$(basename $2)))# 1:suffix 2:basename
emit-rules = $(foreach _,$1,$(call emit-hierachy,$(suffix $_),$(basename $_)))# 1:list of source files
$(call emit-rules,${files})
.PHONY: all
all: ; : $# Success
The key here is to set $files to your list of files.
This list is then passed to emit-rules.
emit-rules passes each file one-at-a-time to emit-hierachy.
emit-hierachy strips off each extension in turn,
generates the appropriate make syntax, which it passes to $(eval …).
emit-hierachy carries on until the file has only one extension left.
Thus a.html.md.gz becomes this make syntax:
a.html.md: a.html.md.gz ; gunzip <$< >$#
a.html: a.html.md ; mdtool $< -o $#
all: a.html
Similarly, b.html.gz becomes:
b.html: b.html.gz ; gunzip <$< >$#
all: b.html
Neato, or what?
If you give emit-rules a file with an unrecognised extension (c.html.pp say),
you get a compile-time error:
1:20: *** Cannot handle .pp files: [c.html.pp]. Stop.
Compile-time? Yeah, before any shell commands are run.
You can tell make how to handle .pp files by defining cmds<.pp> :-)
For extra points it's also parallel safe. So you can use -j9 on your 8 CPU laptop, and -j33 on your 32 CPU workstation. Modern life eh?
as i am currently working on my makefile i encountered another problem. I am using this rule as part of my building process which transforms .mid files into .s files.
$(MIDAS): $(BLDPATH)/%.s: %.mid
$(shell mkdir -p $(dir $#))
#test $($< | sed "*")
$(MID2AGB) $(MIDFLAGS) -G$($< | sed ".*mus/vcg([0-9]{3})/.*\.mid") $< $#
All .mid input files follow the same format: .mus/vcg[0-9]{3}/..mid, meaning they are stored in different directories following the naming convention vcgXXX where X can be any digit from 0-9. (Maybe my regex is even wrong for this).
When i am calling $(MID2AGB) i want to use a compiler flag -GXXX. However the XXX of this flag has to match the XXX from the input file path.
My makefile code does not work. Any idea how to fix this problem?
There is a crude but effective way to do this using Make's string manipulation tools:
# suppose the source is .mus/vcg456/Z.mid
$(MIDAS): $(BLDPATH)/%.s: %.mid
#echo $* # this gives .mus/vcg456/Z
#echo $(subst /, ,$*) # this gives .mus vcg456 Z
#echo $(word 2,$(subst /, ,$*) # this gives vcg456
#echo $(subst vcg,,$(word 2,$(subst /, ,$*)) # this gives 456
So some anonymous developers have decided to use a ridiculous convention of using spaces in their folder names that contain their source files. I would change these folders not to use spaces but sadly I don't make the rules around here so that's not an option (though I wish it were).
LUAC = luac
SRC_DIR = .
SOURCE = \
stupid/naming\ convention/a.lua \
stupid/naming\ convention/very\ annoying/b.lua \
vpath .lua $(SRC_DIR)
OUT_DIR = ../out/
OUTPUT = $(patsubst %.lua, $(OUT_DIR)/%.luac, $(SOURCE))
all: $(OUTPUT)
$(OUT_DIR)/%.luac: %.lua
$(LUAC) "$<"
mv luac.out "$#"
.PHONY: all
Simple Makefile. All it's meant to do is compile all the Lua files that I have and put them into an output directory.
No matter I do it keeps wanting to split the SOURCE string on the spaces in the folder, so I end with a beautiful error like this:
make: *** No rule to make target `stupid/naming ', needed by `all'. Stop.
Is there a way to fix this without renaming the folders?
Thanks in advance.
The very short, but IMO ultimately correct, answer is that make (not just GNU make, but all POSIX-style make implementations) does not support pathnames containing whitespace. If you want to use make, your "anonymous developers" simply cannot use them. If they insist that this is an absolute requirement you should switch to a different build tool altogether, that does support whitespace in filenames.
Yes, it's barely possible to create a makefile that will work with filenames containing whitespace, but you will essentially have to rewrite all your makefiles from scratch, and you will not be able to use many of the features of GNU make so your makefiles will be long, difficult to read, and difficult to maintain.
Just tell them to get over themselves. Or if they really can't, try having them create their workspace in a pathname without any whitespace in the names, then create a symbolic link containing whitespace pointing to the real workspace (the other way around won't work in all situations).
Unfortunately, GNU Make's functions that deal with space-separated list do not
respect the escaping of the space. The only exception is wildcard.
Edit:
Here's my workaround:
LUAC = luac
SRC_DIR = .
SOURCE = \
stupid/naming\ convention/a.lua \
stupid/naming\ convention/very\ annoying/b.lua \
vpath .lua $(SRC_DIR)
OUT_DIR = ../out/
OUTPUT = $(patsubst %.lua,%.luac,$(SOURCE))
all: $(OUTPUT)
%.luac: %.lua
$(LUAC) "$<"
mv luac.out "$#""
.PHONY: all
I tried to output it first like that:
%.luac: %.lua
#echo "$<"
#echo "$#""
Output looks as follows:
stupid/naming convention/a.lua
../out/stupid/naming convention/a.luac
stupid/naming convention/very annoying/b.lua
../out/stupid/naming convention/very annoying/b.luac
If you look at this excellent write up: http://www.cmcrossroads.com/article/gnu-make-meets-file-names-spaces-them, the author suggests that this is mostly a difficult task. But his substitution functions could get you going in case you really can't avoid the spaces.
Putting this into your makefile would look like this (sorry if I changed some of your paths, but this works on my Cygwin installation):
LUAC = luac
s+ = $(subst \\ ,+,$1)
+s = $(subst +,\ ,$1)
SRC_DIR = .
SOURCE := stupid/naming\\ convention/a.lua
SOURCE := $(call s+,$(SOURCE))
vpath .lua $(SRC_DIR)
OUT_DIR = out/
OUTPUT = $(patsubst %.lua, $(OUT_DIR)/%.luac, $(SOURCE))
all: $(call +s,$(OUTPUT))
$(OUT_DIR)/%.luac: %.lua
$(LUAC) "$<"
mv luac.out "$#"
.PHONY: all
I know that's not a complete answer, but maybe an encouragement that it actually is possible. But I agree with the other posters that if you can actually avoid spaces altogether, you will have a much easier life!
Another strategy which works when you are generating your Makefile automatically is this one, also used in Perl's ExtUtils::MakeMaker: to separate the name formatted to be usable in recipes, versus it being usable as a dependency. The example here has a THISFILE and a THISFILEDEP.
AWKWARD_DIR = sub dir
AWKWARD_DIRDEP = sub\ dir
THISFILE = $(AWKWARD_DIR)/d1
THISFILEDEP = $(AWKWARD_DIRDEP)/d1
AWKWARD_DIR_EXISTS = $(AWKWARD_DIR)/.exists
AWKWARD_DIR_EXISTSDEP = $(AWKWARD_DIRDEP)/.exists
TARGET = $(AWKWARD_DIR)/t1
TARGETDEP = $(AWKWARD_DIRDEP)/t1
MAKEFILE = spacemake.mk
$(TARGETDEP): $(THISFILEDEP) $(AWKWARD_DIR_EXISTSDEP)
cat "$(THISFILE)" >"$(TARGET)"
$(THISFILEDEP): $(AWKWARD_DIR_EXISTSDEP)
echo "yo" >"$(THISFILE)"
$(AWKWARD_DIR_EXISTSDEP): $(MAKEFILE)
#echo MAKEFILE = $(MAKEFILE)
-mkdir "$(AWKWARD_DIR)"
touch "$(AWKWARD_DIR_EXISTS)"
You can try it by placing it in a file called e.g. spacemake.mk, then run it with gmake -f spacemake.mk.
I want to write something like regex:
SRC:="a.dat.1 a.dat.2"
$(SRC): %.dat.%: (\\1).rlt.(\\2)
dat2rlt $^ $#
so that a.dat.1 and a.dat.2 will give a.rlt.1 and a.rlt.2.
In GNU Make info page, it says "the % can be used only once".
Is there some trick to achieve this in GNU Make?
I'm afraid what you are trying to do is not possible the way you suggest to do it, since - as you already mention - (GNU) make only allows a single stem '%', see http://www.gnu.org/software/make/manual/make.html#Pattern-Rules:
A pattern rule looks like an ordinary rule, except that its target
contains the character ‘%’ (exactly one of them).
Without it, creating such 'multi-dimensional' targets is cumbersome.
One way around this is by rebuilding the name of the dependency in the command (rather than in the dependency list):
SRC := a.dat.1 a.dat.2
all : $(SRC:%=%.dat2rlt)
%.dat2rlt :
dat2rtl $(word 1,$(subst ., ,$*)).rlt.$(word 2,$(subst ., ,$*)) $*
Of course, however, this way you would lose the dependency, it will not rebuild once the rlt has been updated.
The only way I can see to address that is by generating the rules explicitly:
SRC := a.dat.1 a.dat.2
all : $(SRC)
define GEN_RULE
$1.dat.$2 : $1.rlt.$2
dat2rtl $$< $$#
endef
$(foreach src,$(SRC),$(eval $(call GEN_RULE,$(word 1,$(subst ., ,$(src))),$(word 3,$(subst ., ,$(src))))))
Using named variables, we can write more readable code (based on answer of Paljas):
letters:=a b c
numbers:=1 2 3 4
define GEN_RULE
$(letter).dat.$(number) : $(letter).rlt.$(number)
./rlt2dat $$< $$#
endef
$(foreach number,$(numbers), \
$(foreach letter,$(letters), \
$(eval $(GEN_RULE)) \
) \
)
We can generate SRC in a similar way. Note that using that method SRC will contain all the combinations. That may or may not be beneficial.
Building on the answer of Erzsébet Frigó, you might additionally choose to:
in the inner loop, eval not the macro itself but the result of calling it
name the macro after program you're calling, dat2rtl
in combination, allowing you to
refer to the program name using make's ${0}
define a target, ${0}s (expanding to dat2rts - note the pluralization) with preconditions of all combinations of letters and numbers on which dat2r2 was called
Like this:
letters:=a b c
numbers:=1 2 3 4
define rlt2dat
${0}s::$(letter).dat.$(number)
$(letter).dat.$(number): $(letter).rlt.$(number)
./${0} $$< $$#
endef
$(foreach number,$(numbers), \
$(foreach letter,$(letters), \
$(eval $(call rlt2dat))))
allowing you to build all rlt2dat targets as:
make rlt2dats
For the limited example you gave, you can use a pattern with one %.
SRC := a.dat.1 a.dat.2
${SRC}: a.dat.%: a.rlt.%
dat2rlt $^ $#
$* in the recipe will expand to whatever the % matched.
Note that the "s around your original macro are definitely wrong.
Have a look at .SECONDEXPANSION in the manual for more complicated stuff (or over here).