Complex pattern rule in Makefile - makefile

I have the following makefile I use to generate files from some templates, the generated files have two possible extensions:
%.tex: %.tex*_tpl
./generate $#_tpl -o $#
%.xml: %.xml*_tpl
./generate $#_tpl -o $#
The dependency list will here match things like a.tex_tpl, a.tex-subpart1_tpl, a.tex-subpart2_tpl.
While this works, is there a way to avoid repetition? For example by matching *.{tex,xml} in the rule name and use the whole matched name in the dependency list? Something that would look like that:
%.{tex,xml}: $#_tpl
./generate $< -o $#
(Though I know %.{tex,xml} is not a valid rule name and you can't use $# in the dependency list)
Or any other (cleaner?) way.

Seems to me this does what you are looking for:
#
# I've assumed that files of the form:
#
# a.xml_tpl
# b.tex_tpl
#
# determine what targets you want to build
#
TARGETS:=$(patsubst %_tpl,%,$(wildcard *.xml_tpl *.tex_tpl))
.PHONY: all
all: $(TARGETS)
.SECONDEXPANSION:
$(TARGETS): %: $$(wildcard %*_tpl)
./generate $^ -o $#
The key is to use .SECONDEXPANSION to allow $$(wildcard %*_tpl) to be evaluated in a second expansion phase. The double $ is not a typo, by the way; it protects the expression from being evaluated at the time of the first expansion.
If I populate a directory with these files:
a.tex-subpart1_tpl
a.tex_tpl
a.xml-subpart1_tpl
a.xml-subpart2_tpl
a.xml_tpl
and run make -n, I get this on the console:
./generate a.xml_tpl a.xml-subpart1_tpl a.xml-subpart2_tpl -o a.xml
./generate a.tex_tpl a.tex-subpart1_tpl -o a.tex
Why the Second Expansion?
Without the second expansion, you'd have to have $(wildcard %*_tpl) in the dependency because with the $$ the wildcard function would never execute. Instead, make would treat $$(wildcard..) literally as the dependency, which is obviously wrong.
Ok, so $(wildcard %*_tpl) would be evaluated at the time make first runs across that line (this is the "first expansion"). At that time % has no value yet so wildcard would roughly be doing something like what would be ls %*_tpl at the command line.
For reasons of speed, make does not by default give you the opportunity to do any evaluation later than during the first expansion. If you want a later opportunity you have to specify .SECONDEXPANSION, which turns on the second expansion processing. Make still performs the firts expansion as usual. This is why you need to have $$(wildcard: it is transformed to $(wildcard during the first expansion. At the time of the second expansion make sees $(wildcard %*_tpl), replaces % with the actual stem and then executes the wildcard function with the actual stem rather than with a literal %.
Why $(TARGETS) in the Pattern Rule?
The pattern rule could be written:
%: $$(wildcard %*_tpl)
./generate $^ -o $#
without $(TARGETS). However, this rule would do nothing, as it would be a "match-anything rule". Basically, if make took such a rule at face value, then the computation cost would be significant, and most likely it is not the case that the author of the Makefile really means to apply this rule to any file whatsoever. So such a rule comes with restrictions, which in the Makefile here make it useless.
Adding $(TARGETS) makes it into a static pattern rule, which is not a match-anything rule. The addition of $(TARGETS) in front of the target pattern tells make that the rule applies only to these targets, and nothing else.

Related

gnu make "Removing intermediate files"

I have the following rules
define compile_c
$(ECHO) "CC $<"
$(Q)$(CC) $(CFLAGS) -c -MD -o $# $<
## The following fixes the dependency file.
## See http://make.paulandlesley.org/autodep.html for details.
## Regex adjusted from the above to play better with Windows paths, etc.
#$(CP) $(#:.o=.d) $(#:.o=.P); \
$(SED) -e 's/#.*//' -e 's/^.*: *//' -e 's/ *\\$$//' \
-e '/^$$/ d' -e 's/$$/ :/' < $(#:.o=.d) >> $(#:.o=.P); \
$(RM) -f $(#:.o=.d)
endef
vpath %.c . $(TOP)
$(BUILD)/%.o: %.c $(BUILD)/%.pp
$(call compile_c)
vpath %.c . $(TOP)
$(BUILD)/%.pp: %.c
$(ECHO) "PreProcess $<"
$(Q)$(CC) $(CFLAGS) -E -Wp,-C,-dD,-dI -o $# $<
When the build finishes, GNU make says
Removing intermediate files... and deletes all the .pp files which I do NOT want.
Why is it doing this?
How do I stop it?
Since you're using GNU Make, you can make the following adjustment to your Makefile:
.PRECIOUS: $(BUILD)/%.pp # ADD THIS LINE
$(BUILD)/%.pp: %.c
$(ECHO) "PreProcess $<"
$(Q)$(CC) $(CFLAGS) -E -Wp,-C,-dD,-dI -o $# $<
The documentation has this to say about .PRECIOUS directives:
The targets which .PRECIOUS depends on are given the following special treatment: if make is killed or interrupted during the execution of their recipes, the target is not deleted.
[...]
Also, if the target is an intermediate file, it will not be deleted after it is no longer needed, as is normally done.
[...]
You can also list the target pattern of an implicit rule (such as ‘%.o’) as a prerequisite file of the special target .PRECIOUS to preserve intermediate files created by rules whose target patterns match that file's name.
This has the benefit of not creating an unwanted additional rule. It's also clearer what you're trying to do: keep the precious intermediate files that might be expensive to recreate.
I think the best solution is to use the .SECONDARY special target. Just add this line:
.SECONDARY:
Quoting the manual:
.SECONDARY with no prerequisites causes all targets to be treated as secondary (i.e., no target is removed because it is considered intermediate).
Why is this better than making the targets prerequisites of a throw-away target? That's more clutter, and has to be done explicitly for every set of files that might be generated with pattern rules.
Why is this better than .PRECIOUS? That causes files to be retained even if their recipe fails when using .DELETE_ON_ERROR. The latter is important to avoid failing recipes leaving behind bad outputs that are then treated as current by subsequent make invocations. IMO, you always want .DELETE_ON_ERROR, but .PRECIOUS breaks it.
If you search for "gnu make intermediate files" you'll immediately find the answer as to why it's happening, in the GNU make manual section Chains of Implicit Rules.
It also tells you how to avoid it: a file cannot be intermediate if it is mentioned in the makefile as a target or prerequisite.
So, just list your .pp files as a prerequisite of some rule, somewhere. It doesn't have to be a rule that's ever invoked. You don't give enough of your makefile here for us to provide a complete answer, but it would be something like:
all_pps: $(ALL_OBJECTS:.o=.pp)
assuming you had a variable ALL_OBJECTS containing all your .o files.
Here is a detail that finally got PRECIOUS working for me. The pattern that you give to PRECIOUS has to be exactly the pattern that is being used in the rule that creates the intermediate file. I want to save all files prefixed by moc_. At first I used
.PRECIOUS: moc_%
to no avail. Then I tried
.PRECIOUS: moc_%.cpp
and this did the trick.

Why GNU Make's secondary expansion does not work with pattern rules for dependencies?

Consider this makefile:
%: %.c
#echo making $#
#touch $#
.SECONDEXPANSION:
%.pid: $$(basename $$#)
$(<D)/$(<F) --pidfile=$<.pid
Here, the first rule builds a program and second rule starts it, producing a pid-file. Note: I know that secondary expansion is unnecessary in this example; the real makefile is more complex and I really need secondary expansion there.
So, typing make foo.pid, I expect Make to build foo from foo.c by the first rule and then run ./foo --pidfile=foo.pid by the second one.
However, it does not seem to work:
$ make -f Makefile.test foo.pid
make: *** No rule to make target 'foo.pid'. Stop.
This somehow relates to secondary-expanded dependencies which are provided by pattern rules. If I write either
%.pid: % in the second rule (i. e. get rid of secondary-expansion), or
foo: %: %.c in the first rule (i. e. write an explicit static pattern rule),
it all suddenly works. Why? Is this a limitation of GNU Make? In my case, I'd like to avoid pre-listing all possible programs in the first rule.
I think your situation is described in section 10.5.5 Match-Anything Pattern Rules. Whay you have here is a non-terminal match-anything rule, which is not allowed to match a file name that indicates a specific type of data.
If you had let's say
%:: %.c
#echo making $#
#touch $#
it would work.

GNU Makefile - Pattern rule with multiple targets with one dependency ignores all targets but the first

I want to make a language depend target. In Particular: I have one source-file and I want to create different Objects which where add to the corresponding language folder. That single source file will differ in the C-Flags, the compiler will get. As long as I used it in a static way, it works quite fine.
de/info.o en/info.o es/info.o : info.c
$(ECHO) (DEP) $< for $#
Now I thought, it would be great if it is a bit more dynamic, in case i'll add a new language depending file. So I used a wildcard as followed:
de/%.o en/%.o es/%.o : %.c
$(ECHO) (DEP) $< for $#
But now it just make the first target and ignores the rest. The Make-Debug prints the following thing:
Successfully remade target file `de/info.o'.
Considering target file `en/info.o'.
File `en/info.o' was considered already.
Just in case: No, the objects do not exist. So there is no target, but an existing dependencie, so make should execute the rules.
EDIT: Found a solution for that Problem.
define FOO
$(1)/%.o : %.c
$(ECHO) $$< for $(1)
endef
$(foreach lang,$(LANGUAGE_LIST), $(eval $(call FOO,$(lang))))
Inspired by: http://www.gnu.org/software/make/manual/make.html#Eval-Function
Pattern rules work differently than implicit rules. While an implicit rule such as
a b c: d
command
is equivalent to the longer notation
a: d
command
b: d
command
c: d
command
this does NOT hold for pattern rules. Pattern rules with multiple targets are explicitly required to build all of their targets in a single invocation of command. Thus you would have to write
$ cat GNUmakefile
all: de/x.o en/x.o es/x.o
de/%.o: %.c
#echo $# from $<
en/%.o: %.c
#echo $# from $<
es/%.o: %.c
#echo $# from $<
$ gmake
de/x.o from x.c
en/x.o from x.c
es/x.o from x.c
The relevant documentation is found in 10.5.1 Introduction to Pattern Rules of the GNU make manual:
Pattern rules may have more than one target. Unlike normal rules, this does not act as many different rules with the same prerequisites and recipe. If a pattern rule has multiple targets, make knows that the rule’s recipe is responsible for making all of the targets. The recipe is executed only once to make all the targets. When searching for a pattern rule to match a target, the target patterns of a rule other than the one that matches the target in need of a rule are incidental: make worries only about giving a recipe and prerequisites to the file presently in question. However, when this file’s recipe is run, the other targets are marked as having been updated themselves.

Force make not to re-order pre-requisites in automatic variable expansion

Here a is my makefile (GNU make) to compile a small OCaml program:
SUFFIXES:=
OCAML=ocamlopt
LD=ocamlopt
OFLAGS=
.PHONY: all clean
all: playground
playground.cmx: playground.ml lstream.cmi
playground: lstream.cmx playground.cmx
%.cmi: %.ml
$(OCAML) $(OFLAGS) -c $<
%.cmx: %.ml
$(OCAML) $(OFLAGS) -c $<
%: %.cmx
$(LD) -o $# $^
playground uses functions from the Lstream module. In this case, the ocaml linker requires the files to link to be specified in order of dependency (eg: ocamlopt -o playground lstream.cmx playground.cmx).
Despite the fact that I defined the playground rule's dependencies in the right order,
make consistently re-orders them and executes ocamlopt -o playground playground.cmx lstream.cmx which causes a linker error.
Is there a way to enforce the correct behaviour ? I would like to avoid specifying the link command explicitely and let make infer it from the dependencies.
Implicit rules always force the pattern matching prerequisite to be first, regardless of the order in which they're defined elsewhere. This is almost always what you want, because in most rules the pattern matching prerequisite is special.
For example when compiling an object file the prerequisites consist of one source file and a bunch of header files; the source file is special and needs to be listed on the command line. Make ensures that for a pattern rule %.o : %.c (for example) the prerequisite matching %.c is first in the list, and so is assigned to the $< automatic variable, and it can be treated differently.
In any event the short answer is no, you cannot modify this behavior. An implicit rule % : %.cmx matching a target playground will always force the prerequisite playground.cmx to be listed first in the prerequisite list. The other prerequisites will maintain their order.
If you really need the prerequisites to maintain their order then I recommend using a static pattern rule:
TARGETS = playground
$(TARGETS) : % :
$(LD) -o $# $^
(you can also use a "match anything" pattern rule but this can be a real performance degrader). Here since you have no pattern in the prerequisite list, nothing will be reordered.

GNU make with many target directories

I have to integrate the generation of many HTML files in an existing Makefile.
The problem is that the HTML files need to reside in many different directories.
My idea is to write an implicit rule that converts the source file (*.st) to the corresponding html file
%.html: %.st
$(HPC) -o $# $<
and a rule that depends on all html files
all: $(html)
If the HTML file is not in the builddir, make doesn't find the implicit rule: *** No rule to make target.
If I change the implicit rule like so
$(rootdir)/build/doc/2009/06/01/%.html: %.st
$(HPC) -o $# $<
it's found, but then I have to have an implicit rule for nearly every file in the project.
According to Implicit Rule Search Algorithm in the GNU make manual, rule search works like this:
Split the entire target name t into a directory part, called d, and the rest, called n. For
example, if t is src/foo.o,
then d is src/,
and n is foo.o.
Make a list of all the pattern rules one of whose targets matches t or n.
If the target pattern contains a slash,
it is matched against t;
otherwise, against n.
Why is the implicit rule not found, and what would be the most elegant solution, assuming GNU make is used?
Here is a stripped down version of my Makefile:
rootdir = /home/user/project/doc
HPC = /usr/local/bin/hpc
html = $(rootdir)/build/doc/2009/06/01/some.html
%.html: %.st
$(HPC) -o $# $<
#This works, but requires a rule for every output dir
#$(rootdir)/build/doc/2009/06/01/%.html: %.st
# $(HPC) -o $# $<
.PHONY: all
all: $(html)
The best solution I found so far is to generate an implicit rule per target directory via foreach-eval-call, as explained in the GNU make manual. I have no idea how this scales to a few thousand target directories, but we will see...
If you have a better solution, please post it!
Here is the code:
rootdir = /home/user/project/doc
HPC = /usr/local/bin/hpc
html = $(rootdir)/build/doc/2009/06/01/some.html \
$(rootdir)/build/doc/2009/06/02/some.html
targetdirs = $(rootdir)/build/doc/2009/06/01 \
$(rootdir)/build/doc/2009/06/02
define generateHtml
$(1)/%.html: %.st
-mkdir -p $(1)
$(HPC) -o $$# $$<
endef
$(foreach targetdir, $(targetdirs), $(eval $(call generateHtml, $(targetdir))))
.PHONY: all
all: $(html)
Like Maria Shalnova I like recursive make (though I disagree with "Recursive Make Considered Harmful"), and in general it's better to make something HERE from a source THERE, not the reverse. But if you must, I suggest a slight improvement: have generateHtml generate only the RULE, not the COMMANDS.
Your active implicit rule makes $(rootdir)/build/doc/2009/06/01/some.html depend on $(rootdir)/build/doc/2009/06/01/some.st. If $(rootdir)/build/doc/2009/06/01/some.st doesn't exist then the rule won't be used/found.
The commented out rule makes $(rootdir)/build/doc/2009/06/01/some.html depend on some.st.
One solution is to make you're source layout match your destination/result layout.
Another option is to create the rules as required with eval. But that will be quite complicated:
define HTML_template
$(1) : $(basename $(1))
cp $< $#
endef
$(foreach htmlfile,$(html),$(eval $(call HTML_template,$(htmlfile))))
An other possibility is to have the commando make call itself recursively with the argument -C with every output directory.
Recursive make is somewhat the standard way to deal with subdirectories, but beware of the implications mentioned in the article "Recursive Make Considered Harmful"

Resources