I have, for example, the following Makefile to generate PDF files from Markdown files in subdirectories:
FOLDERS = f1 f2 f3
.PHONY: $(FOLDERS)
f1: f1/f1.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
f2: f2/f2.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
f3: f3/f3.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
The expected result is that make f1 requires the existence of f1/f1.md, and generates the resulting PDF as f1/f1.pdf. The same for f2 and f3. This works, but the declarations seem unnecessarily repetitive.
Is there any way to combine these three rules into one, generic rule? That is, without needing to explicitly write out all of the paths to the PDF files or Markdown files, as I may be dynamically adding subfolders and I'd prefer to just change the definition of FOLDERS in the first line. I've googled around and tried a few things, but I feel like either I can't find the right incantation to use, or I'm missing a piece of knowledge about how Makefiles work. Could someone please point me in the right direction?
First, note that there's no good reason to use PHONY targets here, since these rules appear to be building files whose names are known beforehand. Targets like f1/f1.pdf would be much better.
Unfortunately we can't use a pattern rule when the stem (e.g. f1) is repeated in a prerequisite. But a "canned recipe" can do the trick:
define pdf_template
$(1): $(1)/$(1).md
cd $$# && pandoc $$(notdir $$^) -o $$(patsubst %.md,%.pdf,$$(notdir $$^))
endef
$(eval $(call pdf_template,f1))
$(eval $(call pdf_template,f2))
$(eval $(call pdf_template,f3))
(Note how you must escape the $ signs in the template.)
If those $(eval...) lines look too repetitive, you can replace them with a loop:
$(foreach folder,$(FOLDERS),$(eval $(call pdf_template,$(folder))))
EDIT: Come to think of it, there's another way. You can't construct a pattern rule that uses the stem more than once:
$(FOLDERS): %: %/%.md
cd $# && ... this won't work
And you can't use the automatic variables in the prerequisite list, because they aren't yet defined when they're needed:
$(FOLDERS): $#/$#.md
cd $# && ... this won't work either
But you can use them there if you use Secondary Expansion, which causes Make to expand the prereq list a second time:
.SECONDEXPANSION:
$(FOLDERS): $$#/$$#.md
cd $# && ... this works
Again, note the escaped $ symbols.
Related
I have a Makefile which I use to call different sub-Makefiles.
I have several rules:
all
clean
fclean
re
I can already use those rules, which will call every sub makefile with the same rule.
I have several project, and I would like to generate rules with that format:
$(PROJECT_NAME)-$(RULES)
With that, I would like to have each rule for each project:
project1-all
project1-clean
...
project2-all
project2-clean
...
This way, I would be able to call a specific rule, for a specific project, like project1-fclean.
I tried that:
RULES= all clean fclean re
PROJECTS= project1 project2
define NEWLINE
endef
$(foreach _rule, $(RULES), \
$(foreach _proj, $(PROJECTS), \
$(_proj)-$(_rule): $(NEWLINE) \
$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule) $(NEWLINE) \
) \
)
But it doesn't seems to work. I have searched, but I haven't found advanced makefile techniques to achieve that. Plz help.
The problem is that when you combine lines together with line continuations like that, it compresses out all the newlines and other extraneous whitespace (including those newlines you're trying to insert with $(NEWLINE)) resulting in a huge mess on a single line, rather than multiple lines with multiple patterns. To do this correctly, you need to write your rule as a macro with arguments and then call it:
define PROJ_RULE
$(1)-$(2):
$(MAKE) $(ARGS) $(PROJECT_DIR)$(1) $(2)
endef
$(foreach _rule, $(RULES),
$(foreach _proj, $(PROJECTS),
$(eval $(call PROJ_RULE, $(_proj), $(_rule)))))
note that all this define and foreach stuff in GNU make specific -- other make flavors do not support it.
Okay, so I finally managed to do it this way:
$(foreach _rule, $(RULES), $(addsuffix -$(_rule),$(PROJECTS))):
$(eval _rule := $(lastword $(subst -, ,$#)))
$(eval _proj := $(#:%-$(_rule)=%))
#$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule)
I will decompose it for a better explanation:
$(foreach _rule, $(RULES), ...)):
We loop on every RULES and store it in _rule.
$(addsuffix -$(_rule),$(PROJECTS))
We add that rule as a prefix to each of our project. This part generate a rule with every "composed rules". With projet1 and project2 it should result in:
project1-all project2-all project1-clean project2-clean project1-fclean project2-fclean project1-re project2-re:
This way, for any of those rules name, it will be the same rule executed.
$(eval _rule := $(lastword $(subst -, ,$#)))
Here we take the target (if I call project2-clean, $# will be project2-clean), we replace - by a space to obtain project2 clean and take the last work, wich will be clean here. We then evaluate it to store that into _rule.
$(eval _proj := $(#:%-$(_rule)=%))
We use the same technique to store the project name into _proj. We just use a pattern replacement, to remove the rule name and the dash.
#$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule)
Finally, we call our submakefile we the right path and right rule!
I am VERY new to makefiles. I have discovered a flaw in a make file that causes files in a list to be copied from a single source file instead of each file in the list.
First, there is a sub model variable SUB_MODEL_LIST that contains 0 1 2 3 separated by white space.
Here is the segment that does the copy:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $< to $#
$(call COPY, $(firstword $(TARGET_BIN_LIST)), $#)
TARGET_BIN_LIST_NEW contains new file names separated by white space and is composed of something like this:
file001.200 file001.201 file001.202 file001.203
and TARGET_BIN_LIST contains the existing file names and is composed of something like this:
file001c.200 file001c.201 file001c.202 file001c.203
The last digit in the file extension is the model number.
As I read this, the makefile runs:
#echo copying from $< to $#
$(call COPY, $(firstword $(TARGET_BIN_LIST)), $#)
four times, however, it always use the first file name in the TARGET_BIN_LIST due to the firstword function. This results in file001.200, file001.201, file001.202, file001.203 being created, but they are all copies of file001c.200 when they should be copies of their respective files in the list. Each file relates to a sub model version of the code.
My thought to solve this was to use the word function. Something like this:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $< to $#
$(call COPY, $(word $(sub), $(TARGET_BIN_LIST)), $#)
where sub is an element of SUB_MODEL_LIST, but I am not sure how that will work. Does the above roll out into 4 separate calls, or can it be looked at as a loop that can have an increment value for sub??
I also thought about using a foreach loop:
$(foreach sub,$(SUB_MODEL_LIST),$(call COPY, $(word $(sub), $(TARGET_BIN_LIST)), $(word $(sub), $(TARGET_BIN_LIST_NEW)))
But I get the error:
*** first argument to `word' function must be greater than 0. Stop.
Ok, so I tried:
$(foreach sub,$(SUB_MODEL_LIST),$(call COPY, $(word $(sub)+1, $(TARGET_BIN_LIST)), $(word $(sub)+1, $(TARGET_BIN_LIST_NEW)))
But then I got the error:
*** non-numeric first argument to `word' function. Stop.
Now I'm stuck. I would like to keep the existing implementation in tact at much as possible, but can adopt a loop method if needed.
Thanks for the help!
You have to step back. You're misunderstanding how this works. In make an explicit rule with multiple targets is EXACTLY THE SAME as writing the same rule multiple times, once for each target. So this:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $< to $#
$(call COPY, $(firstword $(TARGET_BIN_LIST)), $#)
If TARGET_BIN_LIST_NEW is file001.200 file001.201 file001.202 file001.203 and TARGET_BIN_LIST is file001c.200 file001c.201 file001c.202 file001c.203, is identical to writing this:
file001.200 : file001c.200 file001c.201 file001c.202 file001c.203
...
file001.201 : file001c.200 file001c.201 file001c.202 file001c.203
...
file001.202 : file001c.200 file001c.201 file001c.202 file001c.203
...
file001.203 : file001c.200 file001c.201 file001c.202 file001c.203
...
So you can clearly see that when each rule is run, the value of $< and $(firstword $(TARGET_BIN_LIST)) will be the same thing (file001c.200).
Is it really the case that whenever ANY of the fileXXXc.YYY files change, you want to rebuild ALL the fileXXX.YYY files? That's what your rule does, but based on the recipe it doesn't seem like that's what you want.
Make is mostly about writing one rule to build one target from zero or more prerequisites. If you use a pattern rule you can do this pretty easily:
all: $(TARGET_BIN_LIST_NEW)
file001.% : file001c.%
#echo copying from $< to $#
$(call COPY,$<,$#)
If your filenames may have a more complex naming convention then you'll need something more complicated.
ETA:
Since your naming convention doesn't fit into make's pattern rule capabilities you'll have to do something fancier. You can use eval to generate the rules, like this:
all: $(TARGET_BIN_LIST_NEW)
define TARGET_BIN_COPY
$(1) : $(basename $(1))c$(suffix $(1))
#echo copying from $$< to $$#
$$(call COPY,$$<,$$#)
endef
$(foreach T,$(TARGET_BIN_LIST_NEW),$(eval $(call TARGET_BIN_COPY,$T)))
# uncomment this for debugging
#$(foreach T,$(TARGET_BIN_LIST_NEW),$(info $(call TARGET_BIN_COPY,$T)))
First off, thank you to MadScientist for your help in clarifying how this works.
This implementation worked for me:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $(filter %$(suffix $#), $(TARGET_BIN_LIST)) to $#
$(call COPY, $(filter %$(suffix $#), $(TARGET_BIN_LIST)), $#)
I have a makefile with multiple targets that are generated by copying a file from outside the working directory.
a.tex : $(wildcard /foo/work1/a.tex)
cp -p $< $#
b.tex : $(wildcard /foo/work2/b.tex)
cp -p $< $#
I use $(wildcard) because sometimes I run Make on systems that do not have access to /foo.
What is the best way to avoid repeating the cp -p $< $# commands for every rule? Some options:
Setting up a %.tex : %.tex rule. This works, but it also applies to targets that aren't specifically indicated so I get lots of warnings like make: Circular a.tex <- a.tex dependency dropped.
Defining a sequence of commands with define. This seems pointless since the command is only one line. So instead of copying cp $< $# to every rule, I'd define a cp-dep sequence and copy cp-dep to every rule.
Defining the command as a variable so that I could do a.tex : $(wildcard /foo/work1/a.tex); $(CP-DEP)
Duplicating the target names as an additional rule. a.tex b.tex : ; cp -p $< $#. Error-prone.
Just copying and pasting. Clunky but effective and easy to understand.
I haven't tested it, but can't you just use a pattern rule without prerequisites, and specify the prerequisite for each target on a separate line?
a.tex: $(wildcard /foo/work1/a.tex)
b.tex: $(wildcard /foo/work2/b.tex)
%.tex:
cp -p $< $#
Btw. doesn't the wildcard function return the empty string when it doesn't find a match, so that $< is empty as well? Wouldn't that give a problem with cp?
I think your copyrule is overkill (and inflexible). If your objection to #eriktous's solution is that it will apply the rule to targets for which you haven't explicitly defined dependencies, that's easy to fix with a static pattern rule:
a.tex: $(wildcard /foo/work1/a.tex)
b.tex: $(wildcard /foo/work2/b.tex)
blue.tex: $(wildcard /some/other/path/green.tex)
TEXES = a.tex b.tex
$(TEXES): %.tex:
cp -p $< $#
(If this solves your problem you should accept eriktous's answer-- this is just a variation on it.)
I ended up doing this:
COPYFILES = /foo/work1/a.tex /foo/work2/b.tex
define copyrule
$(notdir $(1)): $$(wildcard $(1))
cp -p $$< $$#
endef
$(foreach file,$(COPYFILES),$(eval $(call copyrule,$(file))))
The advantage of this method is that I can easily add new files with a minimum of boilerplate text and I can easily copy the rule part of this to a new Makefile. The disadvantages are that I can no longer change the destination filename, and the implementation is rather opaque for people with less makefile experience.
I have a "lib" directory in my applications main directory, which contains an arbitrary number of subdirectories, each having its own Makefile.
I would like to have a single Makefile in the main directory, that calls each subdirectory's Makefile. I know this is possible if I manually list the subdirs, but I would like to have it done automatically.
I was thinking of something like the following, but it obviously does not work. Note that I also have clean, test, etc. targets, so % is probably not a good idea at all.
LIBS=lib/*
all: $(LIBS)
%:
(cd $#; $(MAKE))
Any help is appreciated!
The following will work with GNU make:
LIBS=$(wildcard lib/*)
all: $(LIBS)
.PHONY: force
$(LIBS): force
cd $# && pwd
If there might be something other than directories in lib, you could alternatively use:
LIBS=$(shell find lib -type d)
To address the multiple targets issue, you can build special targets for each directory, then strip off the prefix for the sub-build:
LIBS=$(wildcard lib/*)
clean_LIBS=$(addprefix clean_,$(LIBS))
all: $(LIBS)
clean: $(clean_LIBS)
.PHONY: force
$(LIBS): force
echo make -C $#
$(clean_LIBS): force
echo make -C $(patsubst clean_%,%,$#) clean
There is also a way of listing sub-directories with gmake commands only, without using any shell commands:
test:
#echo $(filter %/, $(wildcard lib/*/))
This will list all sub-directories with trailing '/'. To remove it you can use the substitute pattern:
subdirs = $(filter %/, $(wildcard lib/*/))
test:
#echo $(subdirs:%/=%)
Then to actually create rules executing makefiles in each sub-directory you can use a small trick - a phony target in a non-existent directory. I think in this case an example will tell more than any explanation:
FULL_DIRS =$(filter %/, $(wildcard lib/*/))
LIB_DIRS =$(FULL_DIRS:%/=%)
DIRS_CMD =$(foreach subdir, $(LIB_DIRS), make-rule/$(subdir))
make-rule/%:
cd $* && $(MAKE)
all: DIRS_CMD
Basically, target 'all' lists all sub-directories as prerequisites. For example, if LIB_DIRS contained lib/folder1 lib/folder2 then the expansion would look like this:
all: make-rule/lib/folder1 make-rule/lib/folder2
Then 'make', in order to execute rule 'all', tries to match each prerequisite with an existing target. In this case the target is 'make-rule/%:', which uses '$*' to extract the string after 'make-rule/' and uses it as argument in the recipe. For example, the first prerequisite would be matched and expanded like this:
make-rule/lib/folder1:
cd lib/folder1 && $(MAKE)
What if you want to call different targets than all in an unknown number of subdirectories?
The following Makefile uses macros so create a forwarding dummy-target for a number of subdirectories to apply the given target from the command line to each of them:
# all direct directories of this dir. uses "-printf" to get rid of the "./"
DIRS=$(shell find . -maxdepth 1 -mindepth 1 -type d -not -name ".*" -printf '%P\n')
# "all" target is there by default, same logic as via the macro
all: $(DIRS)
$(DIRS):
$(MAKE) -C $#
.PHONY: $(DIRS)
# if explcit targets where given: use them in the macro down below. each target will be delivered to each subdirectory contained in $(DIRS).
EXTRA_TARGETS=$(MAKECMDGOALS)
define RECURSIVE_MAKE_WITH_TARGET
# create new variable, with the name of the target as prefix. it holds all
# subdirectories with the target as suffix
$(1)_DIRS=$$(addprefix $(1)_,$$(DIRS))
# create new target with the variable holding all the subdirectories+suffix as
# prerequisite
$(1): $$($1_DIRS)
# use list to create target to fullfill prerequisite. the rule is to call
# recursive make into the subdir with the target
$$($(1)_DIRS):
$$(MAKE) -C $$(patsubst $(1)_%,%,$$#) $(1)
# and make all targets .PHONY
.PHONY: $$($(1)_DIRS)
endef
# evaluate the macro for all given list of targets
$(foreach t,$(EXTRA_TARGETS),$(eval $(call RECURSIVE_MAKE_WITH_TARGET,$(t))))
Hope this helps. Really helpfull when dealing with paralelism: make -j12 clean all in a tree with makefiles having these targets... As always: playing with make is dangerous, different meta-levels of programming are too close together ,-)
I have to integrate the generation of many HTML files in an existing Makefile.
The problem is that the HTML files need to reside in many different directories.
My idea is to write an implicit rule that converts the source file (*.st) to the corresponding html file
%.html: %.st
$(HPC) -o $# $<
and a rule that depends on all html files
all: $(html)
If the HTML file is not in the builddir, make doesn't find the implicit rule: *** No rule to make target.
If I change the implicit rule like so
$(rootdir)/build/doc/2009/06/01/%.html: %.st
$(HPC) -o $# $<
it's found, but then I have to have an implicit rule for nearly every file in the project.
According to Implicit Rule Search Algorithm in the GNU make manual, rule search works like this:
Split the entire target name t into a directory part, called d, and the rest, called n. For
example, if t is src/foo.o,
then d is src/,
and n is foo.o.
Make a list of all the pattern rules one of whose targets matches t or n.
If the target pattern contains a slash,
it is matched against t;
otherwise, against n.
Why is the implicit rule not found, and what would be the most elegant solution, assuming GNU make is used?
Here is a stripped down version of my Makefile:
rootdir = /home/user/project/doc
HPC = /usr/local/bin/hpc
html = $(rootdir)/build/doc/2009/06/01/some.html
%.html: %.st
$(HPC) -o $# $<
#This works, but requires a rule for every output dir
#$(rootdir)/build/doc/2009/06/01/%.html: %.st
# $(HPC) -o $# $<
.PHONY: all
all: $(html)
The best solution I found so far is to generate an implicit rule per target directory via foreach-eval-call, as explained in the GNU make manual. I have no idea how this scales to a few thousand target directories, but we will see...
If you have a better solution, please post it!
Here is the code:
rootdir = /home/user/project/doc
HPC = /usr/local/bin/hpc
html = $(rootdir)/build/doc/2009/06/01/some.html \
$(rootdir)/build/doc/2009/06/02/some.html
targetdirs = $(rootdir)/build/doc/2009/06/01 \
$(rootdir)/build/doc/2009/06/02
define generateHtml
$(1)/%.html: %.st
-mkdir -p $(1)
$(HPC) -o $$# $$<
endef
$(foreach targetdir, $(targetdirs), $(eval $(call generateHtml, $(targetdir))))
.PHONY: all
all: $(html)
Like Maria Shalnova I like recursive make (though I disagree with "Recursive Make Considered Harmful"), and in general it's better to make something HERE from a source THERE, not the reverse. But if you must, I suggest a slight improvement: have generateHtml generate only the RULE, not the COMMANDS.
Your active implicit rule makes $(rootdir)/build/doc/2009/06/01/some.html depend on $(rootdir)/build/doc/2009/06/01/some.st. If $(rootdir)/build/doc/2009/06/01/some.st doesn't exist then the rule won't be used/found.
The commented out rule makes $(rootdir)/build/doc/2009/06/01/some.html depend on some.st.
One solution is to make you're source layout match your destination/result layout.
Another option is to create the rules as required with eval. But that will be quite complicated:
define HTML_template
$(1) : $(basename $(1))
cp $< $#
endef
$(foreach htmlfile,$(html),$(eval $(call HTML_template,$(htmlfile))))
An other possibility is to have the commando make call itself recursively with the argument -C with every output directory.
Recursive make is somewhat the standard way to deal with subdirectories, but beware of the implications mentioned in the article "Recursive Make Considered Harmful"