I'm trying to write a rule for make that effectively does "In order to make any <dir>/out, append all <dir>/<num>/source, making them if necessary". In other words, I need to expand a list of all directories <dir>/<num>/, and add source to each, placing that as the prerequisite.
My attempt is
%/out : $(addsuffix source, $(wildcard %/*/))
cat $^ > $#
But the $(wildcard %/*/) doesn't expand. % does not appear to be defined the way I would like at that point (it's a literal %) -- how can I fix this?
I have no a priori knowledge of what directories this will be run on, or how many <num> subdirectories it will find.
If you have a list of directories that can contain out files, you can generate a separate rule for each of them.
You could hard-code such a list, or generate it.
dirs = $(patsubst %/1, %, $(shell find -type d -name 1))
$(foreach %, $(dirs), $(eval $%/out: $(wildcard $%/*/source) ; cat $$^ > $$#))
I don't recommend using secondary expansion because it also would change the semantics of all other rules.
Related
I have, for example, the following Makefile to generate PDF files from Markdown files in subdirectories:
FOLDERS = f1 f2 f3
.PHONY: $(FOLDERS)
f1: f1/f1.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
f2: f2/f2.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
f3: f3/f3.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
The expected result is that make f1 requires the existence of f1/f1.md, and generates the resulting PDF as f1/f1.pdf. The same for f2 and f3. This works, but the declarations seem unnecessarily repetitive.
Is there any way to combine these three rules into one, generic rule? That is, without needing to explicitly write out all of the paths to the PDF files or Markdown files, as I may be dynamically adding subfolders and I'd prefer to just change the definition of FOLDERS in the first line. I've googled around and tried a few things, but I feel like either I can't find the right incantation to use, or I'm missing a piece of knowledge about how Makefiles work. Could someone please point me in the right direction?
First, note that there's no good reason to use PHONY targets here, since these rules appear to be building files whose names are known beforehand. Targets like f1/f1.pdf would be much better.
Unfortunately we can't use a pattern rule when the stem (e.g. f1) is repeated in a prerequisite. But a "canned recipe" can do the trick:
define pdf_template
$(1): $(1)/$(1).md
cd $$# && pandoc $$(notdir $$^) -o $$(patsubst %.md,%.pdf,$$(notdir $$^))
endef
$(eval $(call pdf_template,f1))
$(eval $(call pdf_template,f2))
$(eval $(call pdf_template,f3))
(Note how you must escape the $ signs in the template.)
If those $(eval...) lines look too repetitive, you can replace them with a loop:
$(foreach folder,$(FOLDERS),$(eval $(call pdf_template,$(folder))))
EDIT: Come to think of it, there's another way. You can't construct a pattern rule that uses the stem more than once:
$(FOLDERS): %: %/%.md
cd $# && ... this won't work
And you can't use the automatic variables in the prerequisite list, because they aren't yet defined when they're needed:
$(FOLDERS): $#/$#.md
cd $# && ... this won't work either
But you can use them there if you use Secondary Expansion, which causes Make to expand the prereq list a second time:
.SECONDEXPANSION:
$(FOLDERS): $$#/$$#.md
cd $# && ... this works
Again, note the escaped $ symbols.
I have a Makefile which I use to call different sub-Makefiles.
I have several rules:
all
clean
fclean
re
I can already use those rules, which will call every sub makefile with the same rule.
I have several project, and I would like to generate rules with that format:
$(PROJECT_NAME)-$(RULES)
With that, I would like to have each rule for each project:
project1-all
project1-clean
...
project2-all
project2-clean
...
This way, I would be able to call a specific rule, for a specific project, like project1-fclean.
I tried that:
RULES= all clean fclean re
PROJECTS= project1 project2
define NEWLINE
endef
$(foreach _rule, $(RULES), \
$(foreach _proj, $(PROJECTS), \
$(_proj)-$(_rule): $(NEWLINE) \
$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule) $(NEWLINE) \
) \
)
But it doesn't seems to work. I have searched, but I haven't found advanced makefile techniques to achieve that. Plz help.
The problem is that when you combine lines together with line continuations like that, it compresses out all the newlines and other extraneous whitespace (including those newlines you're trying to insert with $(NEWLINE)) resulting in a huge mess on a single line, rather than multiple lines with multiple patterns. To do this correctly, you need to write your rule as a macro with arguments and then call it:
define PROJ_RULE
$(1)-$(2):
$(MAKE) $(ARGS) $(PROJECT_DIR)$(1) $(2)
endef
$(foreach _rule, $(RULES),
$(foreach _proj, $(PROJECTS),
$(eval $(call PROJ_RULE, $(_proj), $(_rule)))))
note that all this define and foreach stuff in GNU make specific -- other make flavors do not support it.
Okay, so I finally managed to do it this way:
$(foreach _rule, $(RULES), $(addsuffix -$(_rule),$(PROJECTS))):
$(eval _rule := $(lastword $(subst -, ,$#)))
$(eval _proj := $(#:%-$(_rule)=%))
#$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule)
I will decompose it for a better explanation:
$(foreach _rule, $(RULES), ...)):
We loop on every RULES and store it in _rule.
$(addsuffix -$(_rule),$(PROJECTS))
We add that rule as a prefix to each of our project. This part generate a rule with every "composed rules". With projet1 and project2 it should result in:
project1-all project2-all project1-clean project2-clean project1-fclean project2-fclean project1-re project2-re:
This way, for any of those rules name, it will be the same rule executed.
$(eval _rule := $(lastword $(subst -, ,$#)))
Here we take the target (if I call project2-clean, $# will be project2-clean), we replace - by a space to obtain project2 clean and take the last work, wich will be clean here. We then evaluate it to store that into _rule.
$(eval _proj := $(#:%-$(_rule)=%))
We use the same technique to store the project name into _proj. We just use a pattern replacement, to remove the rule name and the dash.
#$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule)
Finally, we call our submakefile we the right path and right rule!
I have a directory structure like
packages/
foo/
lib/
a.js
b.js
bar/
lib/
c.js
d.js
and I'm trying to write a simple Makefile to run the source files through a compiler and output the result of compiling packages/foo/lib/a.js to packages/foo/build/a.js. In the past, I've done something like
JS = $(shell find packages/foo/lib -name "*.js")
BUILD = $(patsubst packages/foo/lib/%.js, packages/foo/build/%.js, $(JS))
.PHONY: all
all: $(BUILD)
packages/foo/lib/%.js: packages/foo/build/%.js
# Compile $^ and output to $#
which works great. However, I'm now doing
JS = $(shell find packages/*/lib -name "*.js")
BUILD = $(patsubst ...)
The problem here is that patsubst doesn't seem to like multiple % wildcards (ie packages/%/lib/%.js). However, if I use packages/%.js, then I can't know which directory to use in the substitution. I'm convinced there's a very simple way to do this that I can't find in the make docs.
To define BUILD, you could use the stupid but simple solution:
BUILD = $(subst /lib/,/build/,$(JS))
as long as you aren't worried about additional "lib" and "build" pathname components in your source.
However, this leaves you with the problem of how to define the actual Makefile rules, unless you want to manually specify a rule for each directory foo, bar, etc.
Instead, maybe try something like this (remember to replace spaces with tabs again where appropriate, if you cut and paste):
PACKAGES = $(wildcard packages/*)
ALL :=
define add_package
JS := $$(shell find $(1)/lib -name "*.js")
BUILD := $$(patsubst $(1)/lib/%.js, $(1)/build/%.js, $$(JS))
$(1)/build/%.js:: $(1)/lib/%.js
echo COMPILE $$^ '->' $$#
cp $$^ $$# # super fast compiler!
ALL += $$(BUILD)
endef
$(foreach p, $(PACKAGES), \
$(eval $(call add_package,$(p))))
all: $(ALL)
This evaluates a template for each package directory, so you can put anything in there, as long as you get the fiddly doubling of dollar signs correct.
I have a "lib" directory in my applications main directory, which contains an arbitrary number of subdirectories, each having its own Makefile.
I would like to have a single Makefile in the main directory, that calls each subdirectory's Makefile. I know this is possible if I manually list the subdirs, but I would like to have it done automatically.
I was thinking of something like the following, but it obviously does not work. Note that I also have clean, test, etc. targets, so % is probably not a good idea at all.
LIBS=lib/*
all: $(LIBS)
%:
(cd $#; $(MAKE))
Any help is appreciated!
The following will work with GNU make:
LIBS=$(wildcard lib/*)
all: $(LIBS)
.PHONY: force
$(LIBS): force
cd $# && pwd
If there might be something other than directories in lib, you could alternatively use:
LIBS=$(shell find lib -type d)
To address the multiple targets issue, you can build special targets for each directory, then strip off the prefix for the sub-build:
LIBS=$(wildcard lib/*)
clean_LIBS=$(addprefix clean_,$(LIBS))
all: $(LIBS)
clean: $(clean_LIBS)
.PHONY: force
$(LIBS): force
echo make -C $#
$(clean_LIBS): force
echo make -C $(patsubst clean_%,%,$#) clean
There is also a way of listing sub-directories with gmake commands only, without using any shell commands:
test:
#echo $(filter %/, $(wildcard lib/*/))
This will list all sub-directories with trailing '/'. To remove it you can use the substitute pattern:
subdirs = $(filter %/, $(wildcard lib/*/))
test:
#echo $(subdirs:%/=%)
Then to actually create rules executing makefiles in each sub-directory you can use a small trick - a phony target in a non-existent directory. I think in this case an example will tell more than any explanation:
FULL_DIRS =$(filter %/, $(wildcard lib/*/))
LIB_DIRS =$(FULL_DIRS:%/=%)
DIRS_CMD =$(foreach subdir, $(LIB_DIRS), make-rule/$(subdir))
make-rule/%:
cd $* && $(MAKE)
all: DIRS_CMD
Basically, target 'all' lists all sub-directories as prerequisites. For example, if LIB_DIRS contained lib/folder1 lib/folder2 then the expansion would look like this:
all: make-rule/lib/folder1 make-rule/lib/folder2
Then 'make', in order to execute rule 'all', tries to match each prerequisite with an existing target. In this case the target is 'make-rule/%:', which uses '$*' to extract the string after 'make-rule/' and uses it as argument in the recipe. For example, the first prerequisite would be matched and expanded like this:
make-rule/lib/folder1:
cd lib/folder1 && $(MAKE)
What if you want to call different targets than all in an unknown number of subdirectories?
The following Makefile uses macros so create a forwarding dummy-target for a number of subdirectories to apply the given target from the command line to each of them:
# all direct directories of this dir. uses "-printf" to get rid of the "./"
DIRS=$(shell find . -maxdepth 1 -mindepth 1 -type d -not -name ".*" -printf '%P\n')
# "all" target is there by default, same logic as via the macro
all: $(DIRS)
$(DIRS):
$(MAKE) -C $#
.PHONY: $(DIRS)
# if explcit targets where given: use them in the macro down below. each target will be delivered to each subdirectory contained in $(DIRS).
EXTRA_TARGETS=$(MAKECMDGOALS)
define RECURSIVE_MAKE_WITH_TARGET
# create new variable, with the name of the target as prefix. it holds all
# subdirectories with the target as suffix
$(1)_DIRS=$$(addprefix $(1)_,$$(DIRS))
# create new target with the variable holding all the subdirectories+suffix as
# prerequisite
$(1): $$($1_DIRS)
# use list to create target to fullfill prerequisite. the rule is to call
# recursive make into the subdir with the target
$$($(1)_DIRS):
$$(MAKE) -C $$(patsubst $(1)_%,%,$$#) $(1)
# and make all targets .PHONY
.PHONY: $$($(1)_DIRS)
endef
# evaluate the macro for all given list of targets
$(foreach t,$(EXTRA_TARGETS),$(eval $(call RECURSIVE_MAKE_WITH_TARGET,$(t))))
Hope this helps. Really helpfull when dealing with paralelism: make -j12 clean all in a tree with makefiles having these targets... As always: playing with make is dangerous, different meta-levels of programming are too close together ,-)
I have to integrate the generation of many HTML files in an existing Makefile.
The problem is that the HTML files need to reside in many different directories.
My idea is to write an implicit rule that converts the source file (*.st) to the corresponding html file
%.html: %.st
$(HPC) -o $# $<
and a rule that depends on all html files
all: $(html)
If the HTML file is not in the builddir, make doesn't find the implicit rule: *** No rule to make target.
If I change the implicit rule like so
$(rootdir)/build/doc/2009/06/01/%.html: %.st
$(HPC) -o $# $<
it's found, but then I have to have an implicit rule for nearly every file in the project.
According to Implicit Rule Search Algorithm in the GNU make manual, rule search works like this:
Split the entire target name t into a directory part, called d, and the rest, called n. For
example, if t is src/foo.o,
then d is src/,
and n is foo.o.
Make a list of all the pattern rules one of whose targets matches t or n.
If the target pattern contains a slash,
it is matched against t;
otherwise, against n.
Why is the implicit rule not found, and what would be the most elegant solution, assuming GNU make is used?
Here is a stripped down version of my Makefile:
rootdir = /home/user/project/doc
HPC = /usr/local/bin/hpc
html = $(rootdir)/build/doc/2009/06/01/some.html
%.html: %.st
$(HPC) -o $# $<
#This works, but requires a rule for every output dir
#$(rootdir)/build/doc/2009/06/01/%.html: %.st
# $(HPC) -o $# $<
.PHONY: all
all: $(html)
The best solution I found so far is to generate an implicit rule per target directory via foreach-eval-call, as explained in the GNU make manual. I have no idea how this scales to a few thousand target directories, but we will see...
If you have a better solution, please post it!
Here is the code:
rootdir = /home/user/project/doc
HPC = /usr/local/bin/hpc
html = $(rootdir)/build/doc/2009/06/01/some.html \
$(rootdir)/build/doc/2009/06/02/some.html
targetdirs = $(rootdir)/build/doc/2009/06/01 \
$(rootdir)/build/doc/2009/06/02
define generateHtml
$(1)/%.html: %.st
-mkdir -p $(1)
$(HPC) -o $$# $$<
endef
$(foreach targetdir, $(targetdirs), $(eval $(call generateHtml, $(targetdir))))
.PHONY: all
all: $(html)
Like Maria Shalnova I like recursive make (though I disagree with "Recursive Make Considered Harmful"), and in general it's better to make something HERE from a source THERE, not the reverse. But if you must, I suggest a slight improvement: have generateHtml generate only the RULE, not the COMMANDS.
Your active implicit rule makes $(rootdir)/build/doc/2009/06/01/some.html depend on $(rootdir)/build/doc/2009/06/01/some.st. If $(rootdir)/build/doc/2009/06/01/some.st doesn't exist then the rule won't be used/found.
The commented out rule makes $(rootdir)/build/doc/2009/06/01/some.html depend on some.st.
One solution is to make you're source layout match your destination/result layout.
Another option is to create the rules as required with eval. But that will be quite complicated:
define HTML_template
$(1) : $(basename $(1))
cp $< $#
endef
$(foreach htmlfile,$(html),$(eval $(call HTML_template,$(htmlfile))))
An other possibility is to have the commando make call itself recursively with the argument -C with every output directory.
Recursive make is somewhat the standard way to deal with subdirectories, but beware of the implications mentioned in the article "Recursive Make Considered Harmful"