I'm trying to make a make file for a static page generator, and I'm using jinja and pandoc so far, and the idea is to have a file structure as such
.
|-content
|-public
|-templates
|-Makefile
VPATH=public
TARGETS=$(find content -regex ".*(htm|md)" | sed -e "s/md$/htm/g;s/^content/public/g")
all: $(TARGETS)
#echo fullbuild
public/%: content/%
content/%.md:
# Pandoc script
pandoc -i $# -o ${${#:.md=.htm}:content=public}
content/%.htm:
# Jinja Script
The problem I'm having (At least I think that's it) is that according to me the syntax is
# For a final result
target: dependency
commands
# A rule for dependency
dependency:
commands
My dependencies are in the content dir and my targets are in the public dir that may or may not exist yet, and almost all the files I generate will be htm files, and in that case if the target is public/some/route/to/file.htm the dependency will be any of this two content/some/route/to/file.(htm|md).
I can easily generate by walking the content dir, and changing extensions.
How should I write the rules properly, so
- Make knows where to "watch" for changes every time I do a make, because right now it points that every file is up to date
- How do I indicate properly the dependency of a file to it's content file.
This rule:
public/%: content/%
does nothing because pattern rules without recipes delete existing pattern rules, they don't define new pattern rules.
It's very simple, you should write two rules like this:
public/%.htm: content/%.md:
# Pandoc script
pandoc -i $< -o $#
public/%.htm: content/%.htm
# Jinja Script
Here's a hint: whenever you're writing a makefile recipe and you discover that you need to create a target which is different than exactly $#, unmodified, immediately stop what you're doing and back up: you've taken a wrong turn.
Related
I am learning makefiles, and can't just wrap my head around this problem i am having, and would like to understand how/why this fail.
I have half a dozen erlang files in a src directory. I want to compile these into a ebin directory, without having to define a rule for each and every one of them. According to the Gnu make documentation, pattern rules should be right up my alley.
However, with the following makefile, all I get from make is make: *** No targets. Stop. Why is that?
ebin/%.beam: src/%.erl
mkdir -p ebin
erlc -o ebin $<
Edit: Based on this answer, I now understand that i would have to explicitly declare the targets, for instance by using make ebin/cmplx.beam. However, i still do not understand how i should write my makefile to get my desired behaviour - since I have half a dozen targets (and in other projects even more), this seems like an unnecessary hassle. Is there not a way to define targets based on the source file names?
The target rule tells make that whenever it needs to produce a beam file in the ebin directory, and there exists a corresponding erl file in the src directory, it can use erlc.
However, this doesn't tell make that this is what it needs to do. You could explicitly tell make what it needs to do by giving it a target on the command line:
make ebin/foo.beam
If you don't give a target on the command line, make will pick the first non-pattern rule in the makefile as its target. However, your makefile doesn't have any non-pattern rules, so there is no target.
What you probably want is that for each existing erl file in src, make should consider the corresponding beam file in ebin to be a target. You can achieve that by calling wildcard and patsubst:
erl_files=$(wildcard src/*.erl)
beam_files=$(patsubst src/%.erl,ebin/%.beam,$(erl_files))
ebin/%.beam: src/%.erl
mkdir -p ebin
erlc -o ebin $<
all: $(beam_files)
(The indented lines need to be actual physical tabs, not spaces.)
That way, running make will rebuild all beam files that are out of date. all gets chosen as the default target, and it in turn depends on all beam existing or potential, each of which in turn depends on the corresponding erl file.
This trick is described in the GNU make manual.
I'm trying to write a Makefile where multiple source files (in my case they are markdown) create multiple target files (pdfs). However, the target files generated have extra characters in the file name that can't be predicted (it happens to be a version number encoded in the source), but ideally the Makefile would not have to read the source itself.
So, for example:
file1.md => file1-v1.pdf
file2.md => file2-v2.pdf
...
I can calculate source name given a target name (by excluding anything after the hyphen and adding .md), but cannot calculate target name given the source.
Is it possible to write a Makefile that builds only the targets where the source have been updated?
This will be ugly, but it will work.
As it often is with Make, our problem divides into these two problems:
1. construct a list of targets
2. build them
Suppose we have five md files which map to pdf files (whose names we don't know beforehand):
file1.md => file1-v1.pdf
file2.md => file2-v1.pdf
file3.md => file3-v1.pdf
file4.md => file4-v1.pdf
file5.md => file5-v1.pdf
We can't use the real output file names as targets, because we don't know them beforehand, but we see five input files and know that we must build one output file for each. For now, a fake target name will do:
file1-dummy.pdf: file1.md
zap file1.md
When Make executes this rule, it produces the file file1-v1.pdf. The fact that it doesn't produce a file named file1-dummy.pdf is disquieting, but not a serious problem. We can turn this into a pattern rule:
%-dummy.pdf: %.md
zap $<
Then all we have to do is turn the list of existing input files (file1.md, file2.md, ...) into a list of dummy targets (file1-dummy.pdf, file2-dummy.pdf, ...), and build them. So far, so good.
But suppose some of the output files already exist. If file2-v2.pdf already exists -- and is newer than file2.md -- then we would prefer that Make not rebuild it (by attempting to build file2-dummy.pdf). In that case we would prefer that file2-v2.pdf be in the target list, with a rule that worked like this:
file2-v2.pdf: file2.md
zap $<
This is not easy to turn into a pattern rule, because Make does not handle wildcards very well, and cannot cope with multiple wildcards in a single phrase, not without a lot of clumsiness. But there is a way to write one rule that will cover both cases. First note that we can obtain the part of a variable before the hyphen with this kludge:
$(basename $(subst -,.,$(VAR)))
Armed with this, and with secondary expansion, we can write a pattern rule that will work with both cases, and construct a target list that will exploit it:
# There are other ways to construct these two lists, but this will do.
MD := $(wildcard *.md)
PDF := $(wildcard *.pdf)
PDFROOTS := $(basename $(subst -,.,$(basename $(PDF))))
MDROOTS := $(filter-out $(PDFROOTS), $(basename $(MD)))
TARGETS:= $(addsuffix -foo.pdf, $(MDROOTS)) $(PDF)
.SECONDEXPANSION:
%.pdf: $$(basename $$(subst -,., $$*)).md
# perform actions on $<
Make's algorithm always starts with the final output product and works its way backwards to the source files, to see what needs to be updated.
Therefore, you HAVE to be able to enumerate the final output product as a target name and correlate that back to the inputs that generate that output, for make to work.
This is also why make is not a great tool for building Java, for example, since the output filenames don't map easily to the input file names.
So, you must have at least one target/prerequisite pair which is derivable (for implicit rules), or state-able (for explicit rules)--that is, known at the time you write the makefile. If you don't then a marker file is your only alternative. Note you CAN add extra generated, non-derivative prerequisites (for example, in compilers you can add header files as prerequisites that are not related to the source file name), in addition to the known prerequisite.
#Beta's answer is informative and helpful, but I needed a solution (using GNU Make 4.1) that worked when the destination filename bears no resemblance to the input filename, for example, if it is generated from its content. I came up with the following, which takes every file matching *.in, and creates a file by reading the contents of the source file, appending a .txt, and using it as a filename to create. (For example, if test.in exists and contains foo, the makefile will create a foo.txt file.)
SRCS := $(wildcard *.in)
.PHONY: all
all: all_s
define TXT_template =
$(2).txt: $(1)
touch $$#
ALL += $(2).txt
endef
$(foreach src,$(SRCS),$(eval $(call TXT_template, $(src), $(shell cat $(src)))))
.SECONDARY_EXPANSION:
all_s: $(ALL)
The explanation:
The define block defines the recipe needed to make the text file from the .in file. It's a function that takes two parameters; $(1) is the .in. file and $(2) is the contents of it, or the base of the output filename. Replace touch with whatever makes the output. We have to use $$# because eval will expand everything once, but we want $# to left after this expansion. Since we have to collect all the generated targets so we known what all the make, the ALL line accumulates the targets into one variable. The foreach line goes through each source file, calls the function with the source filename and the contents of the file (i.e. what we want to be the name of the target, here you'd normally use whatever script generates the desired filename), and then evaluates the resulting block, dynamically adding the recipe to make. Thanks to Beta for explaining .SECONDARY_EXPANSION; I needed it for reasons not entirely clear to me, but it works (putting all: $(ALL) at the top doesn't work). The all: at the top depends on the secondary expansion of all_s: at the bottom and somehow this magic makes it work. Comments welcome.
maybe try this ? or something along those lines
# makefile
SRCS=$(wildcard *.md)
PDFS=$(shell printf *.pdf)
$(PDFS): $(SRCS)
command ...
the printf *.pdf is meant to either expand to the first of the pdf files if they exist, else fail if they don't and that will signal to make that it should build. if this doesn't work i suggest maybe experimenting with find, ls or other listing tools (e.g. compgen, complete), maybe even in combination with xargs to get everything on one line.
Is there a generic was to specify a rule for something that goes from something with an extension to something with no extension? (in this case, '.in' to '')?
I have a project that's using automake. I have some config file templates that need substitutions before installing them. For example, I have a 'foo.ini.in' file that I want to use to create 'foo.ini'. I have a few of these scattered throughout the codebase. However, for each one, I need to create a custom Makefile rule to make them work. Is there a better way? I currently do this in Makefile.am:
bazdir=$(libdir)/#PACKAGE_NAME#/baz
EXTRA_DIST = foo.ini.in bar.ini.in
CLEANFILES = foo.ini bar.ini
foo.ini: foo.ini.in
sed -e 's|#LIBDIR#|$(bazdir)|g' $< > $#
bar.ini: bar.ini.in
sed -e 's|#LIBDIR#|$(bazdir)|g' $< > $#
This works fine, but I need to duplicate the rule for every file. I'd like to write one rule that will do the substitution for any 'ini.in' files to 'ini' files. I've tried:
%.ini.in: %.ini
sed ....
but autoconf complains that % pattern substitution is a non-portable gnumake-ism. It also doesn't like
.ini.in.ini:
sed .....
.. which I can't really blame it for not liking, because I can't even parse that.
Is there a way to do this?
NOTE: I can not use AC_CONFIG_FILES to do this substitution, because 'bazdir' is not fully expanded in that case.
Fortunately, there's a much better way: let the 'configured' files be generated from the templates using the configure script itself. As I mentioned in the comment, I don't see any reason to try to expand $bazdir in the Makefile.am itself. Maybe you can clear this up for me if there's some special reason for this.
In the configure.ac script, substitution variables are defined with the AC_SUBST macro. I'm guessing you want to replace LIBDIR with the value of $bazdir. Note that LIBDIR isn't a good choice of name, as libdir is already used in configure scripts; so let's use a variable name prepended with a project name: BAZ_LIBDIR
1/. set bazdir in the configure script: bazdir="${libdir}/${PACKAGE_NAME}/baz", and substitute with: AC_SUBST(BAZ_LIBDIR, $bazdir). Alternatively, just assign the value to BAZ_LIBDIR and use the single argument form of AC_SUBST. BAZ_LIBDIR is now ready for substitution...
2/. At the end of configure.ac, specify the files to be generated from their <file>.in templates, with AC_CONFIG_FILES. Typically this will list Makefile's, as well as data files, where the .in suffix is implicit.
Assuming a projectdir tree... could be any sort of tree layout of course:
BAZ_LIBDIR="${libdir}/${PACKAGE_NAME}/baz"
AC_SUBST(BAZ_LIBDIR)
...
AC_CONFIG_FILES([Makefile projectdir/Makefile])
AC_CONFIG_FILES([projectdir/foo.ini projectdir/bar.ini])
...
AC_OUTPUT
Instances of #BAZ_LIBDIR# in the <file>.ini.in files will be replaced with the substitution value. No arcane sed invocations are required to generate <file>.ini files.
Another nice feature is that you needn't add foo.ini.in or bar.ini.in to the EXTRA_DIST variable in the Makefile.am - and make distclean will clean the bar.ini and foo.ini files.
Assume I have a make rule:
.PHONY:gen
gen: auto.template
generate-sources auto.template
that creates a bunch of files, for example auto1.src, auto2.src, auto3.src and so on.
If I now have rules to build targets from *.src files, like this:
$(patsubst %.src,%.target,$(wildcard *.src)): %.target: %.src
build $< > $#
How can I tell make to first execute the gen rule and then expand the preconditions for the second rule template? GNU extensions are welcome.
Note: I would like to keep it in one make invocation; A trivial solution to this would be to put the second rule in a secondary Makefile.secondrun and call $(MAKE) -f Makefile.secondrun after gen was processed. But I was wondering if there is a better option.
Building off Beta's answer, here's how you can do it using makefile remaking in GNU make, which is not the same thing as recursive make. Rather, it updates an included makefile using a rule in the main makefile, then restarts the original make instance. This is how *.d dependency files are typically generated and used.
# Get the list of auto-generated sources. If this file doesn't exist, or if it is older
# than auto.template, it will get built using the rule defined below, according to the
# standard behavior of GNU make. If autosrcs.mk is rebuilt, GNU make will automatically
# restart itself after autosrcs.mk is updated.
include autosrcs.mk
# Once we have the list of auto-generated sources, getting the list of targets to build
# from them is a simple pattern substitution.
TARGETS=$(patsubst %.src,%.target,$(AUTO_SRCS))
all: $(TARGETS)
# Rule describing how to build autosrcs.mk. This generates the sources, then computes
# the list of autogenerated sources and writes that to autosrcs.mk in the form of a
# make variable. Note that we use *shell* constructs to get the list of sources, not
# make constructs like $(wildcard), which could be expanded at the wrong time relative
# to when the source files are actually created.
autosrcs.mk: auto.template
./generate-sources auto.template
echo "AUTO_SRCS=`echo *.src`" > autosrcs.mk
# How to build *.target files from *.src files.
%.target: %.src
#echo 'build $< > $#'
Short answer: you can't. Make determines all of the rules it will have to execute before it executes any rule.
Longer answer: maybe you can. As you say, you can use recursive Make explicitly, or surreptitiously by, say, building a file which your makefile will include (I'm looking at you, Jack Kelly). Or if you could somehow obtain a list of the files which gen will build, you could write a rule around that. Or you could take a leap of faith like this:
%.target: %.src
build $< > $#
%.src: gen;
I have a Makefile with the following type of rule:
%.html:
./generate-images.py > $#
make $(patsubst %.png,%.gif,$(wildcard *.png))
The generate-images script writes not only the HTML file (to stdout) but several .png files to the current directory. The goal here is to convert them to .gif. (not really, but this is an example)
This works if I invoke it directly. The problem is: If I invoke it from another rule where foo.html is a dependency, the wildcard statement fails to find any files. In other words, it just called make with no arguments, which is not what I want here.
What's the deal with the wildcard? Or, is there a better way to do this?
While your problem may be something different, I clearly see one.
The whole text of all commands within the rule is simultaneously processed so that make's functions and variables get expanded. Assume you have no .png files in the directory, and you invoke make so it should regenerate them: a.png and b.png. Then, after you invoke make, the text of the rule would effectively look like this:
file.html:
./generate-images.py > file.html
make
because at the moment of reading the makefile there were no .png files! After the first line is executed, the files will appear, but the next line was already generated to be just "make".
And only when you invoke your makefile for the second time, will it expand to
file.html:
./generate-images.py > file.html
make a.gif b.gif
This is not what you want. So I suggest doing it in The Right Way.
# If you have batch conversion program, this may be helpful
images.stamp: *.png
convert_all_images $?
touch images.stamp
# OR, if you want convert one-by-one with means of make
images.stamp: $(wildcard *.png)
touch images.stamp
%.gif: %.png
convert_one --from=$^ --to=$#
# HTML would look like
%.html:
./generate-images.py > $#
make images.stamp
So when you invoke make all, it generates htmls and converts newly generated images. Note that it will only convert the images that are updated, which is what you want.
Thanks to Beta for pointing out the mess with gif/png extensions.
That sounds like it's evaluating all of the $() expressions as it's processing the Makefile, rather than as it executes each rule. You could add a rule to your makefile like so:
images: $(patsubst %.png,%.gif,$(wildcard *.png))
.PHONY: images
and then change your example snippet to
%.html:
./generate-images.py > $#
make images
so that Make evaluates the glob at the right time. This is something about which checking the manual might be worthwhile.