I have created a makefile for the generation of a simple web page. The idea behind the makefile is this:
We're compiling one web page, index.html
index.html requires a stylus css main.sty that must be compiled
There are a number of examples used in the page
The code for example one lives in lib/examples/one
Each example contains three parts
The markup (a .jade template file)
Some code (a .coffee script file)
A description (a .md markdown file)
The build script must render each example into a single html file
Jade, Pygments, and Markdown are used to generate three html files
An example.jade template is used to combine these into one example file
example.jade must be copied to the correct build example directory, because the template language can only do relative imports. So in order to import example/one/code.html, we must copy the template to example/one and have it include code.html.
When finished, each example x will have compiled to tbuild/examples/x.html
The lib/index.jade template is moved to build (so that it can include the example files)
Jade is then used to compile the index.jade template into html
This is a wee bit of a simplification, but it's easier to understand this way. The simplification is that there are actually two markup files (left.html and right.html) in each example, and that the code file is both run through pygments and used as a script, so both code.html and code.coffee need to make it into build.
Right now, the makefile looks like this:
LIB = lib
BUILD = build
LIBEX = $(LIB)/examples
BUILDEX = $(BUILD)/examples
EXAMPLES = $(addsuffix .html,$(addprefix $(BUILDEX)/,$(shell ls $(LIBEX) | grep -v '.jade')))
all: $(BUILD)/main.css index.html
index.html: $(BUILD)/index.jade $(EXAMPLES)
jade < $< --path $< > $#
$(BUILD)/index.jade: $(LIB)/index.jade
mkdir -p $(#D)
cp $< $#
$(BUILD)/main.css: $(LIB)/main.sty
mkdir -p $(#D)
stylus -u nib < $< > $#
$(BUILDEX)/%.html: $(BUILDEX)/%/template.jade $(BUILDEX)/%/left.html $(BUILDEX)/%/right.html $(BUILDEX)/%/code.html $(BUILDEX)/%/code.coffee $(BUILDEX)/%/text.html
jade < $< --path $< > $#
$(BUILDEX)/%/template.jade: $(LIBEX)/template.jade
mkdir -p $(#D)
cp $< $#
$(BUILDEX)/%/left.html: $(LIBEX)/%/left.jade
jade < $< --path $< > $#
$(BUILDEX)/%/right.html: $(LIBEX)/%/right.jade
jade < $< --path $< > $#
$(BUILDEX)/%/code.html: $(LIBEX)/%/code.coffee
pygmentize -f html -o $# $<
$(BUILDEX)/%/code.coffee: $(LIBEX)/%/code.coffee
mkdir -p $(#D)
cp $< $#
$(BUILDEX)/%/text.html: $(LIBEX)/%/text.md
markdown < $< > $#
clean:
rm index.html -f
rm $(BUILD) -rf
This works, but the problem is that when I touch "lib/examples/intro/code.coffee" and re-run make, I get the following:
mkdir -p build/examples/intro
cp lib/examples/template.jade build/examples/intro/template.jade
jade < lib/examples/intro/left.jade --path lib/examples/intro/left.jade > build/examples/intro/left.html
jade < lib/examples/intro/right.jade --path lib/examples/intro/right.jade > build/examples/intro/right.html
pygmentize -f html -o build/examples/intro/code.html lib/examples/intro/code.coffee
mkdir -p build/examples/intro
cp lib/examples/intro/code.coffee build/examples/intro/code.coffee
markdown < lib/examples/intro/text.md > build/examples/intro/text.html
jade < build/examples/intro/template.jade --path build/examples/intro/template.jade > build/examples/intro.html
jade < build/index.jade --path build/index.jade > index.html
rm build/examples/intro/right.html build/examples/intro/code.coffee build/examples/intro/code.html build/examples/intro/left.html build/examples/intro/text.html build/examples/intro/template.jade
Which, as you'll notice, is way more than needs to be done to regenerate the example. What I was hoping for was something more like this:
mkdir -p build/examples/intro
pygmentize -f html -o build/examples/intro/code.html lib/examples/intro/code.coffee
cp lib/examples/intro/code.coffee build/examples/intro/code.coffee
jade < build/examples/intro/template.jade --path build/examples/intro/template.jade > build/examples/intro.html
jade < build/index.jade --path build/index.jade > index.html
In other words, what I'm asking is:
What do I need to do to make the makefile not rebuild too much when I change something small?
In the above example, left.html, right.html, and text.html are all rebuilt when I touch code.coffee. How can I prevent this?
Why does make put the rm command at the end of the output? This seems like it might be causing some re-building where unnecessary.
Thanks for reading all the way through this beast of a question! I know that my make-fu is lacking, so any tips as to how to clean up the makefile and reduce redundancy are more than welcome!
This build system is too large and complex to reproduce easily -- and I hate to post solutions I haven't tested -- but try adding this line:
.SECONDARY:
EDIT:
I haven't been able to reproduce the behavior you describe, but I can offer some pointers.
The .SECONDARY: is a rule; it can go anywhere in the makefile. Basically, if Make detects a chain of chain of implicit rules, A->B->C, where A is a file that exists and C is the target, it considers B an intermediate file and will delete it once the job is done. The .SECONDARY: rule blocks the deletion.
You can combine rules that have the same commands. This:
foo: bar
do something $< $#
baz: quartz
do something $< $#
quince: geef
do something $< $#
can be rewritten as this:
foo: bar
baz: quartz
quince: geef
foo baz quince:
do something $< $#
That will remove a lot of redundancy in your makefile, and perhaps make things clearer.
Like #Beta mentioned, you are running into troubles with intermediate files here.
Each file that is not mentioned explicitely in the Makefile, but rather inferred via a pattern rule (a rule with %), is intermediate and is deleted right after make ran. And then remade on the next run.
That's the core of your "problem": The left.html and right.html files are deleted right away. Beta already mentioned that declaring them as .SECONDARY fixes this problem, like the make manual tells you.
By declaring a target .SECONDARY without any prerequisites anywhere in the Makefile (really: anywhere), you declare all targets as secondary - so, no auto-deletes.
You could also use PHONY for targets that are not files to improve the performance. Like all and clean. See here for more information: What is the purpose of .PHONY in a makefile?
Related
I have a series of directories organized like this:
foo/
foo.file1 foo.file2
bar/
bar.file1 bar.file2
baz/
baz.file1 baz.file2
Right now I'm processing these files using a script that does all the checking for file existence etc but I thought that perhaps I could use a Makefile for it (since said script is very fragile), to avoid reprocessing files that did not change.
The problem is that each directory is independent, and I'd need to do, for example:
foo.file1.processed: foo.file1
run_random_program foo.file1 -o foo.file1.processed
for each of the 71 directories that are in total in that path. This looks like being extremely tedious and I wonder if there's something that would prevent me from writing all of this by hand.
Is such a thing possible?
EDIT: Some examples that show what I have in mind, had I a single Makefile for each directory:
file1.cds.callable: file1.callable
long_script_name -i $< -o $#
file1.rds: file1.cds.callable
another_long_script_name $< additional_file_in_folder $#
file1.csv: file1.rds
yet_another_script $< $#
Seems like pattern rules are exactly what you need:
# These are the original source files (based on the example)
CALLABLE := $(wildcard */*.callable)
# These are the final targets
TARGETS := $(CALLABLE:%.callable=%.csv)
all: $(TARGETS)
%.csv : %.rds
yet_another_script $< $#
%.rds: %.cds.callable
another_long_script_name $< additional_file_in_folder $#
%.cds.callable: %.callable
long_script_name -i $< -o $#
I have a Makefile in which a couple of targets need to be made the same way, but one of the targets is a file whose basename is different from its prerequisite. Here is a minimal example:
ABOUT.html: README.md
help.html: help.md
%.html: %.md
pandoc --standalone --quiet -f gfm -H $(github-css) -o tmp.html $<
inliner -n < tmp.html > $#
rm -f tmp.html
With this Makefile, help.html gets made but ABOUT.html is never made. The reason, I presume, is because the base file name for %.html and %.md don't match up in the case of ABOUT.html because that target depends on README.md.
Is there a way to make this work without having to make a separate recipe for ABOUT.html?
One option is to create ABOUT.md symlink, so that your pattern rule works, by adding the following rule:
ABOUT.md : README.md
ln -s ${<F} ${#F}
You may like to avoid using the same temporary file in the recipe because that breaks in parallel builds. A better way is to use a unique temporary file for the target based on the target name:
%.html: %.md
pandoc --standalone --quiet -f gfm -H $(github-css) -o $#~ $<
inliner -n < $#~ > $#
rm -f $#~
first time here. I am relatively new to makefiles. Here is my current makefile:
# Closure compiler php script path
closure = ../../cli/scripts/Compilers/closure.php
# Destination directory
dest = ../../static/js/
# Build directory
build = build/
# Tell "make" to search build and destination dirs
vpath %.o $(build)
vpath %.js $(dest)
all: main.js
#echo "Done.";
main.js: \
date.o \
jquery.autocomplete.o \
jquery.bullseye.o \
jquery.clickopen.o \
jquery.fbmodal.o \
jquery.helpers.o \
jquery.pulljson.o \
jquery.thumbrotate.o \
jquery.timefmt.o \
jquery.tools.o \
layout.main.o
cat $^ > $(dest)$#
%.o: %.js
php $(closure) $*.js $(build)$#
clean:
rm -rf $(build)*.o
rm -rf $(dest)*.js
The problem is with the following line:
cat $^ > $(dest)$#.
It is supposed to cat all the prerequisite objects (minified javascript) into one final js library. According to makefile docs, $^ is an automatic variable which contains a list of prerequisites with directories they are in. From my experience, it behaves differently depending on wether prerequisite needs to be compiled or not.
If prerequisite is up-to-date, this code works perfectly and $^ contains a list like:
build/date.o build/jquery.autocomplete.o build/jquery.bullseye.o....
However if prerequisite needs a fresh compile, then $^ gets directory part stripped and looks like:
date.o jquery.autocomplete.o jquery.bullseye.o
Only the file which needs a fresh compile gets directory part stripped.
I have managed to work around this issue by replacing
cat $^ > $(dest)$#
with
cat $(addprefix $(build), $(^F) ) > $(dest)$#.
I don't like it because:
It's a hack
$(^F) is semi-deprecated
I want to understand why make behaves like this.
thanks
Look here:
# Tell "make" to search build and destination dirs
vpath %.o $(build)
If Make is looking for foo.o, it will look in the local directory first. If it finds no foo.o there, it will look in $(build) (i.e. build/, and you might reconsider your variable names).
And how would Make build foo.o, if it couldn't find it anywhere? With this rule:
%.o: %.js
php $(closure) $*.js $(build)$#
This rule violates an important guideline of makefiles, in that the target (foo.o) is not the name of the thing actually built (build/foo.o).
Now consider what happens when Make tries to execute this rule:
main.js: date.o ...
cat $^ > $(dest)$#
So if date.o is up to date, it's in build/. Make finds it there, and the automatic variable $^ expands to build/date.o ...
But if date.o must be rebuilt, then Make looks to the %.o rule, which promises to build date.o (not build/date.o), so Make takes that rule at its word and $^ expands to date.o ...
There are several ways to solve this problem. I'd do something like this:
OBJS := date.o jquery.autocomplete.o jquery.bullseye.o ...
OBJS := $(addprefix $(build),$(OBJS))
$(dest)main.js: $(OBJS)
cat $^ > $#
# you might have to tinker with this rule a little
$(build)%.o: %.js
php $(closure) $< $#
the makefile below processes files matching the patterncontent/%.md and outputs the targets in the html directory. Source files are named with the convention of putting a leading number in front of them, like content/01.index.md or content/O2.second-page.md and so on. I would like to remove the leading 0x. number sequence in the target file. For instance, content/01.index.html would generate html/index.html.
How can I do this?
Thanks
MD_FILES = $(shell find content/ -type f -name '*.md')
HTML_FILES = $(patsubst content/%.md, html/%.html, $(MD_FILES))
all: $(HTML_FILES) html/static
html/%.html : content/%.md
mkdir -p $(#D)
python generator/generate.py $< $#
.PHONY: html/static
html/static :
rsync -rupE generator/static html/
.PHONY: clean
clean:
rm -fr html
Replace:
html/%.html : content/%.md
mkdir -p $(#D)
python generator/generate.py $< $#
with:
html/%.html : content/%.md
mkdir -p $(#D)
file='$(#F)'; python generator/generate.py $< "$(#D)/${file#*.}"
Unfortunately, I can't think of a good way of doing that in make itself. I can think of one way but it isn't as simple as that escaping and it isn't safe for files with spaces (not that that matters much here since make already can't handle those).
IMHO, it is a bad idea to use find or wildcards to list files in makefiles. This is because developers have temporary or debugging files sometimes. It is best to list files explicitly. This way, it forces the developer to think about their intent.
If you agree to list files explicitly, then in this case it is best to list the target files, rather than source files, and here is your answer:
HTML_FILES := html/index.html html/second-page.html
.SECONDEXPANSION:
$(HTML_FILES): html/%.html : $$(wildcard content/*.$$*.md)
(put recipe here, using $# and $<)
I'm using GNU Make to build three different editions of a static html document.
I use Less as a CSS preprocessor.
My directory structure looks like this:
Makefile
160x600/style.less
300x250/style.less
728x90/style.less
This is my Makefile:
LESSC=lessc -x # use -x for debugging
.PHONY: all clean
all: 160x600 300x250 728x90
%.css: %.less
$(LESSC) $< > $#
160x600: 160x600/style.css
300x250: 300x250/style.css
728x90: 728x90/style.css
clean:
rm -f 160x600/*.css
rm -f 300x250/*.css
rm -f 728x90/*.css
This way, I can use make 160x600 to build style.css from style.less.
But I don't want to explicitly list a target rule for each directory. Instead, I tried adding this rule instead of the three directory specific ones:
%: %/style.css
But that does not work. I assume it's clear from that example what my goal is. Is there a way to accept any directory as a target, so that I just have to list the directory names in the all: rule?
use static pattern rule:
res_dirs = 160x600 300x250 728x90
$(res_dirs): %: %/style.css