I use TikZ to draw all kinds of vector graphics for my documents. It is rather slow and each drawing makes the compilation time of the main LaTeX document longer. Therefore I have all the figures as standalone files. They are compiled independently and the PDF file is included in the main document. This allows for parallel LaTeX runs on an as-needed basis.
The process looks as follows:
The actual TikZ code is in a LaTeX snippet file at Figures/fig.tex. A Python script (tikzpicture_wrap.py) will wrap the snippet into a standalone document including preamble. This files goes into build/page/fig.tex.
lualatex is run on the file build/page/fig.tex which produces build/page/fig.pdf.
As I use the same document class (scrartcl) as for the same document, the figure is set on A4 paper and therefore needs to be cropped before I can include it in the document. For that I use pdfcrop as a last step. The result is put into build/fig.pdf
My complete makefile looks like this:
# Copyright © 2015-2016 Martin Ueding <dev#martin-ueding.de>
.PRECIOUS: %.tex %.pdf build/page/%.pdf
document_tex := $(wildcard physics*.tex)
document_pdf := $(document_tex:%.tex=%.pdf)
figures_tex := $(wildcard Figures/*.tex)
figures_pdf := $(figures_tex:Figures/%.tex=build/%.pdf)
all: $(figures_pdf)
#all: $(document_pdf) # Disabled to only typeset figures right now.
test:
#echo "document: $(document_pdf)"
#echo "figures_tex: $(figures_tex)"
#echo "figures_pdf: $(figures_pdf)"
$(document_pdf): $(figures_pdf)
$(figures_pdf): build
build:
mkdir -p build/page
build/page/%.tex: Figures/%.tex
../build-system/tikzpicture_wrap.py $< $#
build/%.pdf: build/page/%.pdf
pdfcrop $< $#
touch $# # Added in an attempt to work around the problem, does not make any difference, though.
%.pdf: %.tex
cd $$(dirname $#) && lualatex --halt-on-error $$(basename $<)
clean:
$(RM) *-blx.bib
$(RM) *.aux
$(RM) *.log
$(RM) *.run.xml
$(RM) *.out
$(RM) *.svg
$(RM) *.pdf
$(RM) -r build
It does work, it typesets all figures and they end up at build/*.pdf. The problem is that the pdfcrop step is run again and again, even when there is nothing more to do. In the output you can see the following:
pdfcrop build/page/propagator.pdf build/propagator.pdf
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `build/propagator.pdf'.
touch build/propagator.pdf
This is repeated for every single figure I have in my Figures directory.
I thought that this might be a problem with chained rules and added the intermediate file to the .PRECIOUS target for make to keep it. Now the files used by pdfcrop are not deleted midway.
Next I thought that it might be a problem with the timestamps on the files. If the source is newer than the target, make will run it. Therefore I added the touch to make sure that the target was newer than the source. This is not a problem as can be seen here after the run I did this morning. I did not change anything in that Figure since yesterday.
The source file. stat Figures/propagator.tex:
File: 'Figures/propagator.tex'
Size: 102 Blocks: 8 IO Block: 4096 regular file
Device: fd03h/64771d Inode: 17432618 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ mu) Gid: ( 1000/ mu)
Context: unconfined_u:object_r:user_home_t:s0
Access: 2016-01-16 10:58:34.384515470 +0100
Modify: 2016-01-16 10:58:34.369515566 +0100
Change: 2016-01-16 10:58:34.373515540 +0100
Birth: -
The typeset PDF document. stat build/page/propagator.pdf:
File: 'build/page/propagator.pdf'
Size: 6265 Blocks: 16 IO Block: 4096 regular file
Device: fd03h/64771d Inode: 17432636 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ mu) Gid: ( 1000/ mu)
Context: unconfined_u:object_r:user_home_t:s0
Access: 2016-01-16 10:59:04.317323576 +0100
Modify: 2016-01-16 10:59:04.261323935 +0100
Change: 2016-01-16 10:59:04.261323935 +0100
Birth: -
The cropped final file. stat build/propagator.pdf:
File: 'build/propagator.pdf'
Size: 6612 Blocks: 16 IO Block: 4096 regular file
Device: fd03h/64771d Inode: 17301550 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ mu) Gid: ( 1000/ mu)
Context: unconfined_u:object_r:user_home_t:s0
Access: 2016-01-17 09:54:32.102511429 +0100
Modify: 2016-01-17 09:54:30.943517396 +0100
Change: 2016-01-17 09:54:30.943517396 +0100
Birth: -
Still it performs all the pdfcrop operations again and again. I do not understand why. For the meantime I added the following kludge to make the compilation process finish a bit faster when there is nothing to do:
build/%.pdf: build/page/%.pdf
if [ $< -nt $# ]; then \
pdfcrop $< $#; \
fi
What is the actual problem here and how can I solve it?
#Tsyvarev is on the right track. This is indeed the problem:
$(figures_pdf): build
However, it's not because build is not related to a particular file. build does exist: it's a directory which is created with the rule:
build:
mkdir -p build/page
This builds the subdirectory build/page, which means that build exists (as a directory) and so in subsequent builds when make checks to see if it exists, it will determine that yes, it does. Make doesn't treat directories any different than any other file, when listed as a target.
However the existence of a prerequisite is not the only test that make uses: it also checks to see if the prerequisite is newer than the target. This is where directories and files behave differently, and why you virtually never (except in some special cases) want to use a directory as a normal prerequisite of a target.
A directory has a "time last modified" just like a file, and a directory's TLM value is updated just like a file's: when the directory is modified. What constitutes "modifying" a directory? Well, adding a new file, deleting an existing file, or renaming a file all modify the directory and cause it's TLM to change.
Probably you can see your problem now: every time your makefile adds, removes, or renames a file in the build directory its modification time is updated. This means that the directory build is always newer than almost all the files in it, which means they always rebuild every time.
There are two ways to work around this. One is to just always create the directory as a side-effect; remove build from the prerequisite list and remove the rule above, and instead simply force the directory to be created by make when it parses the makefile:
__dummy := $(shell mkdir -p build/page)
The second way, if you have a sufficiently new version of GNU make, is to use order-only prerequisites:
$(figures_pdf): | build
Related
A code generator is executed from GNU make. The generator produces several files (depending on the input), and only touches the files, when their content change. Therefore a sentinel target needs to be used to record the generator execution time:
GEN_READY : $(gen_input_files)
gen.exe $(gen_input_files)
touch GEN_READY
$(gen_output_files): GEN_READY
It works well, except when a generated file is deleted, but the sentinel file is left in place. Since the sentinel is there, and it's up-to-date, the generator is not executed again.
What is the proper solution to force make to re-run the generator in this case?
Here is one way to group them using an archive:
# create archive of output files from input files passed through gen.exe
GEN_READY.tar: $(gen_input_files)
#echo Generate the files
gen.exe $^
#echo Put generated files in archive
tar -c -f $# $(gen_output_files)
#echo Remove intermediate files (recreated by next recipe)
rm $(gen_output_files)
# Extracting individual files for use as prerequisite or restoration
$(gen_output_files): GEN_READY.tar
#echo Extract one member
tar -x -f $< $#
Since tar (and zip for that matter) allows duplicate entries there could be opportunities updating or appending files in archive instead of rewriting if input-output relation allows.
Edit: Simplified solution.
I'm writing a Makefile to build a Latex document depending on plots whose data is generated from some other data by some python script.
It looks like this
% pdf plot needed by final document
build/tikz-standalone/%.pdf: build/tikz-standalone/%.tex xy_data
cd $$(dirname $#) && ../../latexrun $$(basename $<)
xy_data: $(PLOT_DATA) tools/plots/crunch.py | build
% crunch.py will create data for plots needed by build/tikz-standalone/%.tex
PYTHONPATH=. tools/plots/crunch.py
build:
mkdir -p build build/other_stuff ...
crunch.py generates several data files in build/data which are needed by build/tikz-standalone/%.tex. To create these files it uses other files stored in the variable PLOT_DATA. I could put a list of the intermediate data files in build/data into the Makefile at the position of xy_data. I don't like this as this would require me to update the list whenever a new file is added. What I want is that all data files are recreated whenever crunch.py or $(PLOT_DATA) has changed.
Is there a way to express this in Make?
If you do not want to provide and maintain the list of the generated files you can turn your (implicitly) phony xy_data target into an empty file used as a marker. Simply touch it at the end of the recipe:
BUILDDIRS := build build/other_stuff ...
build/tikz-standalone/%.pdf: build/tikz-standalone/%.tex xy_data
cd $(dir $#) && ../../latexrun $(notdir $<)
xy_data: $(PLOT_DATA) tools/plots/crunch.py | $(BUILDDIRS)
PYTHONPATH=. tools/plots/crunch.py
touch $#
$(BUILDDIRS):
mkdir -p $#
Note: I also improved a bit some other aspects:
Use of make functions dir and notdir instead of the shell equivalents.
Variable declaration for the build directories to avoid writing the same list several times, which is tedious and error prone.
Explicit list of all build directories as order-only prerequisites instead of just one, which could lead to unexpected results if this single one exists but not some others.
Generic rule for all build directories thanks to the $# automatic variable.
I have a very bizzare problem with GNU make. I have the following files:
a/x.html
b/Makefile
b/c/Makefile
The contents of a/x.html are irrelevant. The contents of b/Makefile are as follows:
SRC=../a
all: x.html
%.html: ${SRC}/%.html
rsync $< $#
The contents of b/c/Makefile are the same, except for the definition of SRC:
SRC=../../a
If I run make in b/c/ the result is as expected:
rsync ../../a/x.html x.html
and x.html gets copied from a/ to b/c/.
However, if I run make in b/ the output I get is several lines of:
make: stat: ../a/../a/.. (repeated many times) ../a/x.html: File name too long
It seems that make is applying the rule for %.html recursively, but why? Is there something obvious I am missing?
To build a target that matches the pattern %.html (i.e. any target name that ends in .html), make applies the rule if it can build the dependency (target built from the original target with ../a/ prepended).
You ask to build x.html. This matches the pattern %.html, so the rule applies: make sees if it can build ../a/x.html.
../a/x.html matches the pattern %.html, so the rule applies: make sees if it can build ../a/../a/x.html.
../../a/x.html matches the pattern %.html, so the rule applies, etc.
The stem character can match any part of a path, including directory separators.
You can see what make is trying by running make -r -d (-d to show debugging output, -r to turn off built-in rules which would cause a huge amount of noise).
When you're in b/c, this stops at step 2 because ../../a/x.html exists but ../../../../a/x.html doesn't.
One way to fix this is to list the files on which you want to act. You can build that list from the list of files that already exist in ../a:
$(notdir $(wildcard ${SRC}/*.html)): %.html: ${SRC}/%.html
rsync $< $#
This has the downside that if the HTML files in ../a are themselves built by a rule in b/Makefile, then running make in b won't built them in a pristine source directory. This shouldn't be a problem though: it would be unusual to have a makefile in b build things outside b.
Another approach which doesn't have this defect is to use an absolute path.
%.html: $(abspath ${SRC})/%.html
rsync $< $#
This seems slightly related to How to write Makefile where target and source files have the same extension?. In that question the extensions are the same, but the input and output files seem to be in the same directory and filenames are being conditionally renamed.
I have a large collection of .txt files in ../src/ that need to be processed, and dumped into ./ (which is a directory called target/) as txt files of the same name. I want to use make, so that only files in ../src/ that have been changed get updated in ./. I would like to get the prototype working before I put the real code in.
My Makefile in ./ is as follows:
DIR = ../src
INPUTS = $(wildcard $(DIR)/*.txt)
OUTPUTS = $(patsubst $(DIR)/%.txt,%.txt,$(INPUTS))
all: $(OUTPUTS)
.PHONY: $(INPUTS)
check:
#echo "DIR = $(DIR)"
#echo "INPUTS = $(INPUTS)"
#echo "OUTPUTS = $(OUTPUTS)"
%.txt: $(DIR)/%.txt
sed -e "s/test/cat/g" "$<" > $#
For now, the contents of ../src/ are test1.txt and test2.txt.
As the Makefile stands now, running make test2.txt generates the file as expected.
target/ $ make test2.txt
sed -e "s/test/cat/g" "../src/test2.txt" > test2.txt
Running make check shows the INPUTS and OUTPUTS correctly.
target/ $ make check
DIR = ../src
INPUTS = ../src/test1.txt ../src/test2.txt
OUTPUTS = test1.txt test2.txt
If I run make all, it generates every file, every time. This is expected with the .PHONY $(INPUTS) line in there.
If I remove the .PHONY $(INPUTS) target, Make gets all bound up in itself trying to find the target to make ../src/test1.txt and keeps prefixing $(DIR) in front of it until it makes too long of a filename and gives up.
make: stat: ../src/../src/../src/ [repeat for a few pages] ../src/../src/test1.txt: File name too long
make: stat: ../src/../src/../src/ [repeat for a few pages] ../src/../src/../src/test1.txt: File name too long
make: *** No rule to make target `../src/../src/../src/[repeat]../src/../src/test1.txt', needed by `../src/[repeat]../src/../src/test1.txt'. Stop.
It never does get to processing test2.txt.
As I was drafting this, I had the idea to remove the ../ from the DIR,
and relocate the Makefile so it was parent to both src/ and target/. That approach seems to work, but isn't ideal. Eventually there would be a chain of these Makefiles, each pulling from one directory to another.
Is there a way to keep the Makefile in 'target/' along with the generated destination files, and base those destination files off of something in a relative path?
Replace
%.txt: $(DIR)/%.txt
with:
${CURDIR}/%.txt: $(DIR)/%.txt
This way %.txt does not match any .txt file in any directory. In other words, you limit this rule's scope to files in ${CURDIR}/ only and this prevents that endless recursion.
See §10.5.4 How Patterns Match for more details.
It is also good practice to avoid relative paths:
DIR = $(abspath ../src)
It is clear that the target is newer than the source from these two ls
comands:
[metaperl#andLinux ~/edan/pkg/gist.el] ls -l ../../wares/gist.el/gist.elc #target
-rw-r--r-- 1 metaperl metaperl 10465 Jul 18 10:56 ../../wares/gist.el/gist.elc
[metaperl#andLinux ~/edan/pkg/gist.el] ls -l yank/gist.el/gist.el #source
-rw-r--r-- 1 metaperl metaperl 13025 Jul 18 10:57 yank/gist.el/gist.el
[metaperl#andLinux ~/edan/pkg/gist.el]
However when I run makepp -v I am told that this rule depends not only
on the listed target, but also on the cd and mv commands.
makepplog: Targets
/home/metaperl/edan/wares/gist.el/gist.elc'
depend on/usr/local/bin/emacs',
/home/metaperl/edan/pkg/gist.el/yank/gist.el/gist.el',
/bin/mv'
What aspect of make logic dictates that the actions to produce the
target are part of the dependency chain of deciding on whether to make
the target?
To my mind, only the listed sources should affect whether or not the
target is rebuilt.
The entire makepp -v output is quite long, and exists at:
http://gist.github.com/480468
My makepp file:
include main.makepp
#VER
PKG := gist.el
URL := http://github.com/defunkt/$(PKG).git
TARGET := $(WARES)gist.el/gist.elc
$(TARGET) : yank/gist.el/gist.el
cd $(dir $(input)) && $(BYTECOMPILE) gist.el
mv $(dir $(input)) $(WARES)
perl {{
print 'github username: ';
my $username = <STDIN>;
print 'github API token: ';
my $api_token = <STDIN>;
system "git config --global github.user $username";
system "git config --global github.token $api_token";
use File::Butler;
my $lines = Butler('init.el', 'read');
my $loc = sprintf '%s%s', $EDAN_PKG, "$PKG/";
$lines =~ s/__LOC__/$loc/g;
$lines =~ s/__PKG__/$PKG/g;
Butler( $EDAN_EL, prepend => \$lines );
}}
yank/gist.el/gist.el : yank
cd yank && git clone http://github.com/defunkt/gist.el.git
yank:
mkdir yank
$(phony clean):
$(RM) -rf $(dir $(TARGET)) yank
With a standard make, the contents of the commands to make a target are not taken into account when deciding whether to rebuild the target. Only the dependencies are taken into account; this can go beyond the source if you have dependencies declared elsewhere.
You don't show your makeppfile, so I can't be sure, but the Parsing command... messages from makepp -v make me suspect that makepp behaves differently from standard make on this count.
makepp will rebuild a target if any of the dependencies have changed or if the command has changes. In your case, I suspect that either some of the variables that you use in the rule to make $(TARGET) have changed or that makepp is seeing that the commands are constructed dynamically and is automatically rebuilding the target. Try using the -m target_newer option to makepp to force it to use the old GNU make method (that is, only re-build if the source is newer than the target).
Link