I'm trying to use the linux utility make in order to
Run a script to generate the data
Take all of the output files (data1.txt to data79.txt) and run a script to plot them each
Take all those images and make a movie from them
Yes, I realize that doing this in a shell/python script would be downright simple but I'm trying to learn how to use make in this context to do the work more intelligently.
My current make file looks something like this but is significantly flawed:
movie: data *.png
ffmpeg data_%d.png output.mp4
%.png: %.txt
python plot.py $< $#
data:
python make_data.py
You have several problems, so let's take them in order. (Caveats: I use GNUMake, so I can't promise my solution will work with other flavors, and I am not familiar with ffmpeg.)
The data rule looks correct, but you might want to warn Make that this rule does not actually produce a file called "data":
.PHONY: data
You can test this rule by itself: "make data".
The %.png rule looks correct. You can test it: "make data26.png" (after making sure that data26.txt exists).
The movie rule. This is a problem, because you're using "*.png" to indicate all png files, but at the time you run the rule there are no such files, so this evaluates to nothing. So we must look at all the data files that exist, and translate that list into a list of images to be made:
dfiles = $(wildcard *.txt)
images = $(dfiles:txt=png)
This will work if the data files already exist (and you can test it after "make data"), but when we first run make, the data files don't exist. There are several ways to address this; the simplest is to run Make a second time from within a rule, after the data files have been made:
$(MAKE) output.mp4
Putting it all together, we get something like this:
.PHONY: movie
movie: data
#$(MAKE) -s output.mp4 # I added the "#" and "-s" to make it quieter.
dfiles = $(wildcard *.txt)
images = $(dfiles:txt=png)
output.mp4: $(images)
ffmpeg data_%d.png $#
%.png: %.txt
python plot.py $< $#
.PHONY: data
data:
python make_data.py
(Note that some people like to put all the PHONY declarations together: ".PHONY: movie data". I prefer to do it as above.)
Related
Consider the following setup:
$ touch 1.src 2.src 3.src
$ cat Makefile
%.dst: %.src
#convert -o "$#" "$<"
We can compile our .src files into .dst files by running make 1.dst 2.dst 3.dst which calls the convert (just a placeholder) tool three times.
This setup is fine if there is little overhead in calling convert. However, in my case, it has a startup penalty of a few seconds for every single call. Luckily, the tool can convert multiple files in a single call while paying the startup penalty only once, i.e. convert -o '{}.dst' 1.src 2.src 3.src.
Is there a way in GNU make to specify that multiple src files should be batched into a single call to convert?
Edit: To be more precise, what feature I am looking for: Say that 1.dst is already newer than 1.src so it doesn't need to be recompiled. If I run make 1.dst 2.dst 3.dst, I would like GNU make to execute convert -o '{}.dst' 2.src 3.src.
A quick and dirty way would be creating a .PHONY rule that simply converts all src files to dst files but that way I would convert every src file each and every time. Further more, specifying dst files as prerequisites in other rules would also no longer be possible.
Thanks in advance!
If you have GNU make 4.3 or above, you can use grouped targets like this:
DST_FILES = 1.dst 2.dst 3.dst
SRC_FILES = $(_DST_FILES:.dst=.src)
all: $(DST_FILES)
$(DST_FILES) &: $(SRC_FILES)
convert -o '{}.dst' $?
#touch $(DST_FILES)
If your convert is only updating some of the targets then you need the explicit touch to update the rest.
Here's a way to do it with passing a goal on the command line that might work; change DST_FILES to:
DST_FILES := $(or $(filter %.dst,$(MAKECMDGOALS)),1.dst 2.dst 3.dst)
Is there a way in GNU make to specify that multiple src files should be batched into a single call to convert?
It is possible, but messy, to write make rules for build steps that produce multiple targets with a single run of the recipe, such that the recipe is executed just once if any of the targets needs to be updated. However, you clarify that
[if] 1.dst is already newer than 1.src [, and] I run make 1.dst 2.dst 3.dst, I would like GNU make to execute convert -o '{}.dst' 2.src 3.src.
. That's a slightly different problem. You can use the $? automatic variable in a recipe to get the prerequisites that are newer than the rule's target, but for that to serve the purpose, you need a rule with a single target.
Here's one slightly convoluted way to make it work:
DST_FILES = 1.dst 2.dst 3.dst
SRC_FILES = $(DST_FILES:.dst=.src)
$(DST_FILES): dst.a
ar x $< $#
dst.a: $(SRC_FILES)
convert -o '{}.dst' $?
x='$?'; ar cr $# $${x//src/dst}
The dst.a archive serves as the one target with all the .src files as prerequisites, so as to provide a basis for use of $?. Additionally, it provides a workaround for the problem that whenever that target is updated, it becomes newer than all the then-existing .dst files: .dst files that are out of date with respect to the archive but not with respect to the corresponding .src file are extracted from the archive instead of being rebuilt from scratch.
I'm trying to write a Makefile that automatically calls BibTeX on files that match a specific wildcard but don't exist when I first run Make. Specifically, I have the following:
.FORCE:
all: pdf
foo=something
lat: *.tex
pdflatex $(foo).tex
pdf: lat
open $(foo).pdf &
%.aux: .FORCE
bibtex $#
bib: lat $(foo)?.aux
pdflatex $(foo).tex
pdflatex $(foo).tex
open $(foo).pdf &
What I want to happen is that the following will occur when I run make bib:
pdflatex will be called on $(foo).tex, generating files $(foo)1.aux, $(foo)2.aux, etc.
bibtex will be called on $(foo)1.aux, then $(foo)2.aux, etc.
pdflatex will be called twice on $(foo).tex
open will be called on $(foo).pdf
However, this doesn't happen: in particular, it looks as if Make evaluates the prerequisites $(foo)?.aux up-front, at a point where the files $(foo)1.aux, $(foo)2.aux, etc. don't exist. As a result, BibTeX is never called on them. If I rerun make bib, however, things work, because the files now exist after being created on the previous run.
Question: Is forcing Make to re-evaluate prerequisites for a target the right way to fix this? If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat? If not, how can I achieve what I want please?
What I do in my Maiefile for LaTeX files is rename the targets.
That way, you can have different target names, depending on which phase has been used to create them. This is according to the spirit of make's pattern rules, which assume that files with different contents also have different extensions. So I have rules like this:
%.aux1 : %.tex
rm -f $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%.bbl : %.aux1
cp -pf $< $*.aux
bibtex $* || : > $#
%.aux2 : %.bbl
cp -pf $*.aux1 $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%-tex.pdf: %.aux2
cp -pf $< $*.aux
pdflatex -jobname $*-tex $*
You can't do this in a completely straightforward way, since make fundamentally assumes that one run through a target's commands will update the target. That is, there's no way in principle that you can tell make that ‘you need to run these commands twice’.
You can try to get round this with (admirably clever) tricks such as #reinerpost suggests, but a problem with that general approach is that sometimes/often a single run of BibTeX (or makeindex, or whatever) is sufficient.
After having tried various types of tricks in the past, what I generally do here is to make a command list which explicitly includes two BibTeX calls where necessary:
%.bbl: %.aux
bibtex $(#:.bbl=)
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then \
bibtex $(#:.bbl=); \
fi
That command list re-runs BibTeX if the log file includes the ‘Label(s) may have changed. Rerun to get cross-references right’ message.
To be honest, what I actually do is just the single line bibtex $(#:.bbl=). When I'm writing a document, I inevitably re-run make so many times that the list of references comes out correct very quickly. This means that this target doesn't work for the ‘recreate the final version from a clean directory’ case, but that's sufficiently rare that I tend not to obsess about it.
Whenever I catch myself re-solving this problem, I now recognise that I'm trying to push water up-hill because I'm bored writing this document, so I go and do something else.
I just wanted to share an alternative solution: Using submake processes:
If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat?
You can somewhat achieve that, by adding make lat to the recipe for bib. This will start a new make process targetting at bib. The sub-make doesn't know anything aboutits parents targets/prerequisites. (Such a concept is usually used, when some huge project is built from different smaller projekts each of which have different makefiles.)
This can be done in multiple layers (although it will be confusing):
bib: $(foo)?.aux lat check_for_bib
check_for_bib:
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then make bib; fi
pdf: lat check_for_bib
open $(foo).pdf &
Note that I had to change some orders of prerequisites. The pseud-code would be something like:
latex compilation
while log suggests update:
update aux
latex compilation
Each iteration of the while loop will take place in a separate make process.
I'm trying to write a make file that will create PDF from every inkscape SVG in a directory. From the make manual and various tutorials, it seems a pattern rule is the way to go, so I have
%.pdf : %.svg
inkscape -A $*.pdf $*.svg
I know the inkscape command works if I do it manually. When I invoke make though, I get
$ make
make: *** No targets. Stop.
Since the pdf files don't exist yet, I can't invoke make *.pdf and make *.svg won't match any targets. Also, I can't find a way to put an all target in that depends on the pattern.
One final problem; This is part of a large project, and I would like to invoke make in this directory recursively, but if make alone doesn't work, what target should I invoke recursively and how do I do this?
Your pattern is correct, but a pattern tells make how to build a target if you ask for one. You still have to ask for it, and since you haven't that's why you get the "no targets" message.
If you want to find all the SVG files and convert them, you can use the wildcard function:
SVGFILES := $(wildcard *.svg)
all: $(SVGFILES:%.svg=%.pdf)
%.pdf : %.svg
inkscape -A $*.pdf $*.svg
I don't quite understand your second question. Once you have the above you can just use normal $(MAKE) (always use this, never make) in a parent makefile to build these files.
I have a Makefile with the following type of rule:
%.html:
./generate-images.py > $#
make $(patsubst %.png,%.gif,$(wildcard *.png))
The generate-images script writes not only the HTML file (to stdout) but several .png files to the current directory. The goal here is to convert them to .gif. (not really, but this is an example)
This works if I invoke it directly. The problem is: If I invoke it from another rule where foo.html is a dependency, the wildcard statement fails to find any files. In other words, it just called make with no arguments, which is not what I want here.
What's the deal with the wildcard? Or, is there a better way to do this?
While your problem may be something different, I clearly see one.
The whole text of all commands within the rule is simultaneously processed so that make's functions and variables get expanded. Assume you have no .png files in the directory, and you invoke make so it should regenerate them: a.png and b.png. Then, after you invoke make, the text of the rule would effectively look like this:
file.html:
./generate-images.py > file.html
make
because at the moment of reading the makefile there were no .png files! After the first line is executed, the files will appear, but the next line was already generated to be just "make".
And only when you invoke your makefile for the second time, will it expand to
file.html:
./generate-images.py > file.html
make a.gif b.gif
This is not what you want. So I suggest doing it in The Right Way.
# If you have batch conversion program, this may be helpful
images.stamp: *.png
convert_all_images $?
touch images.stamp
# OR, if you want convert one-by-one with means of make
images.stamp: $(wildcard *.png)
touch images.stamp
%.gif: %.png
convert_one --from=$^ --to=$#
# HTML would look like
%.html:
./generate-images.py > $#
make images.stamp
So when you invoke make all, it generates htmls and converts newly generated images. Note that it will only convert the images that are updated, which is what you want.
Thanks to Beta for pointing out the mess with gif/png extensions.
That sounds like it's evaluating all of the $() expressions as it's processing the Makefile, rather than as it executes each rule. You could add a rule to your makefile like so:
images: $(patsubst %.png,%.gif,$(wildcard *.png))
.PHONY: images
and then change your example snippet to
%.html:
./generate-images.py > $#
make images
so that Make evaluates the glob at the right time. This is something about which checking the manual might be worthwhile.
I'd like to use a makefile to convert a set of svgs to pngs. The following snippet shows what I've done so far.
%.png: origs/%.svg
convert -resize "32x" $< $#
make foo.png works now. but I'm stuck with the all target. I'd like to convert everything (all svgs that is) with a single command.
In all examples I've found the all target does something like this:
all: ${OBJECTS}
But I'm sure there's a simpler way for my case, please help me find it ! :D
Depending on the version of make you're using, you may be able to define a set of targets based on all the svgs that are present. For example, in GNU make:
SVG = $(wildcard *.svg)
PNG = $(SVG:.svg=.png)
all: $(PNG)
I don't remember if make can do that.
You could add a shell foreach statement to your all:
( cd origs ; for file in *.svg ; do ; convert ${file} ; done )
You need the parens to make the foreach share the same environment (shell).
I have too many semicolons; I can never remember where they're needed and not needed when turning a multiline shell command into a one-liner.
What you have up there is a pattern rule. It tells make how to make a certian kind of target (those ending with ".png"), rather than any target in particular. So what you have is something that allows you type in an arbitrary "make foo.png" and it will know how to do that.
The way I would generalise this to create a rule for making "all .png's you can make from this directory" would be using a variable. Put a line something like this at the top of your makefile:
SVGs = *.svg
PNGs = $(subst .svg,.png,$(SVGs))
The first line will create a variable SVGs that contains all the files in your directory ending in ".svg". The second creates a variable PNGs containing the same list, but with ".png" on the end instead.
Then you should create a rule to build all SVG's like so:
allsvgs : $(PNGs)
Note that I did not call it "all". The "all" target is an unnoficial standard target that should always mean "build every target my makefile supports", In your case I suppose you could make a case for putting "allsvgs" on all's list of targets, but as your makefile grows you will need to add stuff to it.