I am interested in downloading yearly data from some location on the internet.
I have a python script GetYearData.py which does this, taking as a command-line argument the year and the output filename.
I'd like to run this script on several years of data at once; I want to use Make so that if I change the start or end year I don't have to re-download all the data.
I can do this for a single year with a very simple Makefile:
data/YearData_2000.txt : GetYearData.py
python $< --year 2000 --outfile $#
However, I'd like to do this a loop (or similar construction) so that, for each year in a sequence in bash I'd say YEARS=($(seq $(SYEAR) 1 $(EYEAR))) -- I pass --year $(Y) to my script and generate an appropriate target for --outfile?
Try something like:
SYEAR = 2000
EYEAR = 2017
YEARS := $(shell seq $(SYEAR) 1 $(EYEAR))
all: $(patsubst %,data/YearData_%.txt,$(YEARS))
data/YearData_%.txt : GetYearData.py
python $< --year $* --outfile $#
You can override SYEAR and EYEAR on the make command line if you want.
Related
Consider the following setup:
$ touch 1.src 2.src 3.src
$ cat Makefile
%.dst: %.src
#convert -o "$#" "$<"
We can compile our .src files into .dst files by running make 1.dst 2.dst 3.dst which calls the convert (just a placeholder) tool three times.
This setup is fine if there is little overhead in calling convert. However, in my case, it has a startup penalty of a few seconds for every single call. Luckily, the tool can convert multiple files in a single call while paying the startup penalty only once, i.e. convert -o '{}.dst' 1.src 2.src 3.src.
Is there a way in GNU make to specify that multiple src files should be batched into a single call to convert?
Edit: To be more precise, what feature I am looking for: Say that 1.dst is already newer than 1.src so it doesn't need to be recompiled. If I run make 1.dst 2.dst 3.dst, I would like GNU make to execute convert -o '{}.dst' 2.src 3.src.
A quick and dirty way would be creating a .PHONY rule that simply converts all src files to dst files but that way I would convert every src file each and every time. Further more, specifying dst files as prerequisites in other rules would also no longer be possible.
Thanks in advance!
If you have GNU make 4.3 or above, you can use grouped targets like this:
DST_FILES = 1.dst 2.dst 3.dst
SRC_FILES = $(_DST_FILES:.dst=.src)
all: $(DST_FILES)
$(DST_FILES) &: $(SRC_FILES)
convert -o '{}.dst' $?
#touch $(DST_FILES)
If your convert is only updating some of the targets then you need the explicit touch to update the rest.
Here's a way to do it with passing a goal on the command line that might work; change DST_FILES to:
DST_FILES := $(or $(filter %.dst,$(MAKECMDGOALS)),1.dst 2.dst 3.dst)
Is there a way in GNU make to specify that multiple src files should be batched into a single call to convert?
It is possible, but messy, to write make rules for build steps that produce multiple targets with a single run of the recipe, such that the recipe is executed just once if any of the targets needs to be updated. However, you clarify that
[if] 1.dst is already newer than 1.src [, and] I run make 1.dst 2.dst 3.dst, I would like GNU make to execute convert -o '{}.dst' 2.src 3.src.
. That's a slightly different problem. You can use the $? automatic variable in a recipe to get the prerequisites that are newer than the rule's target, but for that to serve the purpose, you need a rule with a single target.
Here's one slightly convoluted way to make it work:
DST_FILES = 1.dst 2.dst 3.dst
SRC_FILES = $(DST_FILES:.dst=.src)
$(DST_FILES): dst.a
ar x $< $#
dst.a: $(SRC_FILES)
convert -o '{}.dst' $?
x='$?'; ar cr $# $${x//src/dst}
The dst.a archive serves as the one target with all the .src files as prerequisites, so as to provide a basis for use of $?. Additionally, it provides a workaround for the problem that whenever that target is updated, it becomes newer than all the then-existing .dst files: .dst files that are out of date with respect to the archive but not with respect to the corresponding .src file are extracted from the archive instead of being rebuilt from scratch.
I am working on a writing project and would like to use make for running pandoc on files. So far I've tried to pass arguments to make like I do with a bash script.
For example:
$ make chapter 2
In the make file chapter is the target and 2 would be the argument.
I don't know if makefiles have the facility to take cli arguments. I haven't been able to find what I'm looking for in the documentation.
So far I have tried to run make with this recipe.
chapter:
#pandoc -s -o "$1.epub" "$1.md"
I get this error back
pandoc: .md: openBinaryFile: does not exist (No such file or directory)
make: *** [Makefile:2: chapter] Error 1
This is for turning sections of a book I'm working on into epubs. I'm open to other ways to do this with make seeing as tokens don't to work.
In the make file chapter is the target and 2 would be the argument
$ make chapter num=2
The assignment to variables on the make command-line overrides any definition inside the makefile
(yep, such variables effectively become read-only).
This suggests a makefile something like:
num = $(error You must set $$num to the chapter number you want (make chapter num=4))
.PHONY: chapter
chapter:
pandoc -s -o "${num}.epub" "${num}.md"
What's going on here?
Well, if you forget to set num,
when make expands the recipe for chapter the
$(error) will cause make to stop.
$ make
Makefile:5: *** You must set $num to the chapter number you want (make chapter num=4). Stop.
And your original example?
$ make chapter num=2
pandoc -s -o "2.epub" "2.md"
Tips
I rarely recommend using the # prefix — Users can use make's -s if they don't want to see the shell commands
Don't lie to make — In particular, your rule does not produce a file called chapter, so please tell make that by marking the target .PHONY
The natural way to say this in Make is to enumerate all the chapters as targets, typically as dependencies for make all.
So basically
src := $(wildcard *.md)
epubs := $(patsubst %.md,%.epub,$(src))
.PHONY: all
all: $(epubs)
%.epub: %.md
pandoc -s -o $# $<
You can say make ch4.epub if you have a chapter whose source is ch4.md. You can't really pass in an argument which isn't a file name or a target name, and these cannot contain spaces.
I suppose you could add a phony like
.PHONY: 2
2: ch2.epub
to be able to say make 2 and have it mean make ch2.epub. If file names are systematically named like this, you could generalize to
short := $(patsubst ch%.md,%,$(src))
.PHONY: $(short)
$(short): %: ch%.epub
Don't use # in front, it just makes things harder. You can use make -s if you don't want to see the output and not wreck your Makefile.
Goal
I am trying to use a Make file to automate the generation of a panel figure. The data and figures are generated from Rscripts and subsequently combined using latex to generate a PDF.
Problem description
Let's say, I would like to generate a panel figure consisting of 4 subfigures, arranged in a 2x2 grid. By using the following make file, I am able to generate the data as well as the subfigures:
all : data/plot1.txt plots/plot1.pdf \
data/plot2.txt plots/plot2.pdf \
data/plot3.txt plots/plot3.pdf \
data/plot4.txt plots/plot4.pdf \
plots/figure_5.pdf
#Panel Figure#
#Plot 1
data/plot1.txt : scripts/plot1_sim.R
Rscript scripts/plot1_sim.R $#
plots/plot1.pdf : scripts/plot1_sim.R data/plot1.txt
Rscript scripts/plot1.R $#
#Plot 2
data/plot2.txt : scripts/plot2_sim.R
Rscript scripts/plot2_sim.R $#
plots/plot2.pdf : scripts/plot2_sim.R data/plot2.txt
Rscript scripts/plot2.R $#
#Plot 3
data/plot3.txt : scripts/plot3_sim.R
Rscript scripts/plot3_sim.R $#
plots/plot3.pdf : scripts/plot3_sim.R data/plot3.txt
Rscript scripts/plot3.R $#
#Plot 4
data/plot4.txt : scripts/plot4_sim.R
Rscript scripts/plot4_sim.R $#
plots/plot4.pdf : scripts/plot4_sim.R data/plot4.txt
Rscript scripts/plot4.R $#
Using a latex file, e.g. figure.tex I am able to compile a panel figure as desired.
However, now I would like to add the construction of the final PDF by latex to the make file. This file is, of course, dependent on plots/plot1.pdf up to and including plots/plot4.pdf.
I added it to the code like this:
plots/figure.pdf : scripts/figure.tex plots/plot1.pdf plots/plot2.pdf plots/plot3.pdf plots/plot4.pdf
pdflatex scripts/figure.tex
With four subfigures it is doable (but maybe ugly) to do it like this, but if you generate a panel figure with e.g. 16 subfigures, the code becomes unreadable.
Question
Is there a better way to include a larger list of dependencies, in this case for generating panel figures using make and latex?
As far as I understand you could probably factorize all this with something like:
SIMRS := $(wildcard scripts/plot*_sim.R)
PLOTS := $(patsubst scripts/plot%_sim.R,%,$(SIMRS))
PDFPLOTS := $(patsubst %,plots/plot%.pdf,$(PLOTS))
TXTPLOTS := $(patsubst %,data/plot%.txt,$(PLOTS))
.PHONY: all
all: $(TXTPLOTS) $(PDFPLOTS) plots/figure_5.pdf
data/plot%.txt : scripts/plot%_sim.R
Rscript $< $#
plots/plot%.pdf : scripts/plot%_sim.R data/plot%.txt
Rscript scripts/plot$*.R $#
plots/figure.pdf: scripts/figure.tex $(PDFPLOTS)
pdflatex $<
Note that the plots/plot%.pdf: ... pattern rule looks strange: its recipe uses scripts/plotX.R but its prerequisites are scripts/plotX_sim.R and data/plotX.txt (no scripts/plotX.R). Is it normal?
I am trying to use makefile to manage my building process in a small project, where the target number and target names are not known beforehand but depends on the input. Specifically, I want to generate a bunch of data files (say .csv files) according to a cities_list.txt file with a list of city names inside. For example, if the contents of the txt file are:
newyork
washington
toronto
then a script called write_data.py would generate three files called newyork.csv, washington.csv and toronto.csv. When the content of the cities_list.txt file changes, I want make to deal with this change cleverly, i.e. only update the new-added cities files.
I was trying to define variable names in target names to make this happen but didn't succeed. I'm now trying to create a bunch of intermediate .name files as below:
all: *.csv
%.name: cities_list.txt
/bin/bash gen_city_files.sh $<
%.csv: %.name write_data.py
python3 write_data.py $<
clean:
rm *.name *.csv
This seems to be very close to success, but it only gives me one .csv file. The reason is obvious, because make can't determine what files should be generated for the all target. How can I let make know that this *.csv should contain all the files where there exists a corresponding *.name file? Or is there any better way to achieve what I wanted to do here?
All right, this should do it. We'd like a variable assignment at the head of the file:
CITY_FILES := newyork.csv washington.csv toronto.csv
There are two ways to do this. This way:
-include cities.mak
# this rule can come later in the makefile, near the bottom
cities.mak: cities_list.txt
#sed 's/^/CITIES := /' $< > $#
and this way:
CITIES := $(shell cat cities_list.txt)
After we've done one of those two, we can construct the list of needed files:
CITY_FILES := $(addsuffix .csv, $(CITIES))
and build them:
# It is convenient to have this be the first rule in the makefile.
all: $(CITY_FILES)
%.csv: write_data.py
python3 $< $*.name
I tried looking for answers to this question, so I apologize in advance if this is a duplicate of a question I didn't find. Also sorry that I cannot directly provide the code that I am working with (it would require a lot of environmental dependencies, anyway).
I have a sequence of actions, which all depend on the success of the previous actions, and also don't need to be repeated unless they are out of date. A make solution seemed like the proper one. I've come up with a solution that does almost all of it. Here is the sequence of steps I am trying to replicate, with the output of each step listed below its input:
ZIP file
extract to package/
package/directory/*.comp
execute uncomp.py to create a .uncomp file from a .comp file
Everything works fine up to this point
package/directory/*.uncomp
For *.uncomp files, execute script1 to produce a .html file
For *_ext.uncomp files, execute script2 to produce numbered *_ext.##.png file(s)
Multiple numbered files (_ext.0.png, _ext.1.png, _ext.2.png) are possible, and may not be present at the time make is run. However, make should know that they are the output of the previous step, and only run this recipe if these files (a) don't exist or (b) any are older than the *_ext.uncomp file.
I have put together a Makefile which does almost what I'm looking for, except that it delegates all of the last portion (numbered files) to a shell script which I could program to look at file times, but that defeats the purpose of using make in the first place, in my opinion.
Environment
Debian 8.8 (x86)
GNU Make 4.0
Built for x86_64-pc-linux-gnu
My Question
What rules and recipes can I use to inform GNU make of the relationship between the *_ext.uncomp files and the _ext.##.png files so that those recipes only get executed as necessary (and say 'Target is up-to-date' if all .png files are at least as new as the _ext.uncomp file), that won't also apply to the *.uncomp files, and that will still work of there are no .png files in the output?
I will also need to indicate the relationship between non-_ext files and their corresponding HTML counterparts. So that script1 only gets executed when the HTML file is out of date or doesn't exist. This recipe/rule should not pay attention to _ext.uncomp files.
Any other advice on my Makefile would also be appreciated, because I am not overly familiar with it.
Generalized contents of my current Makefile
.PHONY : all
all : package package/directory/*.uncomp
./process $^
%.comp.uncomp : %.comp package
python uncomp.py $<
package : *.zip
rm -rf package/
unzip *.zip -d package/
Contents of the process script
This script should no longer exist if all the goals of the question are met (make will handle everything). It works great, but it always processes .uncomp files no matter what, even if the output from them already exists and is newer than the source.
#!/bin/bash
if [ $# -lt 2 ]; then
echo "$0 expects at least 2 arguments"
exit 1
fi
# Discard the first agrument, it's always 'package'
shift
# Iterate over each of the remaining arguments
while [ $# -gt 0 ]; do
if [[ $1 == *_ext.uncomp ]] ; then
python script2 $1
elif [[ $1 == *.uncomp ]] ; then
python script1 $1
else
echo "Warning: Unknown file type: $1"
fi
shift
done
I learned a lot about GNU make trying to get this to work. I discovered that the solution to my problem was in not overthinking it.
The most important realization was that I didn't need make to track all of the numbered output files, but just the first one (if the first one is out of date or missing, they all will be, and they all get re-extracted by the script, so a 1:1 relationship was all I needed to indicate there).
I found out that GNU make 3.82 and later uses "shortest stem first" order instead of definition order when matching pattern rules. To make my file compatible with both versions, I made sure to define the most specific stems first.
After that it was a matter of setting up some implicit rules, and just telling make what to expect to be able to find—the concept is a little backwards to my way of thinking which is why I had some trouble at first (look for this file that doesn't exist yet; now, here's a way to make it from a file that does exist). The end result, fully functional:
PACKAGE := package
COMP := .comp
UNCOMP := .comp.uncomp
PNG0 := .comp.0.png
TXT := .comp.txt
SUFFIX := _ext
COMPFILES = $(wildcard $(PACKAGE)/subdir/*$(COMP))
UNCOMPFILES = $(COMPFILES:$(COMP)=$(UNCOMP))
SUFFIXFILES = $(filter %$(SUFFIX)$(UNCOMP),$(UNCOMPFILES))
PNGFILES = $(SUFFIXFILES:$(UNCOMP)=$(PNG0))
NOSUFFIXFILES = $(filter-out %$(SUFFIX)$(UNCOMP),$(UNCOMPFILES))
TXTFILES = $(NOSUFFIXFILES:$(UNCOMP)=$(TXT))
.PHONY : all
all : pngs txts htaccess
.PHONY : txts
txts : $(TXTFILES)
.PHONY : pngs
pngs : $(PNGFILES)
.PHONY : uncomp
uncomp : $(UNCOMPFILES)
make pngs
make txts
.PHONY : htaccess
htaccess : $(PACKAGE)/.htaccess
%$(SUFFIX)$(PNG0) : %$(SUFFIX)$(UNCOMP)
## Ignore failures when extracting PNG files
-python script1.py $<
%$(TXT) : %$(UNCOMP)
## Ignore failures when dumping TXT files
-python script2.py $< > $#
%$(UNCOMP) : %$(COMP)
## Ignore decompression failure
-python uncomp.py $<
$(PACKAGE)/.htaccess : .htaccess | $(PACKAGE)
cp .htaccess $(PACKAGE)/
$(PACKAGE) : *.zip
rm -rf $(PACKAGE)/
unzip *.zip -d $(PACKAGE)/
make uncomp
.PHONY : clean
clean :
rm -rf $(PACKAGE)/