Goal
I am trying to use a Make file to automate the generation of a panel figure. The data and figures are generated from Rscripts and subsequently combined using latex to generate a PDF.
Problem description
Let's say, I would like to generate a panel figure consisting of 4 subfigures, arranged in a 2x2 grid. By using the following make file, I am able to generate the data as well as the subfigures:
all : data/plot1.txt plots/plot1.pdf \
data/plot2.txt plots/plot2.pdf \
data/plot3.txt plots/plot3.pdf \
data/plot4.txt plots/plot4.pdf \
plots/figure_5.pdf
#Panel Figure#
#Plot 1
data/plot1.txt : scripts/plot1_sim.R
Rscript scripts/plot1_sim.R $#
plots/plot1.pdf : scripts/plot1_sim.R data/plot1.txt
Rscript scripts/plot1.R $#
#Plot 2
data/plot2.txt : scripts/plot2_sim.R
Rscript scripts/plot2_sim.R $#
plots/plot2.pdf : scripts/plot2_sim.R data/plot2.txt
Rscript scripts/plot2.R $#
#Plot 3
data/plot3.txt : scripts/plot3_sim.R
Rscript scripts/plot3_sim.R $#
plots/plot3.pdf : scripts/plot3_sim.R data/plot3.txt
Rscript scripts/plot3.R $#
#Plot 4
data/plot4.txt : scripts/plot4_sim.R
Rscript scripts/plot4_sim.R $#
plots/plot4.pdf : scripts/plot4_sim.R data/plot4.txt
Rscript scripts/plot4.R $#
Using a latex file, e.g. figure.tex I am able to compile a panel figure as desired.
However, now I would like to add the construction of the final PDF by latex to the make file. This file is, of course, dependent on plots/plot1.pdf up to and including plots/plot4.pdf.
I added it to the code like this:
plots/figure.pdf : scripts/figure.tex plots/plot1.pdf plots/plot2.pdf plots/plot3.pdf plots/plot4.pdf
pdflatex scripts/figure.tex
With four subfigures it is doable (but maybe ugly) to do it like this, but if you generate a panel figure with e.g. 16 subfigures, the code becomes unreadable.
Question
Is there a better way to include a larger list of dependencies, in this case for generating panel figures using make and latex?
As far as I understand you could probably factorize all this with something like:
SIMRS := $(wildcard scripts/plot*_sim.R)
PLOTS := $(patsubst scripts/plot%_sim.R,%,$(SIMRS))
PDFPLOTS := $(patsubst %,plots/plot%.pdf,$(PLOTS))
TXTPLOTS := $(patsubst %,data/plot%.txt,$(PLOTS))
.PHONY: all
all: $(TXTPLOTS) $(PDFPLOTS) plots/figure_5.pdf
data/plot%.txt : scripts/plot%_sim.R
Rscript $< $#
plots/plot%.pdf : scripts/plot%_sim.R data/plot%.txt
Rscript scripts/plot$*.R $#
plots/figure.pdf: scripts/figure.tex $(PDFPLOTS)
pdflatex $<
Note that the plots/plot%.pdf: ... pattern rule looks strange: its recipe uses scripts/plotX.R but its prerequisites are scripts/plotX_sim.R and data/plotX.txt (no scripts/plotX.R). Is it normal?
Related
I am interested in downloading yearly data from some location on the internet.
I have a python script GetYearData.py which does this, taking as a command-line argument the year and the output filename.
I'd like to run this script on several years of data at once; I want to use Make so that if I change the start or end year I don't have to re-download all the data.
I can do this for a single year with a very simple Makefile:
data/YearData_2000.txt : GetYearData.py
python $< --year 2000 --outfile $#
However, I'd like to do this a loop (or similar construction) so that, for each year in a sequence in bash I'd say YEARS=($(seq $(SYEAR) 1 $(EYEAR))) -- I pass --year $(Y) to my script and generate an appropriate target for --outfile?
Try something like:
SYEAR = 2000
EYEAR = 2017
YEARS := $(shell seq $(SYEAR) 1 $(EYEAR))
all: $(patsubst %,data/YearData_%.txt,$(YEARS))
data/YearData_%.txt : GetYearData.py
python $< --year $* --outfile $#
You can override SYEAR and EYEAR on the make command line if you want.
(Similar question here). For every .x file in the directory I want to run a command to generate an HTML file. The command already works fine, it's the Makefile that I'm having trouble with. Here's what I came up with:
all: $(OBJECTS)
OBJECTS := $(patsubst %.x,%.html,$(wildcard *.x))
%.html: %.x
generate $< > $#
The OBJECTS variable is meant to contain all html files I need to generate. Invoking make states nothing to be done for 'all', and there are no HTML files already in the directory.
Turns out make is sensitive to the order of your variable definitions. The Makefile works when the first two lines are switched!
I have a makefile with a bunch of .R scripts that create .csv files as output. These files are then used in a python simulation.
all: $(FILES)
python simulation.py
$(FILES): %.csv: %.R
Rscript $<
This is straightforward. My wrinkle is that one (and only one) of the .R scripts has its own dependency, say sourcedata. It seems that placing this dependency in the pattern would be annoying
all: $(FILES)
python simulation.py
$(FILES): %.csv: %.R sourcedata
Rscript $<
sourcedata:
wget "sourcedata.zip"
But doing it like all: sourcedata $(FILES) and relying on order-of-operations would be less effective. I guess that I could also give that R file its own phony rule
problem_script.R: sourcedata
#echo R script read
but I haven't been able to test if that will work. Is there an established way to do this?
Unless I'm misunderstanding, you need to specify that the particular CSV file needs to be rebuilt whenever either the R file or the sourcedata file changes, right?
If so, just add this:
problem_script.csv: sourcedata
(no recipe needed). This declares an extra prerequisite for that particular CSV file, so whenever it's out of date with respect to sourcedata it will be rebuilt.
I would like to process files generated from a list into a summary, using parallel processing of each intermediary file. make might be well suited for this.
Let's take an example : given a list of urls, download files, process them in parallel, and generate a report from the processed files.
example (won't work) :
all : report_file
report_file : $(wildcard data/*.processed)
...
data/%.processed : data/%.input
... # this should be processed in parallel
data/%.input : filelist
download all lines of filelist to N files.
filelist :
generate_list url_file > $#
I'd like the processing (and maybe downloading) of each file to be done in parallel and I don't know how many lines will be generated in filelist.
this won't work because the processed files do not exist when run, so rule to build report will be given an empty input.
also, it can be useful to avoid downloading files newer than 1 day and thus not process them, so makefile like dependency has a use here.
I could generate a special makefile from the list, but is there a way to do it with a single makefile ?
Since report_file's dependencies can't be evaluated until after all the target-dependency information is parsed, you'll need to refresh this information again. The only way to do this that I know of is to invoke a submake.
all: filelist
$(MAKE) $(shell cat $<) #make data/a.processed data/b.processed etc...
$(MAKE) report_file
report_file : $(wildcard data/*.processed)
...
data/%.processed : data/%.input
... # this should be processed in parallel
data/%.input :
download all lines of filelist to N files.
filelist :
generate_list url_file > $# #url list
sed -i 's;\(.*\);\1.processed;g' $# #append .processed to all urls
I'm trying to produce some graphs using GNUplot with a makefile. I would like for every *.plt file in the directory to be run through GNUplot, however I can't see to get it to work.
Here's my makefile so far:
all: %.tex
%.tex: %.plt
<tab> gnuplot < $<
The recipe is working fine if I specify a .plt file individually but I want it to pick up my new plots as I produce them.
EDIT:
I think I've got it working now:
# plots all files in the folder with .plt extensions
SOURCES = $(wildcard *.plt)
TARGETS = $(SOURCES:.plt=.tex)
all: $(TARGETS)
%.tex: %.plt
gnuplot < $<
Can someone confirm whether my reasoning (as follows) is correct?
Previously I hadn't specified any files for all (I'm a little confused by %). Now assigning the variable SOURCES by picking up any .plt files using the wildcard (why doesn't it work when using .plt instead of *.plt?). Having assigned SOURCE, the TARGETS variable is then set, now all: has files specified to build. and the matching rule is now run.
all : %.tex won't work because there is no percent in the target name, in other words, it is not a patter rule.
Use wildcard function to get the list of all .plt files and add an all dependence on these files with the extension replaced by .tex:
PLT_FILES := $(wildcard *.plt)
TARGETS := $(PLT_FILES:%.plt=%.tex)
all: $(TARGET)
%.tex: %.plt
gnuplot < $<