This is my contrived Makefile to illustrate my question:
allinfiles = file1.in file2.in file3.in
alloutfiles = file1.out file2.out file3.out
all: $(alloutfiles)
clean:
-rm *.pyc *.in *.out > /dev/null 2>&1
# my.py generates file1.in file2.in file3.in
$(allinfiles): my.py; python my.py
%.out: %.in
cp $< $#
My contrived Python script:
#!/usr/bin/env python
for i in 1,2,3:
filename = 'file{}.in'.format( i )
f = file( filename, 'wt' )
f.write( filename )
f.close()
Result:
$ make clean; make -j 8
python my.py
python my.py
python my.py
Question 1: how do I write my rules so that my.py is executed only once since it generates all the .in files?
NB: Unless I misunderstood, I'm trying to use this answer and not getting the results I expected.
Question 2: is there a way to log all the build outputs by their target? That is, if instead of $(shell cp $< $#), the recipe generated much output, how do I separate the output by target or input file?
Related
There are a couple of kind of similar issues but I could not fit any of the proposed concepts to my case.
Just to give a little bit of context: I have a set of Julia files which create plots as PDFs which are part of a make procedure to create scientific papers, something like:
plots = $(shell find $(PLOT_PATH)/*.jl | sed 's/\.jl/\.pdf/g')
$(PLOT_PATH)/%.pdf: $(PLOT_PATH)/%.jl $(JULIA_SYSIMAGE)
$(JL) --project $< -o $(PLOT_PATH)
$(DOCUMENT_FILENAME).pdf: FORCE $(plots) $(figures)
latexmk $(DOCUMENT_FILENAME).tex
In the current setup, each XYZ.jl file is creating a XYZ.pdf file and it works absolutely fine.
Now I am dealing with cases where it would be much easier to create multiple plots from single Julia files, so a script like this:
#!/usr/bin/env julia
using PGFPlotsX
...
...
pgfsave("whatever.pdf")
pgfsave("another.pdf")
pgfsave("yetanother.pdf")
so that one could do a grep pgfsave SCRIPT | awk... to figure out the targets. However, I could not figure out how to generate dynamic targets (plots) based on the contents of the dependency file (Julia script).
An MWE for my problem is the following: I have a couple of files (dependencies) which are generating a bunch of targets, which are defined inside those files (and can be access via awk/grep/sed/whatever). For now, let's say that these are simply *.txt files and each line is a target.
file: a.txt
foo
bar
baz
file: b.txt
naarf
fjoord
A very basic (non-working) manual Makefile to demonstrate the goal would be something like this (it does not work as it cannot figure out how to make foo etc. but it shows the pattern for *.txt which needs to be repeated):
file: Makefile
all_products := $(shell find *.txt | xargs cat)
final_product: $(all_products)
echo $< > $#
(foo bar baz): a.txt
touch $(shell cat $<)
(narf fjoord): b.txt
touch $(shell cat $<)
so in principle, I need something to "process" the dependency (*.txt) to create a list of the targets, like
$(shell cat $%): %.txt
echo $< > $#
but I cannot manage to get a reference to the dependency on the target side ($% does not work).
Any ideas? Maybe the whole approach is just a bad idea ;)
A combination of GNU make foreach, eval and call functions is probably what you need. With your example:
TXT := $(wildcard *.txt)
.PHONY: all
.DEFAULT_GOAL := all
define MY_MACRO
$(1)-targets := $$(shell cat $(1))
$$($(1)-targets): $(1)
echo $$< > $$#
all: $$($(1)-targets)
endef
$(foreach t,$(TXT),$(eval $(call MY_MACRO,$(t))))
(pay attention to the $$ in the macro definition, they are needed). And then:
$ make
make
echo a.txt > foo
echo a.txt > bar
echo a.txt > baz
echo b.txt > naarf
echo b.txt > fjoord
If you want the recipe to build all targets at once you'll need a recent enough GNU make version (4.3 or later) and its new rule with grouped targets (x y z&: w):
TXT := $(wildcard *.txt)
.PHONY: all
.DEFAULT_GOAL := all
define MY_MACRO
$(1)-targets := $$(shell cat $(1))
$$($(1)-targets)&: $(1)
touch $$($(1)-targets)
all: $$($(1)-targets)
endef
$(foreach t,$(TXT),$(eval $(call MY_MACRO,$(t))))
And then:
$ make
touch foo bar baz
touch naarf fjoord
Note that in this case we could also use a simpler and less GNU make-dependent solution. Just use empty dummy files as time stamps, for instance .a.txt.tag for a.txt, and a static pattern rule:
TXT := $(wildcard *.txt)
TAG := $(patsubst %,.%.tag,$(TXT))
.PHONY: all
all: $(TAG)
$(TAG): .%.tag: %
touch `cat $<` $#
I have the following makefile:
SHELL:/bin/bash
all: euro.n.{a,b}
euro.{a,b}:
touch euro.{a,b}
euro.n.{a,b}: euro.{a,b}
cat euro.a > euro.n.a
cat euro.b > euro.n.b
Now if I run make twice, the makefile won't recognize in the second run that the files euro.n.a and euro.n.b have already been created (and it will be executed again).
What is the problem?
What is the problem?
{a,b} is not syntax recognized by GNU make.
SHELL := /bin/bash (you missed = there) only affects the syntax of recipes.
One alternative solution:
SHELL := /bin/bash
all: $(addprefix euro.n.,a b)
euro.%:
touch $#
euro.n.% : euro.%
cp $< $#
I want
I am trying to compile some latex that has snippets of python code and the output of those snippets. I need the document to be always updated with the last changes made in the snippets and in their outputs, so the idea is maintain a makefile that could monitor this changes and generate the updated outputs.
So if I modify the file a/11.py, I want make to execute it to generate a new output a/11.out.
I have
This is my makefile
DOC=myPdf
STY=st
PY_DIR=a/
TEX=pdflatex -shell-escape -interaction=batchmode -file-line-error
$(DOC).pdf: $(PY_DIR)11.out $(PY_DIR)12.out $(DOC).tex $(STY).sty
$(TEX) $(DOC).tex
$(PY_DIR)11.out:
$(cd PY_DIR && python3 11.py > 11.out)
$(PY_DIR)12.out:
$(cd PY_DIR && python3 12.py > 12.out)
.PHONY: clean
clean:
rm *.aux *.log > /dev/null 2>&1
I wonder
Even when the file a/11.out doesn't exist, and I instruct make a/11.out make says: make: 'a/11.out' is up to date. (I am still learning make, so I probably have more mistakes).
I saw
Make in subfolder, but because I am not using $(MAKE), I cannot use it.
Similar question, but I don't think it is the same.
Thank you for your time :)
Update
This is my new version, based in the answer of Renaud (thanks for your help), some python scripts are intended to output text (xxxt.py), and others to plot images (xxxi.py), so there is no redirection for them:
DOC :=myPdf
STY :=st
PY_DIR :=a/
TEX :=pdflatex -shell-escape -interaction=batchmode -file-line-error
PYS := $(wildcard $(PY_DIR)*.py)
OUTS := $(patsubst %.py,%.out,$(PYS))
.PHONY: all clean
all: $(DOC).pdf
%.pdf: %.tex $(STY).sty $(OUTS)
$(TEX) $<
$(PY_DIR)%.out: $(PY_DIR)%t.py
cd $(PY_DIR) && python3 $*t.py > $*.out
$(PY_DIR)%.png: $(PY_DIR)%i.py
cd $(PY_DIR) && python3 $*i.py
clean:
rm *.aux *.log > /dev/null 2>&1
The directory looks like this:
./st.sty
./myPdf.tex
./myPdf.pdf
./a/11t.py
./a/11.out
./a/12i.py
./a/12.png
./a/21t.py
./a/...
However, now right after modifying myPdf.tex, make says make: Nothing to be done for 'all'.
What am I doing wrong?
Your recipes are wrong. Make expands the recipes before passing them to the shell. As there is no make variable named cd PY_DIR && python3 11.py > 11.out, $(cd PY_DIR && python3 11.py > 11.out) expands as the empty string and make considers that there is nothing to do for $(PY_DIR)11.out. Just write your recipes as plain shell (and fix the other bug with the unexpanded PY_DIR):
$(PY_DIR)11.out:
cd $(PY_DIR) && python3 11.py > 11.out
$(PY_DIR)12.out:
cd $(PY_DIR) && python3 12.py > 12.out
Note: if you want make to re-run the recipes when your python scripts change you should let him know that the output files depend on the python scripts. The best is probably to use a pattern rule instead of one specific rule per file:
$(PY_DIR)%.out: $(PY_DIR)%.py
cd $(PY_DIR) && python3 $*.py > $*.out
($* is a make automatic variable, it expands as the stem of the pattern).
A few more improvements:
You could ask make to find alone the python scripts, compute the names of the output files and store all this in make variables that you can used in your other rules.
You can use a pattern rule for the xx.tex -> xx.pdf process. And use another make automatic variable for it: $< that expands as the first prerequisite.
DOC := myPdf
STY := st
PY_DIR := a/
TEX := pdflatex -shell-escape -interaction=batchmode -file-line-error
PYS := $(wildcard $(PY_DIR)*.py)
OUTS := $(patsubst %.py,%.out,$(PYS))
.PRECIOUS: $(OUTS)
.PHONY: all clean
all: $(DOC).pdf
%.pdf: %.tex $(OUTS) $(STY).sty
$(TEX) $<
$(PY_DIR)%.out: $(PY_DIR)%.py
cd $(PY_DIR) && python3 $*.py > $*.out
.PHONY: clean
clean:
rm *.aux *.log > /dev/null 2>&1
Note: I declared $(OUTS) as precious such that make does not delete them when it is done with the building of $(DOC).pdf.
Update with the new specifications and separated python scripts for xx.out and xx.png production:
DOC := myPdf
STY := st
PY_DIR := a
TEX := pdflatex -shell-escape -interaction=batchmode -file-line-error
PYTS := $(wildcard $(PY_DIR)/*t.py)
PYIS := $(wildcard $(PY_DIR)/*i.py)
OUTS := $(patsubst $(PY_DIR)/%t.py,$(PY_DIR)/%.out,$(PYTS))
PNGS := $(patsubst $(PY_DIR)/%i.py,$(PY_DIR)/%.png,$(PYIS))
.PRECIOUS: $(OUTS) $(PNGS)
.PHONY: all clean
all: $(DOC).pdf
%.pdf: %.tex $(STY).sty $(OUTS) $(PNGS)
$(TEX) $<
$(PY_DIR)/%.out: $(PY_DIR)/%t.py
cd $(PY_DIR) && python3 $*t.py > $*.out
$(PY_DIR)/%.png: $(PY_DIR)/%i.py
cd $(PY_DIR) && python3 $*i.py
clean:
rm -f *.aux *.log > /dev/null 2>&1
Notes:
I slightly modified the definition of PY_DIR such that, when used in other parts of the Makefile, it is clear that it is a directory path. Just a matter of taste, I guess.
I added the -f option to your clean recipe such that it doesn't fail if the files to delete do not exist.
Update:
As noted by MadScientist in a comment, using $* is less generic than referring to the target ($#) and the prerequisite ($<). But as we are operating not directly on them but on their directory ($(PY_DIR)) and base file names (xx[it].py, xx.out, xx.png), switching from $* to other, more generic, automatic variables is not that simple.
But make has some more tricks that can help here: $#, $<... have variants ($(#F), $(#D)...) that expand to just the directory part or the file part. Note that, according the GNU make manual:
These variants are semi-obsolete in GNU make since the functions dir
and notdir can be used to get a similar effect.
Anyway, if we wanted to avoid $* here is what we could use instead:
$(PY_DIR)/%.out: $(PY_DIR)/%t.py
cd $(#D) && python3 $(<F) > $(#F)
$(PY_DIR)/%.png: $(PY_DIR)/%i.py
cd $(#D) && python3 $(<F)
Or (modern version):
$(PY_DIR)/%.out: $(PY_DIR)/%t.py
cd $(dir $#) && python3 $(notdir $<) > $(notdir $#)
$(PY_DIR)/%.png: $(PY_DIR)/%i.py
cd $(dir $#) && python3 $(notdir $<)
Note that I do not want to redirect all make output to file. I only want the output from a $(warning) command to file.
someTarget:
$(warning building $# using $?) >> someLogFile.txt
My example above does not redirect the output from $(warning to someLogFile. Is there a way to do it? Maybe redirect it to a variable and then echo that to a file?
Thanks.
is there a way to redirect $(warning) or $(info) statements to file?
Here's one for GNU Make, but it's not pretty:
Makefile
LOG := log.txt
TARGET_ACQUIRED = \
$(shell echo 'NO_SUCH_TARGET:' | $(MAKE) --eval='$$(info Target acquired: $#...)' -s -f - >> $(LOG))
target_a: target_b
$(TARGET_ACQUIRED)
touch $#
target_b:
$(TARGET_ACQUIRED)
touch $#
clean:
rm -f target_* $(LOG)
With which you'll get:
$ make
touch target_b
touch target_a
$ cat log.txt
Target acquired: target_b...
Target acquired: target_a...
To understand this ruse, see the GNU make commandline options.
If you want this for the purpose of debugging a makefile, you'd probably
fare better with GNU Make's --debug options, documented at the same place.
So I am writing a makefile that will take some files (*.in) as input to my C++ program and compare their output (results.out) to given correct output (*.out).
Specifically I have files t01.in, t02.in, t03.in, t04.in, and t05.in.
I have verified that $TESTIN = t01.in t02.in t03.in t04.in t05.in.
The problem is that it seems to run the %.in: %.out block only for three of these files, 1,3, and 4. Why is it doing this?
OUTPUT = chart
COMPILER = g++
SOURCES = chart.cpp
HEADERS =
OBJS = $(SOURCES:.cpp=.o)
TESTIN = tests/*.in
all: $(OUTPUT)
$(OUTPUT): $(OBJS)
$(COMPILER) *.o -o $(OUTPUT)
%.o: %.cpp
clear
$(COMPILER) -c $< -o $#
test: $(TESTIN)
%.in: %.out
./$(OUTPUT) < $# > tests/results.out
printf "\n"
ifeq ($(diff $< tests/results.out), )
printf "\tTest of "$#" succeeded for stdout.\n"
else
printf "\tTest of "$#" FAILED for stdout!\n"
endif
Additionally, if there is a better way of accomplishing what I am trying to do, or any other improvements I could make to this makefile (as I am rather new at this), suggestions would be greatly appreciated.
EDIT: If I add a second dependency to the block (%.in: %.out %.err), it runs the block for all five files. Still no idea why it works this way but not the way before.
First, I don't see how TESTIN can be correct. This line:
TESTIN = tests/*.in
is not a valid wildcard statement in Make; it should give the variable TESTIN the value tests/*.in. But let's suppose it has the value t01.in t02.in t03.in t04.in t05.in or tests/t01.in tests/t02.in tests/t03.in tests/t04.in tests/t05.in, or wherever these files actually are.
Second, as #OliCharlesworth points out, this rule:
%.in: %.out
...
is a rule for building *.in files, which is not what you intend. As for why it runs some tests and not others, here is my theory:
The timestamp of t01.out is later than that of t01.in, so Make decides that it must "rebuild" t01.in; likewise t03.in and t04.in. But the timestamp of t02.out is earlier than that of t02.in, so Make does not attempt to "rebuild" t02.in; likewise t05.in. The timestamps of t02.err and t05.err are later than those of t02.in and t05.in, respectively, so when you add the %.err prerequisite, Make runs all tests. You can test this theory by checking the timestamps and experimenting with touch.
Anyway, let's rewrite it. We need a new target for a new rule:
TESTS := $(patsubst %.in,test_%,$(TESTIN)) # test_t01 test_t02 ...
.PHONY: $(TESTS) # because there will be no files called test_t01, test_t02,...
$(TESTS): test_%: %.in %.out
./$(OUTPUT) < $< > tests/results.out
Now for the conditional. Your attempted conditional is in Make syntax; Make will evaluate it before executing any rule, so tests/result.out will not yet exist, and variables like $< will not yet be defined. We must put the conditional inside the command, in shell syntax:
$(TESTS): test_%: %.in %.out
./$(OUTPUT) < $< > tests/results.out
if diff $*.out tests/results.out >/dev/null; then \
echo Test of $* succeeded for stdout.; \
else echo Test of $* FAILED for stdout!; \
fi
(Note that only the first line of the conditional must begin with a TAB.)