( my question is different from Force Makefile to execute script before building targets )
I've got the following Makefile:
.PHONY: dump_params all
all: tmpA
tmpA: tmpB tmpC dump_params
cat $(filter tmp%,$^) > $#
tmpB: dump_params
touch $#
tmpC: dump_params
touch $#
dump_params:
echo "Makefile was run." >> config.txt
with the target dump_params I want to create/append a file each time a new target is invoked (to keep track of the version of the tools used). However, when I call
make tmpA
all the targets are built from scratch
$ make tmpA
echo "Makefile was run " >> config.txt
touch tmpB
touch tmpC
cat tmpB tmpC > tmpA
$ make tmpA
echo "Makefile was run." >> config.txt
touch tmpB
touch tmpC
cat tmpB tmpC > tmpA
How can I prevent Make to re-build everything because of that target 'dump_params'? Is there a another way to create this kind of log file ?
EDIT: I'm using a parallel make (option -j). Defining a macro to create the file in the statements section is not an option.
Use order-only prerequisites?
.PHONY: dump_params all
all: tmpA
tmpA: tmpB tmpC | dump_params
cat $(filter tmp%,$^) > $#
tmpB: | dump_params
touch $#
tmpC: | dump_params
touch $#
dump_params:
echo "Makefile was run." >> config.txt
Another option is to use immediately expanded shell functions, like:
__dummy := $(shell echo "Makefile was run." >> config.txt)
Since it's immediately expanded the shell script will be invoked once, as the makefile is read in. There's no need to define a dump_params target or include it as a prerequisite. This is more old-school, but has the advantage that it will run for every invocation of make, without having to go through and ensure every target has the proper order-only prerequisite defined.
Non-answer, but snakemake (and there are others, I suspect) supports tracking of rules (the code), parameters, inputs, and executable versions.
https://bitbucket.org/johanneskoester/snakemake
Related
There are a couple of kind of similar issues but I could not fit any of the proposed concepts to my case.
Just to give a little bit of context: I have a set of Julia files which create plots as PDFs which are part of a make procedure to create scientific papers, something like:
plots = $(shell find $(PLOT_PATH)/*.jl | sed 's/\.jl/\.pdf/g')
$(PLOT_PATH)/%.pdf: $(PLOT_PATH)/%.jl $(JULIA_SYSIMAGE)
$(JL) --project $< -o $(PLOT_PATH)
$(DOCUMENT_FILENAME).pdf: FORCE $(plots) $(figures)
latexmk $(DOCUMENT_FILENAME).tex
In the current setup, each XYZ.jl file is creating a XYZ.pdf file and it works absolutely fine.
Now I am dealing with cases where it would be much easier to create multiple plots from single Julia files, so a script like this:
#!/usr/bin/env julia
using PGFPlotsX
...
...
pgfsave("whatever.pdf")
pgfsave("another.pdf")
pgfsave("yetanother.pdf")
so that one could do a grep pgfsave SCRIPT | awk... to figure out the targets. However, I could not figure out how to generate dynamic targets (plots) based on the contents of the dependency file (Julia script).
An MWE for my problem is the following: I have a couple of files (dependencies) which are generating a bunch of targets, which are defined inside those files (and can be access via awk/grep/sed/whatever). For now, let's say that these are simply *.txt files and each line is a target.
file: a.txt
foo
bar
baz
file: b.txt
naarf
fjoord
A very basic (non-working) manual Makefile to demonstrate the goal would be something like this (it does not work as it cannot figure out how to make foo etc. but it shows the pattern for *.txt which needs to be repeated):
file: Makefile
all_products := $(shell find *.txt | xargs cat)
final_product: $(all_products)
echo $< > $#
(foo bar baz): a.txt
touch $(shell cat $<)
(narf fjoord): b.txt
touch $(shell cat $<)
so in principle, I need something to "process" the dependency (*.txt) to create a list of the targets, like
$(shell cat $%): %.txt
echo $< > $#
but I cannot manage to get a reference to the dependency on the target side ($% does not work).
Any ideas? Maybe the whole approach is just a bad idea ;)
A combination of GNU make foreach, eval and call functions is probably what you need. With your example:
TXT := $(wildcard *.txt)
.PHONY: all
.DEFAULT_GOAL := all
define MY_MACRO
$(1)-targets := $$(shell cat $(1))
$$($(1)-targets): $(1)
echo $$< > $$#
all: $$($(1)-targets)
endef
$(foreach t,$(TXT),$(eval $(call MY_MACRO,$(t))))
(pay attention to the $$ in the macro definition, they are needed). And then:
$ make
make
echo a.txt > foo
echo a.txt > bar
echo a.txt > baz
echo b.txt > naarf
echo b.txt > fjoord
If you want the recipe to build all targets at once you'll need a recent enough GNU make version (4.3 or later) and its new rule with grouped targets (x y z&: w):
TXT := $(wildcard *.txt)
.PHONY: all
.DEFAULT_GOAL := all
define MY_MACRO
$(1)-targets := $$(shell cat $(1))
$$($(1)-targets)&: $(1)
touch $$($(1)-targets)
all: $$($(1)-targets)
endef
$(foreach t,$(TXT),$(eval $(call MY_MACRO,$(t))))
And then:
$ make
touch foo bar baz
touch naarf fjoord
Note that in this case we could also use a simpler and less GNU make-dependent solution. Just use empty dummy files as time stamps, for instance .a.txt.tag for a.txt, and a static pattern rule:
TXT := $(wildcard *.txt)
TAG := $(patsubst %,.%.tag,$(TXT))
.PHONY: all
all: $(TAG)
$(TAG): .%.tag: %
touch `cat $<` $#
Is there a way to force a complete rebuild (eg -B) from within a makefile?
I have added a pre-build step that increments a build number, stored in a text file, that is used by my project. The build number is only incremented if newbuild=1 is passed as an argument to 'make'.
pre-build:
# This option increments build number
ifdef newbuild
increment_build_number
endif
If this code is called, I would like to force a complete rebuild so that nothing gets out of sync, and I don't have to type -B whenever newbuild=1 is used. Is there a way to do this?
Thanks!
You could:
declare the text file with the build number as a prerequisite of the other targets,
use make conditionals to also declare it phony if newbuild is defined.
Demo where build.txt is the text file with the build number and target is one of the targets to rebuild when newbuild is defined:
# Makefile
BUILD := build.txt
TARGETS := target
ifdef newbuild
.PHONY: $(BUILD)
endif
$(TARGETS): $(BUILD)
touch $#
$(BUILD):
#[ -f "$#" ] && build=$$(cat "$#") || build=0; \
printf 'build '; \
printf '%d\n' "$$((build+1))" | tee "$#"; \
$ make
build 1
touch target
$ make
make: 'target' is up to date.
$ make newbuild=1
build 2
touch target
I am trying to write a Makefile that builds PDF outputs with LaTeX, using Latexmk. The outputs have basically the same rule, with different prerequisites, so I tried generalising my original Makefile using GNU Make's "secondary expansion". (I also created .PHONY targets, also with secondary expansion, to make it more user-friendly.) However, this is causing the prerequisite rules to always be run, even when they don't need to be. Fortunately, Latexmk is clever enough to avoid doing unnecessary work, but I wonder if I'm doing anything wrong...
To try to abstract what I'm attempting:
,-> foo -> build/foo.pdf
all -{
`-> bar -> build/bar.pdf
That is, the all target builds foo and bar. These targets open the respective PDF file, which have a prerequisite of build/X.pdf (where X is foo or bar). These are genuine targets which build the appropriate PDF file.
This is what I've come up with:
TARGETS = foo bar
BUILD_DIR = build
OUTPUTS = $(TARGETS:%=$(BUILD_DIR)/%.pdf)
commonSRC = src/preamble.tex src/header.tex # etc...
fooSRC = src/foo.tex $(commonSRC) # etc...
barSRC = src/bar.tex $(commonSRC) # etc...
all: $(TARGETS)
.SECONDEXPANSION:
$(TARGETS): $(BUILD_DIR)/$$#.pdf
open $<
# FIXME This isn't quite right: This rule is still getting called by the
# above rule, even when it doesn't need to be. Latexmk is clever enough
# not to do any extra work, but it shouldn't run at all.
.SECONDEXPANSION:
$(OUTPUTS): $$($$(subst .pdf,SRC,$$(#F))) $(BUILD_DIR)
latexmk -outdir=$(BUILD_DIR) -auxdir=$(BUILD_DIR) -pdf $<
$(BUILD_DIR):
mkdir $#
clean:
rm -rf $(BUILD_DIR)
.PHONY: all $(TARGETS) clean
Just to be clear: The rule for build/X.pdf should run whenever the files enumerated in XSRC (again, where X is foo or bar) are newer than the PDFs, or the PDFs don't exist; but it should not run otherwise.
I believe that this got somewhat complex, more than it needs to be. Part of these second expansion statements can be just replaced with static pattern rules. The other thing is that .SECONDEXPANSION: makes all further Makefile contents to be subject to second expansion, so you don't need to explicitly state it before every target (it would be much clearer to mark .PHONY targets this way to quickly see if a target is phony or not).
Nevertheless, I believe that most important issue here is that you mention a directory as a prerequisite. Remember that make decides on whether to rebuild the target based on dependencies timestamp, and a directory gets its timestamp always updated whenever a file in this directory is updated. Therefore, whenever you write $(BUILD_DIR)/foo.pdf, $(BUILD_DIR) timestamp gets updated and the next call will build again since the directory is newer. You can avoid it by specifying a directory as an order-only prerequisite (which means: build if it doesn't exist, but do not check timestamp).
Putting it all together I would make it this way:
TARGETS = foo bar
BUILD_DIR = build
commonSRC = src/preamble.tex src/header.tex # etc...
fooSRC = src/foo.tex $(commonSRC) # etc...
barSRC = src/bar.tex $(commonSRC) # etc...
.SECONDEXPANSION:
.PHONY: all
all: $(TARGETS)
.PHONY: $(TARGETS)
$(TARGETS): %: $(BUILD_DIR)/%.pdf
echo open $<
$(BUILD_DIR)/%.pdf: $$($$*SRC) | $(BUILD_DIR)
echo latexmk -outdir=$(BUILD_DIR) -auxdir=$(BUILD_DIR) -pdf $< > $#
$(BUILD_DIR):
mkdir -p $#
.PHONY: clean
clean:
rm -rf $(BUILD_DIR)
Output:
$ make all
mkdir -p build
echo latexmk -outdir=build -auxdir=build -pdf src/foo.tex > build/foo.pdf
echo open build/foo.pdf
open build/foo.pdf
echo latexmk -outdir=build -auxdir=build -pdf src/bar.tex > build/bar.pdf
echo open build/bar.pdf
open build/bar.pdf
$ make all
echo open build/foo.pdf
open build/foo.pdf
echo open build/bar.pdf
open build/bar.pdf
Note that subsequent call did not attempt to build anything, just open the pdf. It still reacts on the file change however:
$ touch src/foo.tex
$ make all
echo latexmk -outdir=build -auxdir=build -pdf src/foo.tex > build/foo.pdf
echo open build/foo.pdf
open build/foo.pdf
echo open build/bar.pdf
open build/bar.pdf
$ touch src/header.tex
$ make all
echo latexmk -outdir=build -auxdir=build -pdf src/foo.tex > build/foo.pdf
echo open build/foo.pdf
open build/foo.pdf
echo latexmk -outdir=build -auxdir=build -pdf src/bar.tex > build/bar.pdf
echo open build/bar.pdf
open build/bar.pdf
I have the following Makefile:
ifneq ($(MAKECMDGOALS),clean)
-include generated.mk
endif
FOO ?= foo
all: a.txt
a.txt:
echo $(GEN_FOO) > $#
generated.mk: Makefile
echo GEN_FOO = $(FOO) > $#
.PHONY: clean
clean:
$(RM) a.txt
$(RM) generated.mk
It works OK when building single targets:
$ make clean
rm -f a.txt
rm -f generated.mk
$ make all
echo GEN_FOO = foo > generated.mk
echo foo > a.txt
However when I try to build multiple targets at once things go not so smooth:
$ make clean all
rm -f a.txt
rm -f generated.mk
echo foo > a.txt
$ make all
echo GEN_FOO = foo > generated.mk
make: Nothing to be done for 'all'.
It gets even worse if variables were provided:
$ make clean
rm -f a.txt
rm -f generated.mk
$ make FOO=bar clean all
echo GEN_FOO = bar > generated.mk
rm -f a.txt
rm -f generated.mk
echo bar > a.txt
$ make all
echo GEN_FOO = foo > generated.mk
make: Nothing to be done for 'all'.
$ make FOO=bar clean all
rm -f a.txt
rm -f generated.mk
echo foo > a.txt
Are there any ways to fix such incorrect behavior?
Make is doing exactly what you told it to do, and you haven't told us what you want it to do that's different than what you told it to do (saying fix such incorrect behavior doesn't really help us when you don't define what's incorrect about the behavior), so we can't help you very much.
You are probably getting confused about the interaction between included makefiles and comparing $(MAKECMDGOALS). Please note:
ifneq ($(MAKECMDGOALS),clean)
this will not match unless you specify exactly one target: clean. In situations where you specify multiple targets, one of which is clean, that will match because clean all is not equal to clean. So, when you run make clean all make will include the generated makefile, and will generate it if it doesn't exist.
Because generated include files are only rebuilt once, when the makefile is first parsed, it's not possible to say something like: "first run rule X (e.g., clean) then rebuild the included makefiles, then reinvoke make".
However, it's pretty much always a bad idea to invoke make with clean all. This is because if you were to ever try to add -j for parallelism, the clean and the build would be running in parallel and corrupt everything.
One semi-common option is to provide a different rule that will do both, something like this:
rebuild:
$(MAKE) clean
$(MAKE) all
then run make rebuild instead.
You can certainly force the behavior with the help of the shell. For instance, in bash you could use
for target in "clean" "all";
do
make $target;
done
and if you were going to re-do the procedure a lot you could either make it an executable script or wrap it in a shell function.
Note that I do not want to redirect all make output to file. I only want the output from a $(warning) command to file.
someTarget:
$(warning building $# using $?) >> someLogFile.txt
My example above does not redirect the output from $(warning to someLogFile. Is there a way to do it? Maybe redirect it to a variable and then echo that to a file?
Thanks.
is there a way to redirect $(warning) or $(info) statements to file?
Here's one for GNU Make, but it's not pretty:
Makefile
LOG := log.txt
TARGET_ACQUIRED = \
$(shell echo 'NO_SUCH_TARGET:' | $(MAKE) --eval='$$(info Target acquired: $#...)' -s -f - >> $(LOG))
target_a: target_b
$(TARGET_ACQUIRED)
touch $#
target_b:
$(TARGET_ACQUIRED)
touch $#
clean:
rm -f target_* $(LOG)
With which you'll get:
$ make
touch target_b
touch target_a
$ cat log.txt
Target acquired: target_b...
Target acquired: target_a...
To understand this ruse, see the GNU make commandline options.
If you want this for the purpose of debugging a makefile, you'd probably
fare better with GNU Make's --debug options, documented at the same place.