Dynamic targets based on the dependency in Makefile - makefile

There are a couple of kind of similar issues but I could not fit any of the proposed concepts to my case.
Just to give a little bit of context: I have a set of Julia files which create plots as PDFs which are part of a make procedure to create scientific papers, something like:
plots = $(shell find $(PLOT_PATH)/*.jl | sed 's/\.jl/\.pdf/g')
$(PLOT_PATH)/%.pdf: $(PLOT_PATH)/%.jl $(JULIA_SYSIMAGE)
$(JL) --project $< -o $(PLOT_PATH)
$(DOCUMENT_FILENAME).pdf: FORCE $(plots) $(figures)
latexmk $(DOCUMENT_FILENAME).tex
In the current setup, each XYZ.jl file is creating a XYZ.pdf file and it works absolutely fine.
Now I am dealing with cases where it would be much easier to create multiple plots from single Julia files, so a script like this:
#!/usr/bin/env julia
using PGFPlotsX
...
...
pgfsave("whatever.pdf")
pgfsave("another.pdf")
pgfsave("yetanother.pdf")
so that one could do a grep pgfsave SCRIPT | awk... to figure out the targets. However, I could not figure out how to generate dynamic targets (plots) based on the contents of the dependency file (Julia script).
An MWE for my problem is the following: I have a couple of files (dependencies) which are generating a bunch of targets, which are defined inside those files (and can be access via awk/grep/sed/whatever). For now, let's say that these are simply *.txt files and each line is a target.
file: a.txt
foo
bar
baz
file: b.txt
naarf
fjoord
A very basic (non-working) manual Makefile to demonstrate the goal would be something like this (it does not work as it cannot figure out how to make foo etc. but it shows the pattern for *.txt which needs to be repeated):
file: Makefile
all_products := $(shell find *.txt | xargs cat)
final_product: $(all_products)
echo $< > $#
(foo bar baz): a.txt
touch $(shell cat $<)
(narf fjoord): b.txt
touch $(shell cat $<)
so in principle, I need something to "process" the dependency (*.txt) to create a list of the targets, like
$(shell cat $%): %.txt
echo $< > $#
but I cannot manage to get a reference to the dependency on the target side ($% does not work).
Any ideas? Maybe the whole approach is just a bad idea ;)

A combination of GNU make foreach, eval and call functions is probably what you need. With your example:
TXT := $(wildcard *.txt)
.PHONY: all
.DEFAULT_GOAL := all
define MY_MACRO
$(1)-targets := $$(shell cat $(1))
$$($(1)-targets): $(1)
echo $$< > $$#
all: $$($(1)-targets)
endef
$(foreach t,$(TXT),$(eval $(call MY_MACRO,$(t))))
(pay attention to the $$ in the macro definition, they are needed). And then:
$ make
make
echo a.txt > foo
echo a.txt > bar
echo a.txt > baz
echo b.txt > naarf
echo b.txt > fjoord
If you want the recipe to build all targets at once you'll need a recent enough GNU make version (4.3 or later) and its new rule with grouped targets (x y z&: w):
TXT := $(wildcard *.txt)
.PHONY: all
.DEFAULT_GOAL := all
define MY_MACRO
$(1)-targets := $$(shell cat $(1))
$$($(1)-targets)&: $(1)
touch $$($(1)-targets)
all: $$($(1)-targets)
endef
$(foreach t,$(TXT),$(eval $(call MY_MACRO,$(t))))
And then:
$ make
touch foo bar baz
touch naarf fjoord
Note that in this case we could also use a simpler and less GNU make-dependent solution. Just use empty dummy files as time stamps, for instance .a.txt.tag for a.txt, and a static pattern rule:
TXT := $(wildcard *.txt)
TAG := $(patsubst %,.%.tag,$(TXT))
.PHONY: all
all: $(TAG)
$(TAG): .%.tag: %
touch `cat $<` $#

Related

Can I simplify this Makefile involving files in subfolders?

I have, for example, the following Makefile to generate PDF files from Markdown files in subdirectories:
FOLDERS = f1 f2 f3
.PHONY: $(FOLDERS)
f1: f1/f1.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
f2: f2/f2.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
f3: f3/f3.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
The expected result is that make f1 requires the existence of f1/f1.md, and generates the resulting PDF as f1/f1.pdf. The same for f2 and f3. This works, but the declarations seem unnecessarily repetitive.
Is there any way to combine these three rules into one, generic rule? That is, without needing to explicitly write out all of the paths to the PDF files or Markdown files, as I may be dynamically adding subfolders and I'd prefer to just change the definition of FOLDERS in the first line. I've googled around and tried a few things, but I feel like either I can't find the right incantation to use, or I'm missing a piece of knowledge about how Makefiles work. Could someone please point me in the right direction?
First, note that there's no good reason to use PHONY targets here, since these rules appear to be building files whose names are known beforehand. Targets like f1/f1.pdf would be much better.
Unfortunately we can't use a pattern rule when the stem (e.g. f1) is repeated in a prerequisite. But a "canned recipe" can do the trick:
define pdf_template
$(1): $(1)/$(1).md
cd $$# && pandoc $$(notdir $$^) -o $$(patsubst %.md,%.pdf,$$(notdir $$^))
endef
$(eval $(call pdf_template,f1))
$(eval $(call pdf_template,f2))
$(eval $(call pdf_template,f3))
(Note how you must escape the $ signs in the template.)
If those $(eval...) lines look too repetitive, you can replace them with a loop:
$(foreach folder,$(FOLDERS),$(eval $(call pdf_template,$(folder))))
EDIT: Come to think of it, there's another way. You can't construct a pattern rule that uses the stem more than once:
$(FOLDERS): %: %/%.md
cd $# && ... this won't work
And you can't use the automatic variables in the prerequisite list, because they aren't yet defined when they're needed:
$(FOLDERS): $#/$#.md
cd $# && ... this won't work either
But you can use them there if you use Secondary Expansion, which causes Make to expand the prereq list a second time:
.SECONDEXPANSION:
$(FOLDERS): $$#/$$#.md
cd $# && ... this works
Again, note the escaped $ symbols.

Read variable from file in target recipe enclosed in foreach

I have a top level Makefile that can create multiple targets, for which I have a for loop in place. In this file I am trying to read a text file with a version number that is created only when a sub makefile runs. I have got it to work but it looks very ugly, so I'm thinking there is possibly a better way to do it. Here is the top level Makefile:
T1=_release
T2=_debug
T3=_test
all: bin/mybin_release.bin
debug: bin/mybin_debug.bin
debug: export DBG:=1
test: bin/mybin_test.bin
test: export TST:=1
define build_foo
bin/mybin$$($(1)).bin: bin/abc$$($(1)).bin bin/xyz$$($(1)).bin
cat $$^ > $$#
VER=$$$$(cat bin/rev.txt) && mv bin/mybin$$($(1)).bin bin/mybin$$($(1))_$$$${VER}.bin
bin/abc$$($(1)).bin:
$(MAKE) -C mod1 # <--- rev.txt is produced here
bin/xyz$$($(1)).bin:
$(MAKE) -C mod2
endef
$(foreach suffix, T1 T2 T3, $(eval $(call build_foo,$(suffix))))
clean:
rm bin/*.bin bin/*.txt
Notice the attempt to grab file contents in VER. Is there a better/right way to do this? I cannot use eval since it runs at the beginning and rev.txt only gets created once the sub make runs.
Also, is this a decent way to build multiple targets(using foreach)? I use the exported variables to modify the target built by the sub makefile.
As far as I understand, this bin/rev.txt file, and all the bin/abcXXX.bin are produced when running $(MAKE) -C mod1. So, it is the same for all. What about:
include version.mk
version.mk: bin/rev.txt
{ printf 'VER = '; cat $<; } > $#
bin/rev.txt:
$(MAKE) -C mod1
Demo:
$ ls
Makefile
$ cat Makefile
include version.mk
all:
touch $(VER).txt
version.mk: rev.txt
{ printf 'VER = '; cat $<; } > $#
rev.txt:
echo "1.2.3" > $#
$ make --quiet
Makefile:1: version.mk: No such file or directory
$ ls
1.2.3.txt Makefile rev.txt version.mk
Explanation:
version.mk is a second makefile that make will look for. As per GNU make documentation, section 3.5:
To this end, after reading in all makefiles, make will consider each
as a goal target and attempt to update it. If a makefile has a rule
which says how to update it (found either in that very makefile or in
another one) or if an implicit rule applies to it (see Using Implicit
Rules), it will be updated if necessary. After all makefiles have been
checked, if any have actually been changed, make starts with a clean
slate and reads all the makefiles over again. (It will also attempt to
update each of them over again, but normally this will not change them
again, since they are already up to date.)
Note: as you wrote your makefile, $(MAKE) -C mod1 will be run several times, which is a waste. You could instead exploit a specificity of GNU make pattern rules: when they have multiple targets, make considers that all targets are produced by one single invocation of the recipe. Example:
$ cat Makefile
all: a.bin b.bin c.bin
a.%in b.%in c.%in:
#echo 'building a.bin b.bin c.bin'
$ make all
building a.bin b.bin c.bin
See? The recipe is executed only once to build the 3 targets. The only problem is that the % wildcard must match at least one character. So, in your case you could do something like (the character that % matches is the b of bin/):
T1 := _release
T2 := _debug
T3 := _test
suffixes := T1 T2 T3
pattern := $(foreach suffix,$(suffixes),%in/abc$($(suffix))) %in/rev.txt
$(pattern):
$(MAKE) -C mod1
This will tell make that $(MAKE) -C mod1 builds all the bin/abcXXX.bin and bin/rev.txt at once. Same with $(MAKE) -C mod2.
All in all, you could probably get completely rid of your build_foo generic rule by gluing all these features together. Something like:
T1 := _release
T2 := _debug
T3 := _test
suffixes := T1 T2 T3
patternabc := $(foreach suffix,$(suffixes),%in/abc$($(suffix)).bin) %in/rev.txt
patternxyz := $(foreach suffix,$(suffixes),%in/xyz$($(suffix)).bin)
include version.mk
all: bin/mybin_release_$(VER).bin
debug: bin/mybin_debug_$(VER).bin
debug: export DBG:=1
test: bin/mybin_test_$(VER).bin
test: export TST:=1
version.mk: bin/rev.txt
{ printf 'VER = '; cat $<; } > $#
$(patternabc):
$(MAKE) -C mod1
$(patternxyz):
$(MAKE) -C mod2
bin/mybin%_$(VER).bin: bin/abc%.bin bin/xyz%.bin
cat $^ > $#

How do I force a target to be rebuilt if a variable is set?

Assume I have a build-target foo:
foo:foo.c
$(CC) $(CFLAGS) $(ARGS) -c foo.c -o foo
Now, ARGS is something that I pass on the command line:
$ make ARGS:=-DX=1 foo
So, I need to bypass make's cleverness, because the foo target does not only depend on which files have changed, but also on the value of ARGS.
Is there something in make to do this? My hack (see answer) doesn't seem to be the most elegant but it works. Anything better?
Here is a general solution to your specific problem.
You want to be able to depend on a variable as a prerequisite. That is, you can make it a prerequisite to any target in your makefile, and when the value of the variable changes, you rebuild those targets.
Here is a function that does that, you use this function to declare a variable to be dependable, and then you can use it as a prerequisite.
Note that if the variable is not used on the command line, it will still mean that variable still has a value, namely, the empty string.
define DEPENDABLE_VAR
.PHONY: phony
$1: phony
#if [[ `cat $1 2>&1` != '$($1)' ]]; then \
echo -n $($1) > $1 ; \
fi
endef
#declare ARGS to be dependable
$(eval $(call DEPENDABLE_VAR,ARGS))
foo:foo.c ARGS
$(CC) $(CFLAGS) $(ARGS) -c foo.c -o foo
In fact, we could omit the need for "declaration", and just write a similar function that will make all variables dependable by default. But I don't like that. I prefer that the users that modify makefiles I write, declare their intentions explicitly. It is good for them :)
My solution was to create a dummy phony target:
.PHONY:dummy
dummy:
#:
and have foo depend on dummy if ARGS is nonempty:
foo:foo.c $(patsubst %,dummy,$(ARGS))
Note on Mark's excellent answer
The bare necessities of Mark's answer are actually very simple. It really boils down to just:
.PHONY: phony
ARGS: phony
#if [[ `cat ARGS 2>&1` != '$(ARGS)' ]]; then echo -n $(ARGS) >ARGS; fi
The rest of his code is just to let you reproduce the recipe for other variable names without repeating yourself. Useful in practice, but the above version will help you see what's going on more easily.
In fact, my answer can even be made general (like Mark's) for any variable name, but in a less complicated way as follows:
.PHONY: phony
.ARG~%: phony
#if [[ `cat .ARG~$* 2>&1` != '$($*)' ]]; then echo -n $($*) >.ARG~$*; fi
Then you simply add .ARG~MYVARNAME as a dependency to any target to make that target depend on variable MYVARNAME.
Note that the dot in front of .ARG~% simply causes it to create a dependency-tracking file that is 'hidden' in linux.
So in your case, you would do:
foo: foo.c .ARG~ARGS
$(CC) $(CFLAGS) $(ARGS) -c foo.c -o foo
I don't understand how the other solutions are supposed to work. If the ARGS target is .PHONY or depends on a .PHONY, then it will always be run, right?
Here is my solution using the $(file) function in newer versions of gmake:
.PHONY: FORCE
define DEPENDABLE_VAR
$(1):
echo -n $($(1)) > $(1)
ifneq ("$(file <$(1))","$($(1))")
$(1): FORCE
endif
endef
#declare ARGS to be dependable
$(eval $(call DEPENDABLE_VAR,ARGS))
foo: foo.c ARGS
touch foo
And the result:
~/stuff/makevars> make foo ARGS=1
echo -n 1 > ARGS
touch foo
~/stuff/makevars> make foo ARGS=1
make: 'foo' is up to date.
~/stuff/makevars> make foo ARGS=2
echo -n 2 > ARGS
touch foo
~/stuff/makevars> make foo ARGS=2
make: 'foo' is up to date.

GNU make: friendly way to choose mutually exclusive files

I'm writing a GNU makefile to create a license file like so: if the customer's file exists then copy it to the package directory, else use the generic license. I have about a dozen files that have the same copy pattern. The goal is to copy one of the two mutually exclusive source files onto the target filename.
Is there a better way to express this with GNU make syntax? Here's what I currently have. I considered copying all GENERIC files to the directory, then overwriting with the existing CUST files.
$(PDIR)/license.txt: | $(PDIR)
if [ -f Package/license.txt.$(CUST) ] ; \
then \
cat Package/license.txt.$(CUST) >$(PDIR)/license.txt ; \
else \
cat Package/license.txt.GENERIC >$(PDIR)/license.txt ; \
fi
Edit: Thanks to MadScientist for the help. Here's my final working version:
TARGETS = license.txt ...
final: $(addprefix ${PDIR}/,${TARGETS})
#echo some output
$(foreach T,${TARGETS},$(eval ${PDIR}/$T: $(firstword $(wildcard Package/$T.${CUST} Package/$T.GENERIC Package/$T)) | ${PDIR}))
$(addprefix ${PDIR}/,${TARGETS}):
#echo Creating substituted version of $< as $#
#sed --expression="\
... \
< '$<' \
> '$#'
${PDIR}:
mkdir $#
You can use wildcard, then (as always) you should use automatic variables. Like this:
$(PDIR)/license.txt: $(firstword $(wildcard Package/license.txt.$(CUST) Package/license.txt)) | $(PDIR)
cat '$<' > '$#'
If you have a lot of these you can also use a loop to define the targets:
TARGETS = license.txt
$(foreach T,$(TARGETS),$(eval $(PDIR)/$T: $(firstword $(wildcard Package/$T.$(CUST) Package.$T) | $(PDIR))))
$(addprefix $(PDIR)/,$(TARGETS)):
cat '$<' > '$#'

Finding makefile dependencies

I have several widgets denoted by a config.xml in their root in a directory layout.
The GNUmakefile I have here is able to build them. Though if I update the folders, the dependencies aren't tracked. I don't want to depend on a clean target obviously, so how do I track the contents of each folder?
WGTS := $(shell find -name 'config.xml' | while read wgtdir; do echo `dirname $$wgtdir`.wgt; done )
all: $(WGTS)
%.wgt:
#cd $* && zip -q -r ../$(shell basename $*).wgt .
#echo Created $#
clean:
rm -f $(WGTS)
I hoped something like:
%.wgt: $(shell find $* -type f)
Would work, but it doesn't. Help.
Combining Beta's idea with mine:
WGTS := $(shell find -name config.xml)
WGTS := $(WGTS:/config.xml=.wgt)
WGTS_d := $(WGTS:.wgt=.wgt.d)
all: $(WGTS)
clean:
rm -f $(WGTS) $(WGTS_d)
-include $(WGTS_d)
define WGT_RULE
$(1): $(shell find $(1:.wgt=))
$(1:.wgt=)/%:
#
endef
$(foreach targ,$(WGTS),$(eval $(call WGT_RULE,$(targ))))
%.wgt:
#echo Creating $#
#(echo -n "$#: "; find $* -type f | tr '\n' ' ') > $#.d
#cd $* && zip -q -r ../$(shell basename $*).wgt .
Example:
$ mkdir -p foo bar/nested
$ touch {foo,bar/nested}/config.xml
$ make
Creating bar/nested.wgt
Creating foo.wgt
$ make
make: Nothing to be done for `all'.
$ touch foo/a
$ make
Creating foo.wgt
$ rm foo/a
$ make
Creating foo.wgt
$ make
make: Nothing to be done for `all'.
The only potential problem here is the dummy rule that lets make ignore targets it doesn't know how to build which are nested inside the directories. (foo/a in my example.) If those are real targets that make needs to know how to build, the duplicate recipe definition may be a problem.
Probably the best way to do this is to create the prerequisite lists explicitly, beforehand:
define WGT_RULE
$(1).wgt: $(wildcard $(1)/*)
endef
$(foreach targ,$(WGTS),$(eval $(call WGT_RULE,$(targ))))
There is another way that's very clever (a phrase that makes a good programmer wary). Years ago I came up with a left-handed kludge for treating a directory as a prerequisite. I'll see if I can dig up my old notebooks if the above isn't good enough.
EDIT:
Sorry, I didn't consider subdirectories. Here's a complete makefile (I left out the clean rule) that should do the trick.
WGTS := $(shell find -name 'config.xml' | while read wgtdir; do echo `dirname $\
$wgtdir`.wgt; done )
all: $(WGTS)
# This constructs a rule without commands ("foo.wgt: foo/bar.txt foo/baz.dat...").
define WGT_RULE
$(1).wgt: $(shell find $(1))
endef
# This invokes the above to create a rule for each widget.
$(foreach targ,$(WGTS),$(eval $(call WGT_RULE,$(targ))))
%.wgt:
#cd $* && zip -q -r ../$(shell basename $*).wgt .
#echo Created $#

Resources