Makefile: target-specific variables in target dependencies declaration - makefile

I'm trying to write a makefile that uses target-specific variables, but while the variables are set correctly in each target and prerequisite body, the prerequisites list itself isn't updated with the variable, thus causing the wrong prerequisite to be checked and called.
How can I update target-specific variables in the prerequisites too?
In the example below, both make foo and make bar should print "world", but make foo prints "hello".
X=hello
hello:
echo "hello"
world:
echo "world"
foo:X=world
foo:$(X)
bar:X=world
bar:
make $(X)
The goal I'm trying to achieve is that different targets will build similar prerequisites - the same files, in different folders - by passing the folders as a target-specific variable. The issue is that as in the example below, if one target is called first (foo, in the example), calling the second will not do anything.
DIR=fooDir
FILE=$(DIR)/filename
$(FILE):
touch $(FILE)
echo $(FILE)
foo: $(FILE)
bar:DIR=barDir
bar: $(FILE)

6.11 Target-specific Variable Values:
As with automatic variables, these values are only available within the context of a target’s recipe (and in other target-specific assignments).
But not in the prerequisite list. In other words, in foo:$(X) X is not target specific.
One way to achieve the desired results is:
same_files := filename another_filename
# ${1} is the target name.
# ${2} is the directory.
define similar_prerequisites
${1} : $(addprefix ${2}/,${same_files})
$(addprefix ${2}/,${same_files}) : | ${2}
touch $$#
${2} :
mkdir $$#
endef
all : foo bar
foo bar :
#echo "$# prerequisites are $^"
$(eval $(call similar_prerequisites,foo,fooDir))
$(eval $(call similar_prerequisites,bar,barDir))
Output:
$ make
mkdir fooDir
touch fooDir/filename
touch fooDir/another_filename
foo prerequisites are fooDir/filename fooDir/another_filename
mkdir barDir
touch barDir/filename
touch barDir/another_filename
bar prerequisites are barDir/filename barDir/another_filename

I needed something similar. Does this exemple suit your needs ?
# Basic rules can be ignored
# Use % to not take much space
.PHONY: foo bar
foo bar:
#printf "%s depends on %s\n" "$#" "$^"
fooDir/ barDir/:
#printf "%s\n" "Building $#"
#mkdir "$#"
fooDir/%: fooDir
barDir/%: barDir
%/filename: %/
#printf "%s\n" "Building $#"
#touch "$#"
%/filename2: %/
#printf "%s\n" "Building $#"
#touch "$#"
%/filename3: %/
#printf "%s\n" "Building $#"
#touch "$#"
fooFile = fooDir/filename2
barFile = barDir/filename2
DIR = $(*)Dir
FILE = $(DIR)/filename3
.SECONDEXPANSION:
# Here the fun begin
# See full details on https://stackoverflow.com/a/73679964/7227940
foo bar: %: $$(*)Dir/filename $$($$(*)File) $$(FILE)
output:
$ make foo; make bar
Building fooDir/
Building fooDir/filename
Building fooDir/filename2
Building fooDir/filename3
foo depends on fooDir/filename fooDir/filename2 fooDir/filename3
Building barDir/
Building barDir/filename
Building barDir/filename2
Building barDir/filename3
bar depends on barDir/filename barDir/filename2 barDir/filename3
The tricks is in .SECONDEXPANSION: all target defined after (that's why I put it at the end) will get a second expansion. The $(*) aka the stem of matched rules will only become available in the second expansion. This allow us to build name that is linked to the target.
The pattern targets: %: deps allow to match only on listed target and not catch all target.
I prefer the version 1 or 2 because the third version you might be tempted to use in place before .SECONDEXPANSION: and thus you will get weird error messages. If you really want to use 3 then put this special target at the top. Also 3 might have wrong value (you might want MAKECMDGOALS or something like that instead of *) if used with multiple wildcard rule. Just remember if you want to use 3 then put $$ either in DIR in FILE definition or in the use site.

Related

Dynamic targets based on the dependency in Makefile

There are a couple of kind of similar issues but I could not fit any of the proposed concepts to my case.
Just to give a little bit of context: I have a set of Julia files which create plots as PDFs which are part of a make procedure to create scientific papers, something like:
plots = $(shell find $(PLOT_PATH)/*.jl | sed 's/\.jl/\.pdf/g')
$(PLOT_PATH)/%.pdf: $(PLOT_PATH)/%.jl $(JULIA_SYSIMAGE)
$(JL) --project $< -o $(PLOT_PATH)
$(DOCUMENT_FILENAME).pdf: FORCE $(plots) $(figures)
latexmk $(DOCUMENT_FILENAME).tex
In the current setup, each XYZ.jl file is creating a XYZ.pdf file and it works absolutely fine.
Now I am dealing with cases where it would be much easier to create multiple plots from single Julia files, so a script like this:
#!/usr/bin/env julia
using PGFPlotsX
...
...
pgfsave("whatever.pdf")
pgfsave("another.pdf")
pgfsave("yetanother.pdf")
so that one could do a grep pgfsave SCRIPT | awk... to figure out the targets. However, I could not figure out how to generate dynamic targets (plots) based on the contents of the dependency file (Julia script).
An MWE for my problem is the following: I have a couple of files (dependencies) which are generating a bunch of targets, which are defined inside those files (and can be access via awk/grep/sed/whatever). For now, let's say that these are simply *.txt files and each line is a target.
file: a.txt
foo
bar
baz
file: b.txt
naarf
fjoord
A very basic (non-working) manual Makefile to demonstrate the goal would be something like this (it does not work as it cannot figure out how to make foo etc. but it shows the pattern for *.txt which needs to be repeated):
file: Makefile
all_products := $(shell find *.txt | xargs cat)
final_product: $(all_products)
echo $< > $#
(foo bar baz): a.txt
touch $(shell cat $<)
(narf fjoord): b.txt
touch $(shell cat $<)
so in principle, I need something to "process" the dependency (*.txt) to create a list of the targets, like
$(shell cat $%): %.txt
echo $< > $#
but I cannot manage to get a reference to the dependency on the target side ($% does not work).
Any ideas? Maybe the whole approach is just a bad idea ;)
A combination of GNU make foreach, eval and call functions is probably what you need. With your example:
TXT := $(wildcard *.txt)
.PHONY: all
.DEFAULT_GOAL := all
define MY_MACRO
$(1)-targets := $$(shell cat $(1))
$$($(1)-targets): $(1)
echo $$< > $$#
all: $$($(1)-targets)
endef
$(foreach t,$(TXT),$(eval $(call MY_MACRO,$(t))))
(pay attention to the $$ in the macro definition, they are needed). And then:
$ make
make
echo a.txt > foo
echo a.txt > bar
echo a.txt > baz
echo b.txt > naarf
echo b.txt > fjoord
If you want the recipe to build all targets at once you'll need a recent enough GNU make version (4.3 or later) and its new rule with grouped targets (x y z&: w):
TXT := $(wildcard *.txt)
.PHONY: all
.DEFAULT_GOAL := all
define MY_MACRO
$(1)-targets := $$(shell cat $(1))
$$($(1)-targets)&: $(1)
touch $$($(1)-targets)
all: $$($(1)-targets)
endef
$(foreach t,$(TXT),$(eval $(call MY_MACRO,$(t))))
And then:
$ make
touch foo bar baz
touch naarf fjoord
Note that in this case we could also use a simpler and less GNU make-dependent solution. Just use empty dummy files as time stamps, for instance .a.txt.tag for a.txt, and a static pattern rule:
TXT := $(wildcard *.txt)
TAG := $(patsubst %,.%.tag,$(TXT))
.PHONY: all
all: $(TAG)
$(TAG): .%.tag: %
touch `cat $<` $#

Makefile: defining rules and prerequisites in recipes

I have a setup where the files I want to process with make are dependent on the output of another program. Building the program and all its prerequisites
is also a complicated task so I would like to use make for this as well. Now my problem is, that it doesn't seem that one can generate rules and prerequisites in Makefile recipes. Consider the following code:
bar:
echo target1 target2 target3 > bar
foo: bar
$(eval BAR := $(shell cat bar))
define FUN
$(1):
touch a$(1)
endef
ifdef BAR
$(foreach i,$BAR,$(eval $(call FUN,$(i))))
endif
blub: foo $(BAR)
I replaced a big set of complicated recipes that lead to the generation of the list of files I want to have in the end by the bar recipe. In reality, producing the content of bar is very complicated and should be done by a set of Makefile recipes and cannot just be done by (as the above suggests):
BAR:=$(shell echo target1 target2 target3)
I would like to put the foreach loop into the recipe for foo but that fails with prerequisites cannot be defined in recipes which makes sense and is also explained in function define in makefile
But it seems that when I do make blub that at the time when foo eval's BAR to a different value, the prerequisites for blub are not re-evaluated.
So I think ultimately I'm looking for two things:
how do I generate recipes dynamically at runtime, based on (and dependent on) what another recipe (bar in this case) outputs?
how do I update the prerequisites of a target (blub in this case) dynamically at runtime, based on (and dependent on) what another recipe (bar in this case) outputs?
Thank you!
EDIT: SOLUTION
With the help of #user657267 the following seems to solve my problem:
.PHONY: all
all: blub
-include bar.make
.PHONY: blub
blub: $(BAR)
echo $^
bar.make: Makefile
printf 'BAR=target1 target2 target3\n' > $#
printf 'target1 target2 target3:\n' >>$#
printf '\ttouch $$#' >> $#
.PHONY: clean
clean:
rm -f target1 target2 target3 bar.make
Sounds like you should be using make's self-remaking features
-include bar.make
blub: $(BAR)
#echo $^
bar.make:
#echo BAR := target1 target2 target3 > $#
#echo target1 target2 target3: ; touch $$# >> $#
Obviously the recipes for bar.make are contrived, in the real world they'd probably invoke some kind of script that outputs a valid makefile.

How do I force a target to be rebuilt if a variable is set?

Assume I have a build-target foo:
foo:foo.c
$(CC) $(CFLAGS) $(ARGS) -c foo.c -o foo
Now, ARGS is something that I pass on the command line:
$ make ARGS:=-DX=1 foo
So, I need to bypass make's cleverness, because the foo target does not only depend on which files have changed, but also on the value of ARGS.
Is there something in make to do this? My hack (see answer) doesn't seem to be the most elegant but it works. Anything better?
Here is a general solution to your specific problem.
You want to be able to depend on a variable as a prerequisite. That is, you can make it a prerequisite to any target in your makefile, and when the value of the variable changes, you rebuild those targets.
Here is a function that does that, you use this function to declare a variable to be dependable, and then you can use it as a prerequisite.
Note that if the variable is not used on the command line, it will still mean that variable still has a value, namely, the empty string.
define DEPENDABLE_VAR
.PHONY: phony
$1: phony
#if [[ `cat $1 2>&1` != '$($1)' ]]; then \
echo -n $($1) > $1 ; \
fi
endef
#declare ARGS to be dependable
$(eval $(call DEPENDABLE_VAR,ARGS))
foo:foo.c ARGS
$(CC) $(CFLAGS) $(ARGS) -c foo.c -o foo
In fact, we could omit the need for "declaration", and just write a similar function that will make all variables dependable by default. But I don't like that. I prefer that the users that modify makefiles I write, declare their intentions explicitly. It is good for them :)
My solution was to create a dummy phony target:
.PHONY:dummy
dummy:
#:
and have foo depend on dummy if ARGS is nonempty:
foo:foo.c $(patsubst %,dummy,$(ARGS))
Note on Mark's excellent answer
The bare necessities of Mark's answer are actually very simple. It really boils down to just:
.PHONY: phony
ARGS: phony
#if [[ `cat ARGS 2>&1` != '$(ARGS)' ]]; then echo -n $(ARGS) >ARGS; fi
The rest of his code is just to let you reproduce the recipe for other variable names without repeating yourself. Useful in practice, but the above version will help you see what's going on more easily.
In fact, my answer can even be made general (like Mark's) for any variable name, but in a less complicated way as follows:
.PHONY: phony
.ARG~%: phony
#if [[ `cat .ARG~$* 2>&1` != '$($*)' ]]; then echo -n $($*) >.ARG~$*; fi
Then you simply add .ARG~MYVARNAME as a dependency to any target to make that target depend on variable MYVARNAME.
Note that the dot in front of .ARG~% simply causes it to create a dependency-tracking file that is 'hidden' in linux.
So in your case, you would do:
foo: foo.c .ARG~ARGS
$(CC) $(CFLAGS) $(ARGS) -c foo.c -o foo
I don't understand how the other solutions are supposed to work. If the ARGS target is .PHONY or depends on a .PHONY, then it will always be run, right?
Here is my solution using the $(file) function in newer versions of gmake:
.PHONY: FORCE
define DEPENDABLE_VAR
$(1):
echo -n $($(1)) > $(1)
ifneq ("$(file <$(1))","$($(1))")
$(1): FORCE
endif
endef
#declare ARGS to be dependable
$(eval $(call DEPENDABLE_VAR,ARGS))
foo: foo.c ARGS
touch foo
And the result:
~/stuff/makevars> make foo ARGS=1
echo -n 1 > ARGS
touch foo
~/stuff/makevars> make foo ARGS=1
make: 'foo' is up to date.
~/stuff/makevars> make foo ARGS=2
echo -n 2 > ARGS
touch foo
~/stuff/makevars> make foo ARGS=2
make: 'foo' is up to date.

Makefile with multiple rules sharing same recipe with patternrules?

I want to remove the duplication of recipe in a makefile like the following
SHELL := /bin/bash
a_% : a1_% a2_%
cat $^ > $#
b_% : b1_% b2_% %_b3
cat $^ > $#
However the following does not work. I guess the trick in this SO question does not work with pattern rules.
SHELL := /bin/bash
a_% : a1_% a2_%
b_% : b1_% b2_% %_b3
a_% b_%:
cat $^ > $#
Any suggestions ? ( In my original makefile, recipe duplication is occurring in 4 targets, and each of those take 3 substitutions, so I can't unroll the targets)
--EDIT--
I realized that one way to solve this was the following.
CMD1 = cat $^ > $#
a_% : a1_% a2_%
$(CMD1)
b_% : b1_% b2_% %_b3
$(CMD1)
I believe this does what you asked for:
SHELL := /bin/bash
define STUFF
$(1)_%: $(1)1_% $(1)2_% $(2)
cat $$^ > $$#
endef
$(eval $(call STUFF,a))
$(eval $(call STUFF,b,%_b3))
How this works:
The general form of the rule is defined as STUFF. (You'd obviously want a better name in your own Makefile.) Note the doubling of dollar signs in $$^ and $$#. This protects them from evaluation when $(call ...) is executed. $(1) and $(2) will be replaced by $(call ...) with positional arguments.
$(call STUFF,a) "calls" STUFF with $(1) set to the string a and $(2) set to the empty string. The return value is:
a_%: a1_% a2_%
cat $^ > $#
Note how one $ was stripped from the remaining variables.
$(eval ...) evaluates the return value obtained in the previous step as if that string had been put in the Makefile. So it creates the rule.
Steps 2 and 3 also happen for the b files. It is similar to what happens for the a files except that this time $(2) is set to the string %_b3.
This is essentially the method I've used in the past to avoid duplication of rules for cases where the rules were rather complex. For the specific case you show in your question, I'd use the shared command variable you mention in your question.

Makefile define: recursive expansion question

In a makefile, I define a variable using the define directive. This variable will hold a configurable list of commands that I want to execute.
I would like this variable to get a list of files (.foo files, for example). These files are created during the makefile execution. For example makefile:
MY_VAR = $(wildcard *.foo)
define MY_VAR2
echo $(1) $(MY_VAR)
endef
foo: create_files
$(call MY_VAR2, ls)
rm -f *.foo
create_files:
touch foo.foo
touch bar.foo
I do not get the desired results. It appears that MY_VAR2 is evaluated upon declaration.
Is there a way to get the desired behavior?
edit:
The $(shell) command, as sateesh correctly pointed out, works for the example above. However, it does not work for the example below. The main difference in this example is that the new files are created inside MY_VAR2.
MY_VAR = $(wildcard *.foo)
TEST_VAR = $(shell ls *.foo)
define MY_VAR2
#touch foo.foo
#touch bar.foo
#echo "MY_VAR" $(1) $(MY_VAR)
#echo "TEST_VAR" $(1) $(TEST_VAR)
endef
foo:
$(call MY_VAR2, ls)
#rm -f *.foo
I can solve the above by adding rules. Is there a simpler method?
It looks to me like you are abusing make, trying to write a shell script in make.
If you write a shell script, write a shell script. You execute your commands in sequence, and you are able to know what files are present when executing each line.
touch foo.foo
touch bar.foo
your-command `ls *.foo`
On the other side, if you want to make usage of make, then have rules and dependencies, you won't even have to use define if you go the make way of thinking.
foo: create_files
your-command $(wildcard *.foo)
rm -f *.foo
create_files:
touch foo.foo
touch bar.foo
I'm presuming here that you are using here GNU Make. I think you can get the desired
result using the "shell" built-in function provided with GNU Make.
Below is the make file snippet that demonstrates how you can get the result:
MY_VAR = $(wildcard *.foo)
TEST_VAR = $(shell ls *.foo)
define MY_VAR2
#echo "MY_VAR" $(1) $(MY_VAR)
#echo "TEST_VAR" $(1) $(TEST_VAR)
endef
foo: create_files
$(call MY_VAR2, ls)
#rm -f *.foo
create_files:
#touch foo.foo
#touch bar.foo
The result of running the above make file for the target "foo" is:
MY_VAR ls
TEST_VAR ls bar.foo foo.foo
So as demonstrated usage of shell function gets the desired result.
From the GNU make documentation:
The shell
function performs the same function that backquotes (‘‘’) perform in most
shells: it does command expansion. This means that it takes as an argument a shell
command and evaluates to the output of the command. The only processing make does on
the result is to convert each newline (or carriage-return / newline pair) to a single space.
...
The commands run by calls to the shell function are run when the function calls are expanded.
...
files := $(shell echo .c)
sets files to the expansion of ‘.c’. Unless make is using a very strange shell, this has the
same result as ‘$(wildcard *.c)’ (as long as at least one ‘.c’ file exists).
So I think using a recursively expanded variable (= form) along with shell function you
can get the desired result.
4.4.3 The Function wildcard
Wildcard expansion happens automatically in rules. But wildcard expansion does not normally take place when a variable is set, or inside the arguments of a function. If you want to do wildcard expansion in such places, you need to use the wildcard function
You may work your away around the eval function.
These created files should be dependencies of another rule so that the dependency graph is correct.
References:
4.4.3 The Function wildcard
8.8 The eval function
Could you try the filter command? You can find an example here.
How about using include?
The point is that the make needs two pass of execution.
First, you need to let make create what it wants,
Then you'll let make to parse the generated rule in the second pass after
the execution of the first rule.
define MY_VAR2
echo $(1) $(MY_VAR)
$(1) $(MY_VAR)
endef
-include .listoffiles
foo: create_files
$(call MY_VAR2, ls -al)
rm -f *.foo .listoffiles
create_files: .listoffiles
.listoffiles:
touch foo.foo
touch bar.foo
#echo "MY_VAR = \$$(wildcard *.foo)" > $#
It just does what you want, I suppose.
Two pass evaluation doesn't mean you have to type make twice.
It will be done by make automagically.
% make -f test.mk
touch foo.foo
touch bar.foo
echo ls -al bar.foo foo.foo
ls -al bar.foo foo.foo
ls -al bar.foo foo.foo
-rw-rw-r-- 1 yo-hei clearusers 0 Jan 8 08:44 bar.foo
-rw-rw-r-- 1 yo-hei clearusers 0 Jan 8 08:44 foo.foo
rm -f *.foo .listoffiles
What about doing your expansion in the shell instead of in make?
MY_VAR = $$(ls *.foo)
define MY_VAR2
#echo $(1) $(MY_VAR)
endef
foo: create_files
$(call MY_VAR2, ls)
#rm -f *.foo
create_files:
#touch foo.foo
#touch bar.foo
The double-$ will allow the shell to expand the file search part. I can't help but agree with the other posts about there being a more elegant way to do this, but this is an option for you.
Works for your second example too:
MY_VAR = $(wildcard *.foo)
TEST_VAR = $$(ls *.foo)
define MY_VAR2
#touch foo.foo
#touch bar.foo
#echo "MY_VAR" $(1) $(MY_VAR)
#echo "TEST_VAR" $(1) $(TEST_VAR)
endef
foo:
$(call MY_VAR2, ls)
#rm -f *.foo

Resources