I want to remove the duplication of recipe in a makefile like the following
SHELL := /bin/bash
a_% : a1_% a2_%
cat $^ > $#
b_% : b1_% b2_% %_b3
cat $^ > $#
However the following does not work. I guess the trick in this SO question does not work with pattern rules.
SHELL := /bin/bash
a_% : a1_% a2_%
b_% : b1_% b2_% %_b3
a_% b_%:
cat $^ > $#
Any suggestions ? ( In my original makefile, recipe duplication is occurring in 4 targets, and each of those take 3 substitutions, so I can't unroll the targets)
--EDIT--
I realized that one way to solve this was the following.
CMD1 = cat $^ > $#
a_% : a1_% a2_%
$(CMD1)
b_% : b1_% b2_% %_b3
$(CMD1)
I believe this does what you asked for:
SHELL := /bin/bash
define STUFF
$(1)_%: $(1)1_% $(1)2_% $(2)
cat $$^ > $$#
endef
$(eval $(call STUFF,a))
$(eval $(call STUFF,b,%_b3))
How this works:
The general form of the rule is defined as STUFF. (You'd obviously want a better name in your own Makefile.) Note the doubling of dollar signs in $$^ and $$#. This protects them from evaluation when $(call ...) is executed. $(1) and $(2) will be replaced by $(call ...) with positional arguments.
$(call STUFF,a) "calls" STUFF with $(1) set to the string a and $(2) set to the empty string. The return value is:
a_%: a1_% a2_%
cat $^ > $#
Note how one $ was stripped from the remaining variables.
$(eval ...) evaluates the return value obtained in the previous step as if that string had been put in the Makefile. So it creates the rule.
Steps 2 and 3 also happen for the b files. It is similar to what happens for the a files except that this time $(2) is set to the string %_b3.
This is essentially the method I've used in the past to avoid duplication of rules for cases where the rules were rather complex. For the specific case you show in your question, I'd use the shared command variable you mention in your question.
Related
I'm trying to get Make to build some data analysis, where there are file lists controlled by one overall parameter.
To write it explicitly would be something like:
A_EXTS = a b c d e
B_EXTS = f g h i j
C_EXTS = k l m n o
A.dat : $(foreach EXT, ${A_EXTS}, prefix1_${EXT}.dat prefix2_${EXT}.dat)
python analyse.py $^ > $#
B.dat : $(foreach EXT, ${B_EXTS}, prefix1_${EXT}.dat prefix2_${EXT}.dat)
python analyse.py $^ > $#
C.dat : $(foreach EXT, ${C_EXTS}, prefix1_${EXT}.dat prefix2_${EXT}.dat)
python analyse.py $^ > $#
Obviously the only difference between the three rules is the A vs B vs C.
I thought to try something like
%.dat : $(foreach EXT, ${%_EXTS}, prefix1_${EXT}.dat prefix2_${EXT}.dat)
python analyse.py $^ > $#
…but that doesn't work; e.g. make B.dat runs the rule for B.dat but ignores the dependencies; $^ is set to the empty string.
The files starting prefix2_ are generated by another recipe, so I can't just specify them within the recipe, they need to be marked as dependencies here.
Is this possible to express these dependencies without repeating the same rule?
Well, you can't do it quite like you want to here, but it's not related to looking up variable names: it's because of expansion order.
Variables in targets and prerequisites are expanded when the makefile is parsed, but make doesn't expand the patterns in pattern rules until much later. That means when make expands the ${%_EXTS} variable as it parses the makefile, it has no idea what the value of % will be later when it's actually trying to build things.
You can use secondary expansion to delay expansion of variables until make's second pass where it is actually finding target names. I pulled the logic out into a separate variable and used call to make it a bit more readable:
.SECONDEXPANSION:
EXPANDDEPS = $(foreach EXT,${$1_EXTS},prefix1_${EXT}.dat prefix2_${EXT}.dat)
%.dat : $$(call EXPANDDEPS,$$*)
python analyse.py $^ > $#
I have a Makefile with user-specified input files in the variable INPUT_FILES.
For each input file, I need to create an input file prime.
Some notes:
Each input file can have an arbitrary file location
It is reasonable to assume there aren't duplicate filenames
Each output file needs to go into $(OUTPUT_DIR)
My basic strategy has been to generate the set of targets based INPUT_FILES and then try to determine which input file is the actual dependency of the target.
A few variations I've tried:
# Create a list of targets
OUTPUT_FILES = $(foreach file,$(notdir $(INPUT_FILES)),$(OUTPUT_DIR)/$(file))
# This doesn't work, because all input files are dependencies of each output file
$(OUTPUT_FILES): $(INPUT FILES)
program --input $^ --output $#
# This doesn't work because $# hasn't been resolved yet
$(OUTPUT_FILES): $(filter,$(notdir $#),$(INPUT FILES))
program --input $^ --output $#
# This doesn't work, I think because $# is evaluated too late
.SECONDEXPANSION:
$(OUTPUT_FILES): $(filter,$(notdir $$#),$(INPUT FILES))
program --input $^ --output $#
# This doesn't work either
.SECONDEXPANSION:
$(OUTPUT_FILES): $$(filter,$(notdir $#),$(INPUT FILES))
program --input $^ --output $#
I've looked into static pattern rules as well, but I'm not sure if it can help with what I need.
In your case .SECONDEXPANSION: works because you can use make functions (filter) to compute the prerequisite of each output file. In other circumstances it could be impossible. But there is another GNU make feature that can be used in cases like yours: if you use GNU make you can programmatically instantiate make statements using foreach-eval-call. Just remember that the macro that is used as the statements pattern gets expanded twice, reason why you must double some $ signs (more on this later):
OUTPUT_DIR := dir
OUTPUT_FILES := $(addprefix $(OUTPUT_DIR)/,$(notdir $(INPUT_FILES)))
.PHONY: all
all: $(OUTPUT_FILES)
# The macro used as statements pattern where $(1) is the input file
define MY_RULE
$(1)-output-file := $(OUTPUT_DIR)/$$(notdir $(1))
$$($(1)-output-file): $(1)
#echo program --input $$^ --output $$#
endef
$(foreach i,$(INPUT_FILES),$(eval $(call MY_RULE,$(i))))
Demo:
$ mkdir -p a/a b
$ touch a/a/a b/b c
$ make INPUT_FILES="a/a/a b/b c"
program --input a/a/a --output dir/a
program --input b/b --output dir/b
program --input c --output dir/c
Explanation:
When make parses the Makefile it expands $(foreach ...): it iterates over all words of $(INPUT_FILES), for each it assigns the word to variable i and expands $(eval $(call MY_RULE,$(i))) in this context. So for word foo/bar/baz it expands $(eval $(call MY_RULE,$(i))) with i = foo/bar/baz.
$(eval PARAMETER) expands PARAMETER and instantiates the result as new make statements. So, for foo/bar/baz, make expands $(call MY_RULE,$(i)) with i = foo/bar/baz and considers the result as regular make statements. The expansion of $(eval ...) has no other effect, the result is the empty string. This is why in our case $(foreach ...) expands as the empty string. But it does something: create new make statements dynamically for each input file.
$(call NAME,PARAMETER) expands PARAMETER, assigns it to temporary variable 1 and expands the value of make variable NAME in this context. So, $(call MY_RULE,$(i)) with i = foo/bar/baz expands as the expanded value of variable MY_RULE with $(1) = foo/bar/baz:
foo/bar/baz-output-file := dir/$(notdir foo/bar/baz)
$(foo/bar/baz-output-file): foo/bar/baz
#echo program --input $^ --output $#
which is what is instantiated by eval as new make statements. Note that we had a first expansion here and the $$ became $. Note also that call can have more parameters: $(call NAME,P1,P2) will do the same with $(1) = P1 and $(2) = P2.
When make parses these new statements (as any other statements) it expands them (second expansion) and finally adds the following to its list of variables:
foo/bar/baz-output-file := dir/baz
and the following to its list of rules:
dir/baz: foo/bar/baz
#echo program --input $^ --output $#
This may look complicated but it is not if you remember that the make statements added by eval are expanded twice. First when $(eval ...) is parsed and expanded by make, and a second time when make parses and expands the added statements. This is why you frequently need to escape the first of these two expansions in your macro definition by using $$ instead of $.
And it is so powerful that it is good to know.
When asking for help please provide some kind of actual example names so we can understand more clearly what you have. It also helps us use terminology which is not confusing.
You really want to use $< in your recipes, not $^, I expect.
IF your "input files" are truly input-only (that is, they are not themselves generated by other make rules) then you can easily solve this problem with VPATH.
Just use this:
VPATH := $(sort $(dir $(INPUT_FILES)))
$(OUTPUT_DIR)/% : %
program --input $< --output $#
I finally found a permutation that works - I think the problem was forgetting that filter requires a % for matching patterns. The rule is:
.SECONDEXPANSION:
$(OUTPUT_FILES): $$(filter %$$(#F),$(INPUT_FILES))
program --input $^ --output $#
I also realized I can use #F (equivalent to $$(notdir $$#)) for cleaner syntax.
The rule gets the target's filename on its second expansion ($$(#F)) and then gets the input file (with path) that corresponds to it on second expansion ($$(filter %$$(#F),$(INPUT_FILES))).
Of course, the rule only works if filenames are unique. If someone has a cleaner solution, feel free to post.
I have 2 types of files. Scripts and datasets. I want to write a makefile to run each script with each dataset.
My idea was to create a target for each output, where the target has a name containing the script and the model name. These targets would match a "target-pattern"-rule (of type: some_folder/%.eval). Now the "target-pattern"-rule would need to figure out from its name which files are needed. But this seems to be a hard job.
Is there a better, more elegant way?
Example:
Files: ScriptA, ScriptB, InputA, InputB
Targets/Outputfiles: InputA_ScriptA, InputA_ScriptB, InputB_ScriptA, InputB_ScriptB
# generate all combinations
RT_HW = $(foreach script,$(RT_SCRIPT_HW),$(foreach input, $(RT_INPUTS), $(input)_$(script)))
$(SANDBOX)%.eval: <requires Script X and input X>
You've run into one of the major shortcomings of Make: it isn't very good with wildcards.
You can get the effect you want by generating rules with eval, either by iterating over both variables:
define template
$(1)_$(2).eval: $(1) $(2)
#echo target is $$#
#echo running $(2) on $(1)
endef
$(foreach script,$(RT_SCRIPT_HW),$(foreach input, $(RT_INPUTS), $(eval $(call template,$(input),$(script)))))
or by generating pattern rules, iterating over only one, such as the input:
define template
$(1)_%.eval: $(1) %
#echo target is $$#
#echo running $$* on $(1)
endef
$(foreach input, $(RT_INPUTS), $(eval $(call template,$(input))))
Assume I have a build-target foo:
foo:foo.c
$(CC) $(CFLAGS) $(ARGS) -c foo.c -o foo
Now, ARGS is something that I pass on the command line:
$ make ARGS:=-DX=1 foo
So, I need to bypass make's cleverness, because the foo target does not only depend on which files have changed, but also on the value of ARGS.
Is there something in make to do this? My hack (see answer) doesn't seem to be the most elegant but it works. Anything better?
Here is a general solution to your specific problem.
You want to be able to depend on a variable as a prerequisite. That is, you can make it a prerequisite to any target in your makefile, and when the value of the variable changes, you rebuild those targets.
Here is a function that does that, you use this function to declare a variable to be dependable, and then you can use it as a prerequisite.
Note that if the variable is not used on the command line, it will still mean that variable still has a value, namely, the empty string.
define DEPENDABLE_VAR
.PHONY: phony
$1: phony
#if [[ `cat $1 2>&1` != '$($1)' ]]; then \
echo -n $($1) > $1 ; \
fi
endef
#declare ARGS to be dependable
$(eval $(call DEPENDABLE_VAR,ARGS))
foo:foo.c ARGS
$(CC) $(CFLAGS) $(ARGS) -c foo.c -o foo
In fact, we could omit the need for "declaration", and just write a similar function that will make all variables dependable by default. But I don't like that. I prefer that the users that modify makefiles I write, declare their intentions explicitly. It is good for them :)
My solution was to create a dummy phony target:
.PHONY:dummy
dummy:
#:
and have foo depend on dummy if ARGS is nonempty:
foo:foo.c $(patsubst %,dummy,$(ARGS))
Note on Mark's excellent answer
The bare necessities of Mark's answer are actually very simple. It really boils down to just:
.PHONY: phony
ARGS: phony
#if [[ `cat ARGS 2>&1` != '$(ARGS)' ]]; then echo -n $(ARGS) >ARGS; fi
The rest of his code is just to let you reproduce the recipe for other variable names without repeating yourself. Useful in practice, but the above version will help you see what's going on more easily.
In fact, my answer can even be made general (like Mark's) for any variable name, but in a less complicated way as follows:
.PHONY: phony
.ARG~%: phony
#if [[ `cat .ARG~$* 2>&1` != '$($*)' ]]; then echo -n $($*) >.ARG~$*; fi
Then you simply add .ARG~MYVARNAME as a dependency to any target to make that target depend on variable MYVARNAME.
Note that the dot in front of .ARG~% simply causes it to create a dependency-tracking file that is 'hidden' in linux.
So in your case, you would do:
foo: foo.c .ARG~ARGS
$(CC) $(CFLAGS) $(ARGS) -c foo.c -o foo
I don't understand how the other solutions are supposed to work. If the ARGS target is .PHONY or depends on a .PHONY, then it will always be run, right?
Here is my solution using the $(file) function in newer versions of gmake:
.PHONY: FORCE
define DEPENDABLE_VAR
$(1):
echo -n $($(1)) > $(1)
ifneq ("$(file <$(1))","$($(1))")
$(1): FORCE
endif
endef
#declare ARGS to be dependable
$(eval $(call DEPENDABLE_VAR,ARGS))
foo: foo.c ARGS
touch foo
And the result:
~/stuff/makevars> make foo ARGS=1
echo -n 1 > ARGS
touch foo
~/stuff/makevars> make foo ARGS=1
make: 'foo' is up to date.
~/stuff/makevars> make foo ARGS=2
echo -n 2 > ARGS
touch foo
~/stuff/makevars> make foo ARGS=2
make: 'foo' is up to date.
Borne out of morbid curiosity and seeing CMake's ExternalProject, I've tried to hack up a cute little attempt at an automatic git-dependency manager for a C++ project, however I can't quite make Make dance the way I want it to.
# shortname, git address, configure, make (install), make clean
DEPENDENCIES:=\
catch,https://github.com/philsquared/Catch.git,true,true,true
, := ,
hit_subtree = git subtree $1 --prefix deps/$2 $2 master --squash
define get_or_update
$(if $(wildcard deps/$1/*),
git fetch $1 master && $(call hit_subtree,pull,$1),
$(if $(shell git ls-remote catch),
true,
git remote add -f $1 $2) && $(call hit_subtree,add,$1)
)
endef
update_cxx_flags = $$(if $$(wildcard deps/$1/include/*),$$(eval CXXFLAGS += -Ideps/$1/include),)
update_ld_flags = $$(if $$(wildcard deps/$1/lib/*),$$(eval LDFLAGS += -Ideps/$1/lib),)
define update_flags
$(eval $(call update_cxx_flags,$1))
export CXXFLAGS
$(eval $(call update_ld_flags,$1))
export LDFLAGS
endef
build_project = cd deps/$1 && $4 && $2 && $3
define git_dependency
$(call get_or_update,$1,$2)
$(call build_project,$1,$3,$4,$5)
$(call update_flags,$1)
endef
caller = $(call git_dependency,$(word 1,$1),$(word 2,$1),$(word 3,$1),$(word 4,$1),$(word 5,$1))
git_dependencies:
$(foreach dep,$(DEPENDENCIES),$(call caller,$(subst $(,), ,$(dep))))
#echo ${CXXFLAGS}
#echo ${LDFLAGS}
The problem lies in the update_flags function: specifically, update_flags tries to modify CXXFLAGS and LDFLAGS to account for new include/lib dirs however it seems that $(eval ...) isn't doing what I want it do. On the first run (i.e. when the directory is first being cloned) the $(wildcard ...) function sees no sub-directories of deps/$1 however if I invoke make a second time it then works fine. To me, this suggests that $(eval ...) isn't actually evaluating update_cxx_flags and instead the function is being non-lazily evaluated. What am I doing wrong?
Here is your SSCCE:
all:
touch foobar
echo $(wildcard foobar)
This 'does not work', as you observe, first time, but second time, it works. Why? Because, GNU Make first evaluates the whole recipe, before executing any lines of it. Then, after the recipe is evaluated (translated into the shell language), only then it is executed.
OK, you wanted to do it with $$, it still won't work, the double $ won't make it defer to the recipe execution, it will just evaluate twice during the processing of eval:
all:
touch foobar
$(eval $$(info $$(wildcard foobar)))
On the chat, I told you what is happening, but you are assuming some "caching".
You are a very knowledgeable person in certain areas, but you must remember when you learn something new, to start from the beginning and follow simple examples and manual. I am giving you simple examples, analyze them with the help of the manual and do not spin your own theories.
Mark's answer led me to google to work out why $(eval $$(wildcard foobar)) wouldn't behave as intended -- after all, at the very least GNU make promises to evaluate the argument as though it was 'typed' into your makefile.
It turns out that $(wildcard ...) is a little too smart for its own good: it caches directories and only updates the cache if a file is generated via a makefile rule. In this instance, the file is generated by dropping to shell and using git which violates the assumption that files are generated via makefile rules. Thus, the check in update_cxx_flags is incorrect (as well as update_ld_flags). Instead, it should be modified as so:
update_cxx_flags = $$(if `ls deps/$1/include/* 2>/dev/null`,$$(eval CXXFLAGS += -Ideps/$1/include),)
where the /dev/null clobber is so that an error message doesn't appear when the file doesn't exist. This makes the makefile behave as expected, which is what I wanted!