I'm trying to write a Makefile that uses the same recipe for several variables. The variables are grouped into several lots which need to be built separately but with the same method.
In the example below, I want to be able to build output1.txt, output2.txt etc. using the same recipe. The catch is that each of these relies on the corresponding variable $(target1), $(target2) etc. I've tried to use the % pattern to expand these variables but that didn't work because % is expanded last.
I have found that I can modify variable calls using recursive variables, eg. $($(subst 0,$(N),target0)). However, this cannot be done unless the number is already known.
Is there a simple way to do this?
target1:=a_5_2 a_3_5 a_6_2 a_0_0 a_9_1
target2:=a_2_2 a_1_3 a_5_9 a_2_7 a_3_6
target3:=a_2_3 a_6_5 a_9_0 a_3_4 a_3_9
target4:=a_7_8 a_8_2 a_4_8 a_7_1 a_0_7
output%.txt: $($(subst 0,%,target0))
cat $^ > $# # do some actual stuff here
You can do this with secondary expansion:
.SECONDEXPANSION:
output%.txt: $$(target$$*)
...
Of course, you could also just change the variables to be prerequisite settings instead; it's about the same amount of typing:
output1.txt : a_5_2 a_3_5 a_6_2 a_0_0 a_9_1
output2.txt : a_2_2 a_1_3 a_5_9 a_2_7 a_3_6
output3.txt : a_2_3 a_6_5 a_9_0 a_3_4 a_3_9
output4.txt : a_7_8 a_8_2 a_4_8 a_7_1 a_0_7
output%.txt:
cat $^ > $# # do some actual stuff here
Related
I have a complicated set of rules I need to write to generate a rather large number of "parameterised" output files and thought that, rather than expand them all out by hand, I could repeatedly "include" a template file with sets of rules and use (GNU)make's facility for allowing "simply expanded" variables to avoid the pain.
(In the past I've always been using the "recursively expanded" variable approach, so this is new to me)
As a trivial example of what I thought would work, I tried putting the following in a Makefile
Targ:=A
Param1:=Pa
Param2:=Qa
$(Targ):
#echo expect A, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
Targ:=B
Param1:=Pb
Param2:=Qb
$(Targ):
#echo expect B, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
Targ:=C
Param1:=Pc
Param2:=Qc
$(Targ):
#echo expect C, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
The eventual plan was to replace the rules with an include file containing dozens of different rules, each referencing the various "parameter" variables.
However, what I get is...
prompt> make A
expect A, get C, Target is A. Params are Pc and Qc
prompt> make B
expect B, get C, Target is B. Params are Pc and Qc
Essentially, unlike each rule's target, which is picking up the intended definition, the $(Targ), $(Param1), and $(Param2) in each rule's command is instead being run with the final definition.
Does anyone know how to prevent this, i.e. how do you force the command to use the definition at the time it is encountered in the Makefile?
Simple vs recursive expansion makes no difference here; regardless of which you use you'll see the same behavior. A GNU make variable is global and obviously can have only one value.
You have to understand when variables are expanded. The documentation provides a detailed description of this. Targets and prerequisites are expanded when the makefile is read in, so the value of Targ as the makefile is being parsed is used.
Recipes are expanded when the recipe is to be invoked, which is not until after all makefiles are parsed and make starts to build targets. At that time of course the variable Targ has its last set value.
Without knowing what your makefile really does it's hard to suggest an alternative. One option is to use target-specific variables:
Targ := A
$(Targ): LocalTarg := $(Targ)
$(Targ):
#echo expect A, get $(LocalTarg), Target is $#
Another option is to use constructed variable names:
Targ := A
Targ_$(Targ) := $(Targ)
$(Targ):
#echo expect A, get $(Targ_$#), Target is $#
Apologies for answering my own question, but I now realised it is possible to solve the issue I was having by running make recursively.
E.g. if the parameter variables for the rules are Targ, Param1 and Param2 then
#Set up "default" values for the parameters (As #madscientist points out,
#these will safely be overridden by the defs on the #(make) commands below
Targ=XXXXXXXXXX
Param=XXXXXXXXXX
Param2=XXXXXXXXXX
Recursing=
#
# define N (==3) templated rule(s)
#
$(Targ)%a:
#echo Run Combo_a $(Targ) $(Param1) $(Param2) $#
$(Targ)%b:
#echo Run Combo_b $(Targ) $(Param2) $(Param1) reversed $#
$(Targ)%c:
#echo Run Combo_c $(Param1) $(Targ) $(Param2) mixed again $#
#
#Enumerate "M" (==2) sets of parameters,
# (Except if we are already recursing because unrecognised targets may cause
# it to descend forever)
#
ifneq ($(Recursing), Yes)
Set1%:
#$(MAKE) Targ=Set1 Param1=foo Param2=bar Recursing=Yes $#
Set2%:
#$(MAKE) Targ=Set2 Param1=ray Param2=tracing Recursing=Yes $#
endif
This then allows N*M different combos for N+M typing cost.
eg. (removing messages from make re recursion)
>make Set1.a
Run Combo_a Set1 foo bar Set1.a
>make Set2.c
Run Combo_c ray Set2 tracing mixed again Set2.c
I have a file "ORIGINAL", which, if updated, I would like to copy, modify, and distribute to a few places on the drive. The modification is made by a little bash script which takes one parameter, a parameter unique for each spawned remote file.
In my Makefile, I can do this with a separate rule/recipe for each parameter, like so:
parameters = AWK BAT CAT DOG
$(DEST_FILE_AWK) : $(ORIGINAL)
./copyAndModify "AWK" ## Creates $(ORIGINAL)_AWK, substed copy of ORIGINAL
mv - f $(ORIGINAL)_AWK $(DEST_FILE_AWK)
$(DEST_FILE_BAT) : $(ORIGINAL)
./copyAndModify "BAT" ## Creates $(ORIGINAL)_BAT, substed copy of ORIGINAL
mv - f $(ORIGINAL)_BAT $(DEST_FILE_BAT)
The dereferenced values of DEST_FILE_AWK and DEST_FILE_BAT have nothing to do with each other, but other than that, the two recipes above are exactly the same with the only difference the parameter, so I can't help but want to merge them into one super rule/recipe with a multiple target rule line.
But I just can't make it happen. I've tried all kinds of foreach() and other stuff in the target section of the rule, but the problem is that no matter what, I can't get the value of the parameter into the recipe part.
Is there a way?
With the information provided here the best you can do (assuming you're using GNU make) is an eval/call combination. As anishsane suggests, depending on the value of the DEST_FILE_* variables it might be possible to do something simpler.
But this should work:
define COPY_TO_DEST
$$(DEST_FILE_$1) : $$(ORIGINAL)
./copyAndModify "$1"
mv - f $$(ORIGINAL)_$1 $$#
endef
parameters = AWK BAT CAT DOG
$(foreach P,$(parameters),$(eval $(call COPY_TO_DEST,$P)))
It can be done without $(eval), at least in gnu make :)
Start with one recipe that specifies all of the targets, i.e. the list of targets is on the left side of the recipe. Let's assume we have a variable that holds the names of all these targets.
Now observe that both functions and variables will be evaluated separately for a given recipe as it gets expanded for each of the targets. Recall that, say $# is just a variable, and will be substituted separately for each target. Function calls behave the same.
Provide a list of types, and a list of type:target pairs. I presume that there's no need to put the targets into separate variables like you did ($(DEST_FILE_AWK) etc).
The TARGET_FOR_TYPE function takes the pairs and the types and generates a list of destination files.
The TYPE variable is assigned once for each target, computed by the TYPE_FOR_TARGET function. That way the repeated function call doesn't pollute the recipe :)
Note that the DESTINATIONS list contains plain filenames, without any further indirection.
types = AWK BAT
ORIGINAL = an_original
DESTINATIONS = \
AWK:dest_for_awk \
BAT:dest_for_bat
TARGET_FOR_TYPE = $(patsubst $(1):%,%,$(filter $(1):%,$(DESTINATIONS)))
TYPE_FOR_TARGET = $(patsubst %:$(1),%,$(filter %:$(1),$(DESTINATIONS)))
# Usage example for the functions above:
$(info type: $(call TYPE_FOR_TARGET,dest_for_awk))
$(info target: $(call TARGET_FOR_TYPE,AWK))
$(info $())
DEST_FILES = $(foreach type,$(types),$(call TARGET_FOR_TYPE,$(type)))
all: $(DEST_FILES)
$(DEST_FILES) : TYPE=$(call TYPE_FOR_TARGET,$#)
$(DEST_FILES) : $(ORIGINAL)
#echo ./copyAndModify $(TYPE)
#echo mv - f $(ORIGINAL)_$(TYPE) $#
I would like to make a complete text document from several sources (since one of the file source change, I want the doc to change).
I have to pass it through a translator I develop. I would like to pass the language as argument, to make it cleaner.
Yesterday, late at night, I dreamed of a makefile like this...
#makefile
# ...
my_complete_doc.%.html: my_trans_exe header.%.html $(wildcard source/*.%.html)
$< --language $(variable_for_%) > $#
(?) Does it replace % by all the languages which have their own header.language.html files. And does the file my_completed_doc.language.html get changed as soon as one of the source/*.language.html get changed?
(?) How to get the % replaced in several prerequisites, possibly into the wildcard and necessarily in the recipe?
First, the easy problem: you wish to use the '%' variable in the recipe. The answer is to use the '$*' automatic variable:
my_complete_doc.%.html: my_trans_exe ...
$< --language $* > $#
Then the easy question: yes, the header.%.html prerequisite is correct. When you try to build my_complete_doc.dutch.html then Make will evaluate it as header.dutch.html, when you try to build my_complete_doc.french.html, Make will evaluate it as header.french.html.
Now the tricky problem: the prerequisite $(wildcard source/*.%.html). Ordinarily, Make expands $(wildcard ...) statements before executing any rule, or deciding which targets to build. So it searches for any files such as source/foo.%.html or source/bar.%.html (that is, files whose names contain the character '%'), finds none, and evaluates the statement as an empty string. But Make will defer this evaluation until it has chosen the rule, if you use SECONDEXPANSION:
.SECONDEXPANSION:
my_complete_doc.%.html: my_trans_exe header.%.html $$(wildcard source/*.%.html)
$< --language $* > $#
(Note the '$$'. In the first -- ordinary -- expansion, Make reduces "$$(...)" to "$(...)", and in the second -- when '%' has a value -- it expands "$(...)".) Now if you modify any file such as source/foo.german.html, Make will consider the file my_complete_doc.german.html out of date and in need of rebuilding.
Here is two targets in my Makefile.
.SECONDARY:
exp-%.ans:
echo $* > eval/$#
%.scores: %.ans
cat eval/$< > eval/$#
When I write make -n exp-40.scores output would be:
echo 40 > eval/exp-40.ans
cat eval/exp-40.ans > eval/exp-40.scores
which is good except one thing. It does not aware of the dependency is already hold. If I create eval/exp-40.scores (first time) then I expect that make will say it is already in there if I run the same command. So I try to change my Makefile like this:
.SECONDARY:
exp-%.ans:
echo $* > eval/$#
%.scores: eval/%.ans
cat eval/$< > $#
When I write make -n exp-40.scores again and output would be:
echo eval/40 > eval/eval/exp-40.ans
cat eval/eval/exp-40.ans > exp-40.scores
which is completely wrong because my parameter should be 40 not eval/40.
How can I achieve the best of the these two worlds? Namely, I want to create *.scores in eval/ folder. I also want make to check whether file is already exist or not. Then make should proceed according to that file existence.
Thanks.
One of the core rules of make is that you need to build the target your rule told make that you'd build. A rule like foo : bar tells make that if it runs that recipe, the recipe will create or update a file named foo. If the recipe creates or updates a file named biz/foo instead, then your makefile is wrong and things will not work.
Make always puts the target it expects the recipe to create into the $# automatic variable. Your recipe should create or update $# and exactly $#. Not something else.
So in your case, you need to write:
eval/exp-%.ans:
echo $* > $#
eval/%.scores: eval/%.ans
cat $< > $#
If you want to be able to run make exp-40.scores and have it actually create eval/exp-40.scores, then you can add a rule like this:
%.scores: eval/%.scores ; #:
(you have to provide some kind of recipe for pattern rules, they cannot have an empty recipe; the one above does nothing).
How can I write the contents of a makefile variable to file, without invoking a shell command?
The problem is that the contents of the variable is possible longer than the shell allows for a command (i.e. longer than MAX_ARG_STRLEN (131072) characters).
In particular, in a makefile I have a variable containing a long list of filenames to process (including their absolute pathes for out-of-source builds). Now I need to write those filenames to a (temporary) file, which I can then pass to another command.
So far, we had a rule like ($COLLATED_FILES is the variable containing the paths):
$(outdir)/collated-files.tely: $(COLLATED_FILES)
$(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" \
--author="$(AUTHOR)" $^
This breaks if COLLATED_FILES is longer than about 130000 characters, we get the error message:
make[2]: execvp: /bin/sh: Argument list too long
As a solution, we are now trying to write the contents of the variable to a file and use that file in the $(LYS_TO_TELY) command. Unfortunately, I have not yet found a way to do this without invoking the shell.
My attempts include:
$(outdir)/collated-files.list: $(COLLATED_FILES)
echo "" > $#
$(foreach f,$^,echo $f >> $#;)
But this also invokes all echo commands at once in a shell, so the shell command is just as long.
Is there any way to write the contents of $(COLLATED_FILES) to a file on disk without passing them on the command line to a shell command?
I also searched whether I could pipe the contents of the variable to the shell, but I couldn't find anything in that direction, either...
Assuming you are using GNU Make, there is the file function!
https://www.gnu.org/software/make/manual/html_node/File-Function.html
$(file op filename,text)
where op is either > or >>.
This requires GNU Make 4.0+
You could move whatever makefile code you use to build up the value of COLLATED_FILES to a trivial helper makefile, then invoke make recursively from your original makefile and use trivial shell redirection to capture the stdout of the recursive make invocation -- basically using make as a rudimentary text-processing tool in that context. For example, create a makefile called get_collated_files.mk with these contents:
COLLATED_FILES=abc
COLLATED_FILES+=def
COLLATED_FILES+=ghi
# ... etc ...
# Use $(info) to print the list to stdout. If you want each filename on a
# separate line, use this instead:
#
# $(foreach name,$(COLLATED_FILES),$(info $(name)))
$(info $(COLLATED_FILES))
all: ;##shell no-op to quell make complaints
Then, in your original makefile:
collated-files.list:
$(MAKE) -f get_collated_files.mk > $#
$(outdir)/collated-files.tely: collated-files.list
$(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" \
--author="$(AUTHOR)" --filelist=collated-files.list
This will be quite a lot more efficient than using hundreds or thousands of individual echo invocations to append to the file one path at a time.
EDIT: One final option, if you really want to have each filename on a separate line, and you have a lot of control over how COLLATED_FILES is defined:
define NL
endef
COLLATED_FILES=abc
COLLATED_FILES+=$(NL)def
COLLATED_FILES+=$(NL)ghi
$(info $(COLLATED_FILES))
all: ;##no-op
This approach allows you to again use just one call to $(info), if that's important to you for some reason.
Here's a patch to gnu make that lets you directly write a variable into a file. It creates a new 'writefile' function, similar to the existing 'info' function, except it takes a filename argument and writes to the file:
https://savannah.gnu.org/bugs/?35384
It looks to me as if you should rethink your build design-- surely there's a better way than letting a variable get this big. But here's a way to do it:
# Make sure this doesn't collide with any of your other targets.
NAMES_TO_WRITE = $(addprefix write_,$(COLLATED_FILES))
collated-files.list: $(NAMES_TO_WRITE)
write_blank:
echo "" > collated-files.list
.PHONY: $(NAMES_TO_WRITE)
$(NAMES_TO_WRITE) : write_% : write_blank
echo $* >> collated-files.list