Starting from the same problem discussed here: Multiple stems on makefile target
Assume ls produces a b c d and I want to create files a-b b-d c-b,
etc. To make a-b, I'd use a command such as cat a b > a-b, and
similarly for the other ones. I wanted to use a makefile, but couldn't
figure out how. I needed something such as:
FILES := a-b b-d c-b
$(FILES): %1-%2: %1 %2
cat $^ > $#
Here %1 and %2 would be something like \1 and \2 in some regex notations.
How complex it would be improve the gnu-make code to allow such type of multiple stems?
I know that there are other workflow management system (like snakemake) that support this features, but I like very much gnu-make. This is the biggest issue I found.
You explicitly say that you want to stick to GNU make. Still, I'd like to post a solution using snakemake in case you find it appealing enough to change your mind:
SINGLE_FILES = ['a', 'b', 'c', 'd'] # Or some code that generates this list
wildcard_constraints:
# This prevents a, b, c, etc to be interpreted as regex
file= '|'.join([re.escape(x) for x in SINGLE_FILES]),
rule all:
input:
['a-b', 'b-d', 'c-b'],
rule ls:
output:
f= '{file}',
shell:
r"""
touch {output.f}
"""
rule cat:
input:
# Assume output file names 'a-b', 'b-d' etc can be safely
# split at '-' to obtain the input pair of files
ff= lambda wildcard: wildcards.cat.split('-'),
output:
cat= '{cat}',
shell:
r"""
cat {input.ff} > {output.cat}
"""
Save as a file named Snakemake and execute as something like:
snakemake -p -j 1
I don't use GNU make but I think that for complex operations snakemake is a better solution.
Related
I'm trying to write a Makefile that uses the same recipe for several variables. The variables are grouped into several lots which need to be built separately but with the same method.
In the example below, I want to be able to build output1.txt, output2.txt etc. using the same recipe. The catch is that each of these relies on the corresponding variable $(target1), $(target2) etc. I've tried to use the % pattern to expand these variables but that didn't work because % is expanded last.
I have found that I can modify variable calls using recursive variables, eg. $($(subst 0,$(N),target0)). However, this cannot be done unless the number is already known.
Is there a simple way to do this?
target1:=a_5_2 a_3_5 a_6_2 a_0_0 a_9_1
target2:=a_2_2 a_1_3 a_5_9 a_2_7 a_3_6
target3:=a_2_3 a_6_5 a_9_0 a_3_4 a_3_9
target4:=a_7_8 a_8_2 a_4_8 a_7_1 a_0_7
output%.txt: $($(subst 0,%,target0))
cat $^ > $# # do some actual stuff here
You can do this with secondary expansion:
.SECONDEXPANSION:
output%.txt: $$(target$$*)
...
Of course, you could also just change the variables to be prerequisite settings instead; it's about the same amount of typing:
output1.txt : a_5_2 a_3_5 a_6_2 a_0_0 a_9_1
output2.txt : a_2_2 a_1_3 a_5_9 a_2_7 a_3_6
output3.txt : a_2_3 a_6_5 a_9_0 a_3_4 a_3_9
output4.txt : a_7_8 a_8_2 a_4_8 a_7_1 a_0_7
output%.txt:
cat $^ > $# # do some actual stuff here
I have a complicated set of rules I need to write to generate a rather large number of "parameterised" output files and thought that, rather than expand them all out by hand, I could repeatedly "include" a template file with sets of rules and use (GNU)make's facility for allowing "simply expanded" variables to avoid the pain.
(In the past I've always been using the "recursively expanded" variable approach, so this is new to me)
As a trivial example of what I thought would work, I tried putting the following in a Makefile
Targ:=A
Param1:=Pa
Param2:=Qa
$(Targ):
#echo expect A, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
Targ:=B
Param1:=Pb
Param2:=Qb
$(Targ):
#echo expect B, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
Targ:=C
Param1:=Pc
Param2:=Qc
$(Targ):
#echo expect C, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
The eventual plan was to replace the rules with an include file containing dozens of different rules, each referencing the various "parameter" variables.
However, what I get is...
prompt> make A
expect A, get C, Target is A. Params are Pc and Qc
prompt> make B
expect B, get C, Target is B. Params are Pc and Qc
Essentially, unlike each rule's target, which is picking up the intended definition, the $(Targ), $(Param1), and $(Param2) in each rule's command is instead being run with the final definition.
Does anyone know how to prevent this, i.e. how do you force the command to use the definition at the time it is encountered in the Makefile?
Simple vs recursive expansion makes no difference here; regardless of which you use you'll see the same behavior. A GNU make variable is global and obviously can have only one value.
You have to understand when variables are expanded. The documentation provides a detailed description of this. Targets and prerequisites are expanded when the makefile is read in, so the value of Targ as the makefile is being parsed is used.
Recipes are expanded when the recipe is to be invoked, which is not until after all makefiles are parsed and make starts to build targets. At that time of course the variable Targ has its last set value.
Without knowing what your makefile really does it's hard to suggest an alternative. One option is to use target-specific variables:
Targ := A
$(Targ): LocalTarg := $(Targ)
$(Targ):
#echo expect A, get $(LocalTarg), Target is $#
Another option is to use constructed variable names:
Targ := A
Targ_$(Targ) := $(Targ)
$(Targ):
#echo expect A, get $(Targ_$#), Target is $#
Apologies for answering my own question, but I now realised it is possible to solve the issue I was having by running make recursively.
E.g. if the parameter variables for the rules are Targ, Param1 and Param2 then
#Set up "default" values for the parameters (As #madscientist points out,
#these will safely be overridden by the defs on the #(make) commands below
Targ=XXXXXXXXXX
Param=XXXXXXXXXX
Param2=XXXXXXXXXX
Recursing=
#
# define N (==3) templated rule(s)
#
$(Targ)%a:
#echo Run Combo_a $(Targ) $(Param1) $(Param2) $#
$(Targ)%b:
#echo Run Combo_b $(Targ) $(Param2) $(Param1) reversed $#
$(Targ)%c:
#echo Run Combo_c $(Param1) $(Targ) $(Param2) mixed again $#
#
#Enumerate "M" (==2) sets of parameters,
# (Except if we are already recursing because unrecognised targets may cause
# it to descend forever)
#
ifneq ($(Recursing), Yes)
Set1%:
#$(MAKE) Targ=Set1 Param1=foo Param2=bar Recursing=Yes $#
Set2%:
#$(MAKE) Targ=Set2 Param1=ray Param2=tracing Recursing=Yes $#
endif
This then allows N*M different combos for N+M typing cost.
eg. (removing messages from make re recursion)
>make Set1.a
Run Combo_a Set1 foo bar Set1.a
>make Set2.c
Run Combo_c ray Set2 tracing mixed again Set2.c
I have a shell command where it outputs multiple lines. I want to store it in a variable in makefile for later processing in the target.
A simplified example:
I have this file called zfile1
#zfile1
some text
$echo 123
more text
$$$#&^$
more text
The makefile:
a:
#$(eval v1 = $(shell cat zfile1))
# need to process the variable here, example:
#echo "$(v1)"
# I want to prevent expansion of values in the file and print in multi-line
If you have GNU make 4.2 or above you can use the $(file <zfile1) function. See https://www.gnu.org/software/make/manual/html_node/File-Function.html
If you don't have a new-enough version of GNU make, you can't do it. Of course in your example there's no real need to put the contents of the file into a make variable at all: you can just have your recipe use the file itself. But maybe your real use-case isn't so simple.
ETA
You should never use either the make function eval or the make function shell in a recipe [1].
You can just write:
v1 := $(file <zfile1)
.ONESHELL:
a:
#echo "$(v1)"
You must have the .ONESHELL because otherwise each line of the recipe (after it expands into multiple lines) is considered a separate recipe line. Note that .ONESHELL is in effect for the entire make process so could cause other recipes to break if they rely on each line being invoked in a different shell.
Another option is to export the result into the environment, and use a shell variable like this:
export v1 := $(file <zfile1)
a:
#echo "$$v1"
There are probably better ways to do it but since you've only given us this example and not what you really want to do, that's about all we can say.
[1] There are times where it can be useful but if you have a sufficiently sophisticated requirement to need this you'll probably know how to use it.
I think you're making things too complicated.
Start by writing your recipes as proper self-contained shell scripts.
You can then either store the whole script in a file and run it from make, or you can include it directly in your makefile as a single logical line, as in the following:
a:
#v1=$$(< zfile1); \
echo $$v1
Note the need to "escape" the dollar sign by repeating it.
You could also use global make variables, depending on the actual logic of your real-world use.
I have a file "ORIGINAL", which, if updated, I would like to copy, modify, and distribute to a few places on the drive. The modification is made by a little bash script which takes one parameter, a parameter unique for each spawned remote file.
In my Makefile, I can do this with a separate rule/recipe for each parameter, like so:
parameters = AWK BAT CAT DOG
$(DEST_FILE_AWK) : $(ORIGINAL)
./copyAndModify "AWK" ## Creates $(ORIGINAL)_AWK, substed copy of ORIGINAL
mv - f $(ORIGINAL)_AWK $(DEST_FILE_AWK)
$(DEST_FILE_BAT) : $(ORIGINAL)
./copyAndModify "BAT" ## Creates $(ORIGINAL)_BAT, substed copy of ORIGINAL
mv - f $(ORIGINAL)_BAT $(DEST_FILE_BAT)
The dereferenced values of DEST_FILE_AWK and DEST_FILE_BAT have nothing to do with each other, but other than that, the two recipes above are exactly the same with the only difference the parameter, so I can't help but want to merge them into one super rule/recipe with a multiple target rule line.
But I just can't make it happen. I've tried all kinds of foreach() and other stuff in the target section of the rule, but the problem is that no matter what, I can't get the value of the parameter into the recipe part.
Is there a way?
With the information provided here the best you can do (assuming you're using GNU make) is an eval/call combination. As anishsane suggests, depending on the value of the DEST_FILE_* variables it might be possible to do something simpler.
But this should work:
define COPY_TO_DEST
$$(DEST_FILE_$1) : $$(ORIGINAL)
./copyAndModify "$1"
mv - f $$(ORIGINAL)_$1 $$#
endef
parameters = AWK BAT CAT DOG
$(foreach P,$(parameters),$(eval $(call COPY_TO_DEST,$P)))
It can be done without $(eval), at least in gnu make :)
Start with one recipe that specifies all of the targets, i.e. the list of targets is on the left side of the recipe. Let's assume we have a variable that holds the names of all these targets.
Now observe that both functions and variables will be evaluated separately for a given recipe as it gets expanded for each of the targets. Recall that, say $# is just a variable, and will be substituted separately for each target. Function calls behave the same.
Provide a list of types, and a list of type:target pairs. I presume that there's no need to put the targets into separate variables like you did ($(DEST_FILE_AWK) etc).
The TARGET_FOR_TYPE function takes the pairs and the types and generates a list of destination files.
The TYPE variable is assigned once for each target, computed by the TYPE_FOR_TARGET function. That way the repeated function call doesn't pollute the recipe :)
Note that the DESTINATIONS list contains plain filenames, without any further indirection.
types = AWK BAT
ORIGINAL = an_original
DESTINATIONS = \
AWK:dest_for_awk \
BAT:dest_for_bat
TARGET_FOR_TYPE = $(patsubst $(1):%,%,$(filter $(1):%,$(DESTINATIONS)))
TYPE_FOR_TARGET = $(patsubst %:$(1),%,$(filter %:$(1),$(DESTINATIONS)))
# Usage example for the functions above:
$(info type: $(call TYPE_FOR_TARGET,dest_for_awk))
$(info target: $(call TARGET_FOR_TYPE,AWK))
$(info $())
DEST_FILES = $(foreach type,$(types),$(call TARGET_FOR_TYPE,$(type)))
all: $(DEST_FILES)
$(DEST_FILES) : TYPE=$(call TYPE_FOR_TARGET,$#)
$(DEST_FILES) : $(ORIGINAL)
#echo ./copyAndModify $(TYPE)
#echo mv - f $(ORIGINAL)_$(TYPE) $#
I have a list of filenames:
FILES := a.b c.d e.f
and I want to remove the extensions (suffixes) of all words to obtain:
a c e
what is the best way to do that?
The best I could come up with was "cheating" with shell:
$(shell for f in $(INS_NODIR); do echo -n "$${f%.*} "; done )
but I am surprised there was not a more "built-in" way of doing this only with make built-in functions.
thing I tried:
patsubst. It seems that it can only have one single wildcard, others being treated literally, and I'd like to do something like %.%, %
looking for a notsufix function.
I was surprised that this does not exist, since the dir function has notdir counterpart, but the suffix function that exactly extracts extensions does not have a notsuffix counterpart
Simple, just:
NAMES = $(basename $(FILES))
See the GNU make manual section on Functions for File Names