I have 2 types of files. Scripts and datasets. I want to write a makefile to run each script with each dataset.
My idea was to create a target for each output, where the target has a name containing the script and the model name. These targets would match a "target-pattern"-rule (of type: some_folder/%.eval). Now the "target-pattern"-rule would need to figure out from its name which files are needed. But this seems to be a hard job.
Is there a better, more elegant way?
Example:
Files: ScriptA, ScriptB, InputA, InputB
Targets/Outputfiles: InputA_ScriptA, InputA_ScriptB, InputB_ScriptA, InputB_ScriptB
# generate all combinations
RT_HW = $(foreach script,$(RT_SCRIPT_HW),$(foreach input, $(RT_INPUTS), $(input)_$(script)))
$(SANDBOX)%.eval: <requires Script X and input X>
You've run into one of the major shortcomings of Make: it isn't very good with wildcards.
You can get the effect you want by generating rules with eval, either by iterating over both variables:
define template
$(1)_$(2).eval: $(1) $(2)
#echo target is $$#
#echo running $(2) on $(1)
endef
$(foreach script,$(RT_SCRIPT_HW),$(foreach input, $(RT_INPUTS), $(eval $(call template,$(input),$(script)))))
or by generating pattern rules, iterating over only one, such as the input:
define template
$(1)_%.eval: $(1) %
#echo target is $$#
#echo running $$* on $(1)
endef
$(foreach input, $(RT_INPUTS), $(eval $(call template,$(input))))
Related
I am trying to write a custom "function" in GNU Makefile. However, The "$(foreach" function behaves differently between inside "defined" and outside
Makefile steps:
Define current folder is "."
Get all the sub-folders
For each sub-folder, add the suffix "-subdir"
define get_folders
dirs:=$(dir $(wildcard $(1)/*/))
test_function:=$(foreach subdir,$$(dirs), $(subdir)"-subdir")
endef
CURRENT_FOLDER:=.
dirs:=$(dir $(wildcard $(CURRENT_FOLDER)/*/))
test_outside:=$(foreach subdir,$(dirs), $(subdir)"-subdir")
.PHONNY:test_function
test_function:
$(eval $(call get_folders,$(CURRENT_FOLDER)))
#echo "in-function" $(test_function)
#echo "outside" $(test_outside)
when I run the make, following is printed out:
in-function ./header/ ./Tools/-subdir
outside ./header/-subdir ./Tools/-subdir
As you can see, the result of commands outside are as I expected but for the in-function, it seems that all sub-folders are considered one string so only one suffix was added
Can you please kindly explain me why it happened this way and how can I change it?
Your problem comes from the way make expands your recipe. $(eval ...) first expands its parameter ($(call get_folders,$(CURRENT_FOLDER))). If you try to imagine the result you'll understand that the first line of your macro expands as:
dirs:=./header/ ./Tools/
But consider the second line:
test_function:=$(foreach subdir,$$(dirs), $(subdir)"-subdir")
The foreach will iterate only once, on the $(dirs) token, leading to:
test_function:=$(dirs)"-subdir"
Thus the result you observe. Try this, instead:
define get_folders
dirs:=$(dir $(wildcard $(1)/*/))
test_function:=$(foreach subdir,$(dirs), $(subdir)"-subdir")
endef
Note that mixing make and shell programming in your recipes is not ideal. It is difficult to understand, maintain and debug. You could as well separate the two worlds:
define get_folders
dirs:=$$(dir $$(wildcard $(1)/*/))
test_function:=$$(foreach subdir,$$(dirs), $$(subdir)"-subdir")
endef
CURRENT_FOLDER:=.
$(eval $(call get_folders,$(CURRENT_FOLDER)))
.PHONY:test_function
test_function:
#echo "in-function" $(test_function)
The $$ in the macro are used to escape the first expansion by eval, leading to:
dirs:=$(dir $(wildcard ./*/))
test_function:=$(foreach subdir,$(dirs), $(subdir)"-subdir")
If you plan to use this macro with several CURRENT_FOLDER values you can pass it a second parameter to personalize the variable names and avoid conflicts:
# $(1): current folder
# $(2): variable names prefix
define get_folders
$(2)-dirs:=$$(dir $$(wildcard $(1)/*/))
$(2)-subdirs:=$$(foreach subdir,$$($(2)-dirs), $$(subdir)"-subdir")
endef
CURRENT_FOLDER:=.
$(eval $(call get_folders,$(CURRENT_FOLDER),test_function))
.PHONY:test_function
test_function:
#echo "in-function" $($#-subdirs)
Even better, you could turn CURRENT_FOLDER into a target-specific variable:
# $(1): current folder
# $(2): variable names prefix
define get_folders
$(2)-dirs = $$(dir $$(wildcard $(1)/*/))
$(2)-subdirs = $$(foreach subdir,$$($(2)-dirs),$$(subdir)"-subdir")
endef
$(eval $(call get_folders,$$(CURRENT_FOLDER),test_function))
.PHONY:test_function
test_function: CURRENT_FOLDER = .
test_function:
#echo "in-function" $($#-subdirs)
I have a Makefile which I use to call different sub-Makefiles.
I have several rules:
all
clean
fclean
re
I can already use those rules, which will call every sub makefile with the same rule.
I have several project, and I would like to generate rules with that format:
$(PROJECT_NAME)-$(RULES)
With that, I would like to have each rule for each project:
project1-all
project1-clean
...
project2-all
project2-clean
...
This way, I would be able to call a specific rule, for a specific project, like project1-fclean.
I tried that:
RULES= all clean fclean re
PROJECTS= project1 project2
define NEWLINE
endef
$(foreach _rule, $(RULES), \
$(foreach _proj, $(PROJECTS), \
$(_proj)-$(_rule): $(NEWLINE) \
$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule) $(NEWLINE) \
) \
)
But it doesn't seems to work. I have searched, but I haven't found advanced makefile techniques to achieve that. Plz help.
The problem is that when you combine lines together with line continuations like that, it compresses out all the newlines and other extraneous whitespace (including those newlines you're trying to insert with $(NEWLINE)) resulting in a huge mess on a single line, rather than multiple lines with multiple patterns. To do this correctly, you need to write your rule as a macro with arguments and then call it:
define PROJ_RULE
$(1)-$(2):
$(MAKE) $(ARGS) $(PROJECT_DIR)$(1) $(2)
endef
$(foreach _rule, $(RULES),
$(foreach _proj, $(PROJECTS),
$(eval $(call PROJ_RULE, $(_proj), $(_rule)))))
note that all this define and foreach stuff in GNU make specific -- other make flavors do not support it.
Okay, so I finally managed to do it this way:
$(foreach _rule, $(RULES), $(addsuffix -$(_rule),$(PROJECTS))):
$(eval _rule := $(lastword $(subst -, ,$#)))
$(eval _proj := $(#:%-$(_rule)=%))
#$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule)
I will decompose it for a better explanation:
$(foreach _rule, $(RULES), ...)):
We loop on every RULES and store it in _rule.
$(addsuffix -$(_rule),$(PROJECTS))
We add that rule as a prefix to each of our project. This part generate a rule with every "composed rules". With projet1 and project2 it should result in:
project1-all project2-all project1-clean project2-clean project1-fclean project2-fclean project1-re project2-re:
This way, for any of those rules name, it will be the same rule executed.
$(eval _rule := $(lastword $(subst -, ,$#)))
Here we take the target (if I call project2-clean, $# will be project2-clean), we replace - by a space to obtain project2 clean and take the last work, wich will be clean here. We then evaluate it to store that into _rule.
$(eval _proj := $(#:%-$(_rule)=%))
We use the same technique to store the project name into _proj. We just use a pattern replacement, to remove the rule name and the dash.
#$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule)
Finally, we call our submakefile we the right path and right rule!
I am VERY new to makefiles. I have discovered a flaw in a make file that causes files in a list to be copied from a single source file instead of each file in the list.
First, there is a sub model variable SUB_MODEL_LIST that contains 0 1 2 3 separated by white space.
Here is the segment that does the copy:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $< to $#
$(call COPY, $(firstword $(TARGET_BIN_LIST)), $#)
TARGET_BIN_LIST_NEW contains new file names separated by white space and is composed of something like this:
file001.200 file001.201 file001.202 file001.203
and TARGET_BIN_LIST contains the existing file names and is composed of something like this:
file001c.200 file001c.201 file001c.202 file001c.203
The last digit in the file extension is the model number.
As I read this, the makefile runs:
#echo copying from $< to $#
$(call COPY, $(firstword $(TARGET_BIN_LIST)), $#)
four times, however, it always use the first file name in the TARGET_BIN_LIST due to the firstword function. This results in file001.200, file001.201, file001.202, file001.203 being created, but they are all copies of file001c.200 when they should be copies of their respective files in the list. Each file relates to a sub model version of the code.
My thought to solve this was to use the word function. Something like this:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $< to $#
$(call COPY, $(word $(sub), $(TARGET_BIN_LIST)), $#)
where sub is an element of SUB_MODEL_LIST, but I am not sure how that will work. Does the above roll out into 4 separate calls, or can it be looked at as a loop that can have an increment value for sub??
I also thought about using a foreach loop:
$(foreach sub,$(SUB_MODEL_LIST),$(call COPY, $(word $(sub), $(TARGET_BIN_LIST)), $(word $(sub), $(TARGET_BIN_LIST_NEW)))
But I get the error:
*** first argument to `word' function must be greater than 0. Stop.
Ok, so I tried:
$(foreach sub,$(SUB_MODEL_LIST),$(call COPY, $(word $(sub)+1, $(TARGET_BIN_LIST)), $(word $(sub)+1, $(TARGET_BIN_LIST_NEW)))
But then I got the error:
*** non-numeric first argument to `word' function. Stop.
Now I'm stuck. I would like to keep the existing implementation in tact at much as possible, but can adopt a loop method if needed.
Thanks for the help!
You have to step back. You're misunderstanding how this works. In make an explicit rule with multiple targets is EXACTLY THE SAME as writing the same rule multiple times, once for each target. So this:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $< to $#
$(call COPY, $(firstword $(TARGET_BIN_LIST)), $#)
If TARGET_BIN_LIST_NEW is file001.200 file001.201 file001.202 file001.203 and TARGET_BIN_LIST is file001c.200 file001c.201 file001c.202 file001c.203, is identical to writing this:
file001.200 : file001c.200 file001c.201 file001c.202 file001c.203
...
file001.201 : file001c.200 file001c.201 file001c.202 file001c.203
...
file001.202 : file001c.200 file001c.201 file001c.202 file001c.203
...
file001.203 : file001c.200 file001c.201 file001c.202 file001c.203
...
So you can clearly see that when each rule is run, the value of $< and $(firstword $(TARGET_BIN_LIST)) will be the same thing (file001c.200).
Is it really the case that whenever ANY of the fileXXXc.YYY files change, you want to rebuild ALL the fileXXX.YYY files? That's what your rule does, but based on the recipe it doesn't seem like that's what you want.
Make is mostly about writing one rule to build one target from zero or more prerequisites. If you use a pattern rule you can do this pretty easily:
all: $(TARGET_BIN_LIST_NEW)
file001.% : file001c.%
#echo copying from $< to $#
$(call COPY,$<,$#)
If your filenames may have a more complex naming convention then you'll need something more complicated.
ETA:
Since your naming convention doesn't fit into make's pattern rule capabilities you'll have to do something fancier. You can use eval to generate the rules, like this:
all: $(TARGET_BIN_LIST_NEW)
define TARGET_BIN_COPY
$(1) : $(basename $(1))c$(suffix $(1))
#echo copying from $$< to $$#
$$(call COPY,$$<,$$#)
endef
$(foreach T,$(TARGET_BIN_LIST_NEW),$(eval $(call TARGET_BIN_COPY,$T)))
# uncomment this for debugging
#$(foreach T,$(TARGET_BIN_LIST_NEW),$(info $(call TARGET_BIN_COPY,$T)))
First off, thank you to MadScientist for your help in clarifying how this works.
This implementation worked for me:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $(filter %$(suffix $#), $(TARGET_BIN_LIST)) to $#
$(call COPY, $(filter %$(suffix $#), $(TARGET_BIN_LIST)), $#)
I want to remove the duplication of recipe in a makefile like the following
SHELL := /bin/bash
a_% : a1_% a2_%
cat $^ > $#
b_% : b1_% b2_% %_b3
cat $^ > $#
However the following does not work. I guess the trick in this SO question does not work with pattern rules.
SHELL := /bin/bash
a_% : a1_% a2_%
b_% : b1_% b2_% %_b3
a_% b_%:
cat $^ > $#
Any suggestions ? ( In my original makefile, recipe duplication is occurring in 4 targets, and each of those take 3 substitutions, so I can't unroll the targets)
--EDIT--
I realized that one way to solve this was the following.
CMD1 = cat $^ > $#
a_% : a1_% a2_%
$(CMD1)
b_% : b1_% b2_% %_b3
$(CMD1)
I believe this does what you asked for:
SHELL := /bin/bash
define STUFF
$(1)_%: $(1)1_% $(1)2_% $(2)
cat $$^ > $$#
endef
$(eval $(call STUFF,a))
$(eval $(call STUFF,b,%_b3))
How this works:
The general form of the rule is defined as STUFF. (You'd obviously want a better name in your own Makefile.) Note the doubling of dollar signs in $$^ and $$#. This protects them from evaluation when $(call ...) is executed. $(1) and $(2) will be replaced by $(call ...) with positional arguments.
$(call STUFF,a) "calls" STUFF with $(1) set to the string a and $(2) set to the empty string. The return value is:
a_%: a1_% a2_%
cat $^ > $#
Note how one $ was stripped from the remaining variables.
$(eval ...) evaluates the return value obtained in the previous step as if that string had been put in the Makefile. So it creates the rule.
Steps 2 and 3 also happen for the b files. It is similar to what happens for the a files except that this time $(2) is set to the string %_b3.
This is essentially the method I've used in the past to avoid duplication of rules for cases where the rules were rather complex. For the specific case you show in your question, I'd use the shared command variable you mention in your question.
Is there a way to do such thing in a makefile for gmake:
GOALS := g1
define fun_one
#echo "blabla" #this causes an error - maybe can't be recognized as a recipe
endef
define fun_two
$(1):
$(eval $(call fun_one,$(1)))
endef
$(forech goal, $(GOALS), $(eval $(call fun_two,$(goal))))
all: ${GOALS}
As far as I understand, I can't define a part of a recipe outside a function that defines a rule, am I write?
There's no need for the first eval; it just tells Make to enact that line while it's still parsing the definition of fun_one. Eliminate the eval and the makefile will work:
define fun_two
$(1):
$(call fun_one,$(1))
endef