GNU make: foreach behave different in defined and outside - makefile

I am trying to write a custom "function" in GNU Makefile. However, The "$(foreach" function behaves differently between inside "defined" and outside
Makefile steps:
Define current folder is "."
Get all the sub-folders
For each sub-folder, add the suffix "-subdir"
define get_folders
dirs:=$(dir $(wildcard $(1)/*/))
test_function:=$(foreach subdir,$$(dirs), $(subdir)"-subdir")
endef
CURRENT_FOLDER:=.
dirs:=$(dir $(wildcard $(CURRENT_FOLDER)/*/))
test_outside:=$(foreach subdir,$(dirs), $(subdir)"-subdir")
.PHONNY:test_function
test_function:
$(eval $(call get_folders,$(CURRENT_FOLDER)))
#echo "in-function" $(test_function)
#echo "outside" $(test_outside)
when I run the make, following is printed out:
in-function ./header/ ./Tools/-subdir
outside ./header/-subdir ./Tools/-subdir
As you can see, the result of commands outside are as I expected but for the in-function, it seems that all sub-folders are considered one string so only one suffix was added
Can you please kindly explain me why it happened this way and how can I change it?

Your problem comes from the way make expands your recipe. $(eval ...) first expands its parameter ($(call get_folders,$(CURRENT_FOLDER))). If you try to imagine the result you'll understand that the first line of your macro expands as:
dirs:=./header/ ./Tools/
But consider the second line:
test_function:=$(foreach subdir,$$(dirs), $(subdir)"-subdir")
The foreach will iterate only once, on the $(dirs) token, leading to:
test_function:=$(dirs)"-subdir"
Thus the result you observe. Try this, instead:
define get_folders
dirs:=$(dir $(wildcard $(1)/*/))
test_function:=$(foreach subdir,$(dirs), $(subdir)"-subdir")
endef
Note that mixing make and shell programming in your recipes is not ideal. It is difficult to understand, maintain and debug. You could as well separate the two worlds:
define get_folders
dirs:=$$(dir $$(wildcard $(1)/*/))
test_function:=$$(foreach subdir,$$(dirs), $$(subdir)"-subdir")
endef
CURRENT_FOLDER:=.
$(eval $(call get_folders,$(CURRENT_FOLDER)))
.PHONY:test_function
test_function:
#echo "in-function" $(test_function)
The $$ in the macro are used to escape the first expansion by eval, leading to:
dirs:=$(dir $(wildcard ./*/))
test_function:=$(foreach subdir,$(dirs), $(subdir)"-subdir")
If you plan to use this macro with several CURRENT_FOLDER values you can pass it a second parameter to personalize the variable names and avoid conflicts:
# $(1): current folder
# $(2): variable names prefix
define get_folders
$(2)-dirs:=$$(dir $$(wildcard $(1)/*/))
$(2)-subdirs:=$$(foreach subdir,$$($(2)-dirs), $$(subdir)"-subdir")
endef
CURRENT_FOLDER:=.
$(eval $(call get_folders,$(CURRENT_FOLDER),test_function))
.PHONY:test_function
test_function:
#echo "in-function" $($#-subdirs)
Even better, you could turn CURRENT_FOLDER into a target-specific variable:
# $(1): current folder
# $(2): variable names prefix
define get_folders
$(2)-dirs = $$(dir $$(wildcard $(1)/*/))
$(2)-subdirs = $$(foreach subdir,$$($(2)-dirs),$$(subdir)"-subdir")
endef
$(eval $(call get_folders,$$(CURRENT_FOLDER),test_function))
.PHONY:test_function
test_function: CURRENT_FOLDER = .
test_function:
#echo "in-function" $($#-subdirs)

Related

Pass argument of call to another function in Makefile

I would like a way to take the argument to a call invocation in a Makefile rule and pass that to a builtin function, in this case wildcard.
This doesn't seem to work:
MODULE_OBJS = $(patsubst %.cc, %.o, $(wildcard $(1)/*.cc))
lib%.a: $(call MODULE_OBJS, %)
#echo $# : $^
In this case I would expect make libfoo.a to print a list of .o files corresponding to the .cc files found under foo/, but nothing is printed. The parameter is getting there because if I append $(1) to the end of MODULE_OBJS the value of % gets printed, but it seems to be lost when getting passed into wildcard.
You need to understand that make will execute $(call MODULE_OBJS, %) before it has even begun building the dependency tree, you cannot accomplish this with a pattern rule. You could use eval hackery but there's a case to made against trying to be too clever with make.
Something like the following is easy enough to maintain
MODULE_OBJS = $(patsubst %.cc, %.o, $(wildcard $(1)/*.cc))
libfoo.a: $(call MODULE_OBJS, foo)
lib%.a:
#echo $#: $^
but after wrestling with clever ways of generating library and binary dependencies I prefer simply listing them explicitly.
I got what I wanted with some hacking of the eval rule:
EXCLUDE_MODULES = obj
MODULES = $(filter-out $(EXCLUDE_MODULES), $(patsubst %/, %, $(wildcard */)))
define MODULE_RULE
lib$(MODULE).a: $(patsubst %.cc, obj/%.o, $(wildcard $(MODULE)/*.cc))
#echo $# : $^
endef
$(foreach MODULE, $(MODULES), $(eval $(MODULE_RULE)))
This allows you to call make libfoo.a and get out a list of all the .o's corresponding with the .cc's in that subdirectory.
For those curious, I uploaded a complete example here.
The Metaprogramming Make articles were a useful resource here.

Generate dynamically Makefile rules

I have a Makefile which I use to call different sub-Makefiles.
I have several rules:
all
clean
fclean
re
I can already use those rules, which will call every sub makefile with the same rule.
I have several project, and I would like to generate rules with that format:
$(PROJECT_NAME)-$(RULES)
With that, I would like to have each rule for each project:
project1-all
project1-clean
...
project2-all
project2-clean
...
This way, I would be able to call a specific rule, for a specific project, like project1-fclean.
I tried that:
RULES= all clean fclean re
PROJECTS= project1 project2
define NEWLINE
endef
$(foreach _rule, $(RULES), \
$(foreach _proj, $(PROJECTS), \
$(_proj)-$(_rule): $(NEWLINE) \
$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule) $(NEWLINE) \
) \
)
But it doesn't seems to work. I have searched, but I haven't found advanced makefile techniques to achieve that. Plz help.
The problem is that when you combine lines together with line continuations like that, it compresses out all the newlines and other extraneous whitespace (including those newlines you're trying to insert with $(NEWLINE)) resulting in a huge mess on a single line, rather than multiple lines with multiple patterns. To do this correctly, you need to write your rule as a macro with arguments and then call it:
define PROJ_RULE
$(1)-$(2):
$(MAKE) $(ARGS) $(PROJECT_DIR)$(1) $(2)
endef
$(foreach _rule, $(RULES),
$(foreach _proj, $(PROJECTS),
$(eval $(call PROJ_RULE, $(_proj), $(_rule)))))
note that all this define and foreach stuff in GNU make specific -- other make flavors do not support it.
Okay, so I finally managed to do it this way:
$(foreach _rule, $(RULES), $(addsuffix -$(_rule),$(PROJECTS))):
$(eval _rule := $(lastword $(subst -, ,$#)))
$(eval _proj := $(#:%-$(_rule)=%))
#$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule)
I will decompose it for a better explanation:
$(foreach _rule, $(RULES), ...)):
We loop on every RULES and store it in _rule.
$(addsuffix -$(_rule),$(PROJECTS))
We add that rule as a prefix to each of our project. This part generate a rule with every "composed rules". With projet1 and project2 it should result in:
project1-all project2-all project1-clean project2-clean project1-fclean project2-fclean project1-re project2-re:
This way, for any of those rules name, it will be the same rule executed.
$(eval _rule := $(lastword $(subst -, ,$#)))
Here we take the target (if I call project2-clean, $# will be project2-clean), we replace - by a space to obtain project2 clean and take the last work, wich will be clean here. We then evaluate it to store that into _rule.
$(eval _proj := $(#:%-$(_rule)=%))
We use the same technique to store the project name into _proj. We just use a pattern replacement, to remove the rule name and the dash.
#$(MAKE) $(ARGS) $(PROJECT_DIR)$(_proj) $(_rule)
Finally, we call our submakefile we the right path and right rule!

Makefile: Targets for Crosscombinations

I have 2 types of files. Scripts and datasets. I want to write a makefile to run each script with each dataset.
My idea was to create a target for each output, where the target has a name containing the script and the model name. These targets would match a "target-pattern"-rule (of type: some_folder/%.eval). Now the "target-pattern"-rule would need to figure out from its name which files are needed. But this seems to be a hard job.
Is there a better, more elegant way?
Example:
Files: ScriptA, ScriptB, InputA, InputB
Targets/Outputfiles: InputA_ScriptA, InputA_ScriptB, InputB_ScriptA, InputB_ScriptB
# generate all combinations
RT_HW = $(foreach script,$(RT_SCRIPT_HW),$(foreach input, $(RT_INPUTS), $(input)_$(script)))
$(SANDBOX)%.eval: <requires Script X and input X>
You've run into one of the major shortcomings of Make: it isn't very good with wildcards.
You can get the effect you want by generating rules with eval, either by iterating over both variables:
define template
$(1)_$(2).eval: $(1) $(2)
#echo target is $$#
#echo running $(2) on $(1)
endef
$(foreach script,$(RT_SCRIPT_HW),$(foreach input, $(RT_INPUTS), $(eval $(call template,$(input),$(script)))))
or by generating pattern rules, iterating over only one, such as the input:
define template
$(1)_%.eval: $(1) %
#echo target is $$#
#echo running $$* on $(1)
endef
$(foreach input, $(RT_INPUTS), $(eval $(call template,$(input))))

gmake: eval inside a recipe in function

Is there a way to do such thing in a makefile for gmake:
GOALS := g1
define fun_one
#echo "blabla" #this causes an error - maybe can't be recognized as a recipe
endef
define fun_two
$(1):
$(eval $(call fun_one,$(1)))
endef
$(forech goal, $(GOALS), $(eval $(call fun_two,$(goal))))
all: ${GOALS}
As far as I understand, I can't define a part of a recipe outside a function that defines a rule, am I write?
There's no need for the first eval; it just tells Make to enact that line while it's still parsing the definition of fun_one. Eliminate the eval and the makefile will work:
define fun_two
$(1):
$(call fun_one,$(1))
endef

Defining custom GNU make functions

What is the problem with the dep2 function in the sample code below?
dep1 = $(eval makefile_list_$1 := $(MAKEFILE_LIST))$(eval -include $1.mk)$(eval MAKEFILE_LIST := $(makefile_list_$1))
define dep2
$(eval makefile_list_$1 := $(MAKEFILE_LIST))
$(eval -include $1.mk)
$(eval MAKEFILE_LIST := $(makefile_list_$1))
endef
$(call dep1,test)
$(call dep2,test)
.DEFAULT_TARGET: all
.PHONY: all
all:
#echo $#
GNU make 3.81 and 3.82 produce Makefile:10: *** missing separator. Stop. which points to the dep2 call, dep1 is run without errors. The only difference between the two variants is the newlines in dep2 (and the whole point why I'd like to use define).
You forgot the =:
define dep2 =
EDIT:
Put a semicolon at the end of each line. I've tested this and it works (in GNUMake 3.81).
define dep2
$(eval makefile_list_$1 := $(MAKEFILE_LIST));
$(eval -include $1.mk);
$(eval MAKEFILE_LIST := $(makefile_list_$1));
endef
Why these semicolons are necessary I don't know, but in the documentation define seems to be used for multi-line "variables" only when defining sequences of shell commands to be used in recipes, not Make commands, so maybe the rules are a little different.
I would move the $(eval ...) calls outside of dep2. By doing it this way, there's no need for semicolons in dep2. This means doubling the $ signs of some expansions to avoid expansion being done too early. So:
define dep2
makefile_list_$1 := $$(MAKEFILE_LIST)
-include $1.mk
MAKEFILE_LIST := $$(makefile_list_$1)
endef
$(eval $(call dep2,test))
# Quick checks for testing, to be removed from the final code...
$(info $(makefile_list_test))
$(info $(MAKEFILE_LIST))
.DEFAULT_TARGET: all
.PHONY: all
all:
#echo $#
I've tested the code above and it works with Gnu Make 4.0. I would expect it to work back to Gnu Make 3.8x. The $(eval $(call ...)) pattern is what I always do to execute my custom functions, and I've used it for quite a while now.
You can do as the below line to kill the error:
FOO := $(call dep2, test)
I guess the reason is the early version of gcc (3.8.1/2) can only accept nothing as the return of expression.
eg $(info string) returns nothing, but $(call dep2, test) returns 2 newlines charaters.
There is much that can be improved in what you are doing. For one thing you really want to factor the eval calls to a single call at the top.
Your particular problem, however, stems from not understanding that the multiline recursive string the make's define command uses never includes the lady new line. The most natural convention for writing evalable functions is
define Foo
Line1
Line2
endef
You can look at the string eval is seeing and see what this does via the info command, e.g.
$(info $(call Foo,x) $(call Foo,y)).

Resources