I am attempting a small project using meta-programming in GNU Make, and I have a specialized need to programmatically create all of the arguments that will be passed to some of my user-defined functions.
override CMD:=foo
define foo =
$(info You have called function [$0] with $$1='$1' and $$2='$2')
endef
# this doesn't work
override ARGS:=bar,baz
$(call $(CMD),$(ARGS))
# this doesn't work either
override COMMA:=,
override ARGS:=bar;baz
$(call $(CMD),$(subst ;,$(COMMA),$(ARGS)))
$(info ---)
$(info CMD was '$(CMD)')
$(info ARGS was '$(ARGS)')
$(info COMMA was '$(COMMA)')
The problem I run into is that GNU Make is smarter than I want it to be, and can discern the difference between a syntactical comma, and a comma within a variable or immediate value:
You have called function [foo] with $1='bar,baz' and $2=''
You have called function [foo] with $1='bar,baz' and $2=''
---
CMD was 'foo'
ARGS was 'bar;baz'
COMMA was ','
Are there any sneaky ways to do this so that my 'foo' function sees $1 as 'bar', and $2 as 'baz'? Or will I have to abandon this approach altogether?
NOTE1: space delimiters and $(foreach) is not an adequate solution, as that will just be kicking the can down the road. Ultimately, I need foo to be fooled into thinking it has been passed the expected number of arguments. (Although, I am not opposed to solutions that use $(foreach) as some intermediate step.)
NOTE2: While I don't want to discourage solutions that rely on the programmatic generation of a secondary makefile to be included at the bottom of this one-- as that might be useful to other SO readers-- I am not personally interested in any such approach.
Eval will save you from unnecessary includes:
$(eval $$(call $$(CMD),$(ARGS)))
Related
I have a rule something, that works on the variable VAR. I also have another rule something-all, that needs to run something, with VAR set to each value in vars.
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach VAR,$(vars),something)
This doesn't quite work, I get
noob#work:~/Desktop$ make something-all
something something
make: something: No such file or directory
make: *** [something-all] Error 1
It should probably print hello\nworld.
I used to do this with wildcard rules by retrieving VAR from %, but got the feeling that was the wrong way to do it. This looked like this:
vars = hello world
all: $(foreach VAR,$(vars),something-$(VAR))
something-%:
echo $*
The below should fix your problem
Using foreach (Tried on GNU Make 3.80 on sparc-solaris 2.8 and windows)
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach i, $(vars), $(MAKE) something VAR=$i || exit 1;)
Using shell for-loop (Tried on GNU Make 3.80 and cc make on sparc-solaris 2.8)
vars = hello world
something:
echo $(VAR)
something-all:
for i in $(vars); do $(MAKE) something VAR=$$i || exit 1; done
TL;DR: If you want to program make, drop GNU Make in favor of BSD Make.
This is a personal recommendation. While BSD Make seems more limited than GNU Make, as it offers less programming facilities, it is much easier to program and has a few unique killer features. This is why I propose a solution with GNU Make and another solution for BSD Make:
Doing it in GNU Make
Using GNU Make, you can write a macro to define a target. The canonical way to define a sequence in a Makefile is to add the steps of the sequence as dependencies to a target, as reflected by the snippet below:
vars= hello world
define something_t =
something: something-$(1)
something-$(1):
#echo $(1)
endef
$(foreach _,$(vars),$(eval $(call something_t,$_)))
It is recommended to use this organisation (rather than defining just one target), because you can work on it to make the task easily resumable if you interrupt the sequence. A Makefile describes a job whose advancement is entirely described by the state of the file system. A task is then easily resumable, if each step is associated to a file, usually a compilation object but sometimes also an empty file which is touch'ed to indicate that important checkpoints have been passed.
Using an auxiliary macro is a flexible solution that can be adapted to more complicated tasks than just echoing a name. Note that this does work with newest versions of GNU Make (4.1). On GNU Make 3.81, you should remove the equal sign from the macro definition.
Adapting your example for BSD Make
If this is an option for you, I recommand dropping the use of GNU Make and replace it by BSD Make, which is way easier to program: it has a short and to the point documentation, while the documentation of GNU Make is very verbose and somewhat unclear, BSD Make has industrial-strength examples of complex rulesets (FreeBSD Build system or BSD Owl), and it has a simple and predictable macro language.
vars= hello world
something:
.for _var in ${vars}
echo ${_var}
.endfor
This can evolve to support more complicated tasks, just by replacing the echo by the adapted commands, or using intermediary steps.
Allow the user to override some tasks, also in BSD Make
In this slightly more advanced variation, we allow the user to override our own recipes for building targets something-hello and something-world.
For each item in our list, a target something-* is created it if it does not already exist, and added to the dependencies of something. The whole operation of defining these targets only happens if something has been left undefined. Therefore, users of these macros can:
Override the recipes for something-hello and something-world
Override the full procedure bound to something.
Implementing such customisation possibilities is mandatory if we want to write useful, reusable, macros for Make. Unluckily, customisation of this sort is nearly impossible in GNU Make.
vars = hello world
.if!target(depend)
.for _var in ${vars}
.if!target(something-${_var})
something-${_var}:
echo ${_var}
.endif
something: something-${_var}
.endfor
.endif
Here's one way to do it:
VARS := hello world
THINGS := $(addprefix something-, $(VARS))
allthings: $(THINGS)
something-%:
echo $*
It should be no surprise that
vars := hello world
something-all:
$(foreach VAR,$(vars),something)
tries to run something something. That's exactly what the foreach expands to, since you don't reference VAR in the third expression.
All you need to do is reference VAR and use a command such as echo:
vars := hello world
something-all:
$(foreach VAR,$(vars),echo $(VAR);)
$ make
echo hello; echo world;
hello
world
Note how chaining the commands with a semicolon avoids forking several shells or -- GASP! -- recursive make invocations. It doesn't get more performant than that.
Alternatively, if your command accepts several somethings as arguments,
vars := hello world
something-all:
echo $(foreach VAR,$(vars),$(VAR))
$ make
echo hello world
hello world
But that is equivalent to the super simple echo $(vars). So it might pay off to think outside the box trying to change your requirements to make this simple solution work.
My idea is to use computed names to emulate namespaces, for example:
GetVar = $(or $($2#$1),$($2))
fubar#namespace = foo bar
frob = baz
$(info $(call GetVar,namespace,fubar))
$(info $(call GetVar,namespace,frob))
furthermore this could be used in rules to have target local variables
cmd#target = echo hello
target:
$(call GetVar,$#,cmd)
This seem to work on GNU-make, but it raises a few questions:
First of all this will result in using non-normal characters in variable names. Of course one may use another separator than #, but in using the target name as the namespace name you would inevitably run into cases where dots and slashes end up in the variable name. How bad is this? The GNU-make manual tells me to avoid this and mentions the ability to share this via environment to submakes, but this is in a non-recursive makefile setup.
Is this exploiting a non-portable feature of GNU-make? Need I be concerned with feature versions of make breaking this feature?
Is there any example of this idiom actually being used? Not being able to find any cases makes me wonder if there's anything I've missed that makes this a bad idea.
If it's not that bad idea, are there any particular namespace separator that would be more suitable than other?
As far as I can see you're mostly recreating target-specific variables. Is there some reason you don't want to use them?
target: cmd = echo hello
target:
$(cmd)
I have a file "ORIGINAL", which, if updated, I would like to copy, modify, and distribute to a few places on the drive. The modification is made by a little bash script which takes one parameter, a parameter unique for each spawned remote file.
In my Makefile, I can do this with a separate rule/recipe for each parameter, like so:
parameters = AWK BAT CAT DOG
$(DEST_FILE_AWK) : $(ORIGINAL)
./copyAndModify "AWK" ## Creates $(ORIGINAL)_AWK, substed copy of ORIGINAL
mv - f $(ORIGINAL)_AWK $(DEST_FILE_AWK)
$(DEST_FILE_BAT) : $(ORIGINAL)
./copyAndModify "BAT" ## Creates $(ORIGINAL)_BAT, substed copy of ORIGINAL
mv - f $(ORIGINAL)_BAT $(DEST_FILE_BAT)
The dereferenced values of DEST_FILE_AWK and DEST_FILE_BAT have nothing to do with each other, but other than that, the two recipes above are exactly the same with the only difference the parameter, so I can't help but want to merge them into one super rule/recipe with a multiple target rule line.
But I just can't make it happen. I've tried all kinds of foreach() and other stuff in the target section of the rule, but the problem is that no matter what, I can't get the value of the parameter into the recipe part.
Is there a way?
With the information provided here the best you can do (assuming you're using GNU make) is an eval/call combination. As anishsane suggests, depending on the value of the DEST_FILE_* variables it might be possible to do something simpler.
But this should work:
define COPY_TO_DEST
$$(DEST_FILE_$1) : $$(ORIGINAL)
./copyAndModify "$1"
mv - f $$(ORIGINAL)_$1 $$#
endef
parameters = AWK BAT CAT DOG
$(foreach P,$(parameters),$(eval $(call COPY_TO_DEST,$P)))
It can be done without $(eval), at least in gnu make :)
Start with one recipe that specifies all of the targets, i.e. the list of targets is on the left side of the recipe. Let's assume we have a variable that holds the names of all these targets.
Now observe that both functions and variables will be evaluated separately for a given recipe as it gets expanded for each of the targets. Recall that, say $# is just a variable, and will be substituted separately for each target. Function calls behave the same.
Provide a list of types, and a list of type:target pairs. I presume that there's no need to put the targets into separate variables like you did ($(DEST_FILE_AWK) etc).
The TARGET_FOR_TYPE function takes the pairs and the types and generates a list of destination files.
The TYPE variable is assigned once for each target, computed by the TYPE_FOR_TARGET function. That way the repeated function call doesn't pollute the recipe :)
Note that the DESTINATIONS list contains plain filenames, without any further indirection.
types = AWK BAT
ORIGINAL = an_original
DESTINATIONS = \
AWK:dest_for_awk \
BAT:dest_for_bat
TARGET_FOR_TYPE = $(patsubst $(1):%,%,$(filter $(1):%,$(DESTINATIONS)))
TYPE_FOR_TARGET = $(patsubst %:$(1),%,$(filter %:$(1),$(DESTINATIONS)))
# Usage example for the functions above:
$(info type: $(call TYPE_FOR_TARGET,dest_for_awk))
$(info target: $(call TARGET_FOR_TYPE,AWK))
$(info $())
DEST_FILES = $(foreach type,$(types),$(call TARGET_FOR_TYPE,$(type)))
all: $(DEST_FILES)
$(DEST_FILES) : TYPE=$(call TYPE_FOR_TARGET,$#)
$(DEST_FILES) : $(ORIGINAL)
#echo ./copyAndModify $(TYPE)
#echo mv - f $(ORIGINAL)_$(TYPE) $#
Alright I am stuck on this and I have no idea what I am doing wrong. Everything was going great working on a more complicated makefile but then all of a sudden I got the "Missing separator" error. I was able to isolate it down to a very simple scenario:
test.mk
define push_dir
$(info ${1})
endef
define pop_dir
$(info ${1})
endef
define include_submake
$(call push_dir,${1})
$(call pop_dir,${1})
endef
Simple
include test.mk
INITIAL_SUBMAKE:= includeme.mk
$(call include_submake,${INITIAL_SUBMAKE})
process:
#echo Processed...
And the output:
C:\project>make -f Simple process
includeme.mk
includeme.mk
Simple:4: *** missing separator. Stop.
includeme.mk does not actually exist. I have no idea what is going wrong here I have tried a multitude of things. If I surround the call to include_submake in info like so:
$(info $(call include_submake,${INITIAL_SUBMAKE}))
The missing separator error goes away. Also If in the include_submake define I only call one of the functions it works fine. Additionally if I directly call the functions instead of calling them include_submake it works as well:
include test.mk
INITIAL_SUBMAKE:= includeme.mk
$(call push_dir,${INITIAL_SUBMAKE})
$(call pop_dir,${INITIAL_SUBMAKE})
process:
#echo Processed...
C:\project>make -f Simple process
includeme.mk
includeme.mk
Processed...
I feel like I'm overlooking something fundamental here. Thanks for your help.
The missing separator error happens because of a non-empty return value of include_submake, which is a single line feed character in your case. Make only permits whitespace characters (that is, a space or tab) to occur in an expression which is not assumed to be a part of some rule or another directive.
Rewrite your functions using plain-old Make variable assignment and the error should go away:
push_dir = \
$(info $1)
pop_dir = \
$(info $1)
include_submake = \
$(call push_dir,$1) \
$(call pop_dir,$1)
UPD.: define vs plain old variable assignment
Answering to a question from the first comment. Personally I would prefer using define directive in several cases.
Using with eval function
As the GNU Make manual suggests, define directive is very useful in conjunction with the eval function. Example from the manual (emphasis is mine):
PROGRAMS = server client
server_OBJS = server.o server_priv.o server_access.o
server_LIBS = priv protocol
client_OBJS = client.o client_api.o client_mem.o
client_LIBS = protocol
# Everything after this is generic
.PHONY: all
all: $(PROGRAMS)
define PROGRAM_template
$(1): $$($(1)_OBJS) $$($(1)_LIBS:%=-l%)
ALL_OBJS += $$($(1)_OBJS)
endef
$(foreach prog,$(PROGRAMS),$(eval $(call PROGRAM_template,$(prog))))
$(PROGRAMS):
$(LINK.o) $^ $(LDLIBS) -o $#
clean:
rm -f $(ALL_OBJS) $(PROGRAMS)
Generator templates
Verbatim variables fit perfectly for cases when you want to generate a file from GNU Make. For example, consider generating a header file based on some information from Makefile.
# Args:
# 1. Header identifier.
define header_template
/* This file is generated by GNU Make $(MAKE_VERSION). */
#ifndef $(inclusion_guard)
#define $(inclusion_guard)
$(foreach inc,$($1.includes),
#include <$(inc).h>)
/* Something else... */
#endif /* $(inclusion_guard) */
endef
# 1. Unique header identifier.
inclusion_guard = \
__GEN_$1_H
# Shell escape.
sh_quote = \
'$(subst ','"'"',$1)'
foo.includes := bar baz
HEADERS := foo.h
$(HEADERS) : %.h :
#printf "%s" $(call sh_quote,$(call header_template,$(*F)))> $#
Extended Make syntax
In our project we use our own build system called Mybuild, and it is implemented entirely on top of GNU Make. As one of low-level hacks that we used to improve the poor syntax of the builtin language of Make, we have developed a special script which allows one to use extended syntax for function definitions. The script itself is written in Make too, so it is a sort of meta-programming in Make.
In particular, one can use such features as:
Defining multiline functions without the need to use backslash
Using comments inside functions (in plain-old Make comments can only occur outside variable assignment directives)
Defining custom macros like $(assert ...) or $(lambda ...)
Inlining simple functions like $(eq s1,s2) (string equality check)
This is an example of how a function can be written using the extended syntax. Note that it becomes a valid Make function and can be called as usual after a call to $(def_all).
# Reverses the specified list.
# 1. The list
# Return:
# The list with its elements in reverse order.
define reverse
# Start from the empty list.
$(fold ,$1,
# Prepend each new element ($2) to
# the result of previous computations.
$(lambda $2 $1))
endef
$(def_all)
Using these new features we were able to implement some really cool things (well, at least for Make :-) ) including:
Object-Oriented layer with dynamic object allocation, class inheritance, method invocations and so on
LALR parser runtime engine for parsers generated by GOLD Parser Builder
Modelling library with runtime support for models generated with EMF
Feel free to use any part of the code in your own projects!
I ran into the same problem. I inserted 'tab', deleted 'tab', reinserted to be sure. Same error message.
But, I did all of this inside of XCodem which to my surprise inserted white spaces, not '\t'. Once I used different editor these 'phantom' errors went away.
HTH...
I'm trying to create a generic build template for my Makefiles, kind of like they discuss in the eval documentation.
I can't seem to get the wildcard function to work within an eval. The basic code I'm having issues with looks like this.
SRC_DIR = ./src/
PROG_NAME = test
define PROGRAM_template
$(1)_SRC_DIR = $(join $(SRC_DIR), $(1)/)
$(1)_SRC_FILES = $(wildcard $$($(1)_SRC_DIR)*.c)
endef
$(eval $(call PROGRAM_template, $(PROG_NAME)))
all:
#echo $(test_SRC_DIR)
#echo $(test_SRC_FILES)
#echo $(wildcard $(wildcard $(test_SRC_DIR)*.c)
When I run make with this, the output is
./src/test
[correct list of all .c files in ./src/test/]
Basically, the wildcard call within PROGRAM_template is not being eval'd as I expect it. The call results in an empty list.
The join call is being eval'd correctly though.
So, what am I doing wrong? My guess is that
$$($(1)_SRC_DIR)
is not correct, but I can't figure out the right way to do it.
EDIT
Once this was solved, it didn't take long for me to hit another issue with eval.
I posted it as a new question at
Workaround for GNU Make 3.80 eval bug
You need to double escape virtually all of the functions and variables when you use eval. In most cases, the only things that don't need to be double-escaped are function arguments (because the call function will fully expand them). In this case, you technically don't need to double-escape join or SRC_DIR either, but it will simplify your life if you just always double-escape all variables and functions when using eval.
The reason you need the double escapes is that expansion happens twice when using eval. The eval function itself performs expansion, and then expansion is done again when the block is finally parsed as makefile syntax (i.e. when it is actually evaluated).
The way you've got it written, wildcard is invoked on the string literal $( test_SRC_DIR)*.c. If you want, you can see this for yourself by replacing wildcard with info in your version and see what happens.
You need to hold off on actually invoking wildcard until the second expansion, so that it's argument is the result of the expansion of $(test_SRC_DIR).
Try this:
SRC_DIR = ./src/
PROG_NAME = test
define PROGRAM_template
$(1)_SRC_DIR = $$(join $$(SRC_DIR),$(1)/)
$(1)_SRC_FILES = $$(wildcard $$($(1)_SRC_DIR)*.c)
endef
$(eval $(call PROGRAM_template,$(PROG_NAME)))
all:
#echo $(test_SRC_DIR)
#echo $(test_SRC_FILES)
#echo $(wildcard $(test_SRC_DIR)*.c)
EDIT: After posting this, I thought I'd better test it out to make sure it actually works. In doing so, I discovered another problem. You should avoid putting spaces between the comma and argument when calling functions. It causes a literal space character to be prepended to the argument that is passed to the function and leads to unintended results. I've removed the spaces after the commas in the function calls in my version (while this isn't a problem for the call to join, I removed the space there as well just because it's a good habit to get into).