I'm wondering if there are any potential race conditions when using evals across targets. So for example, if I have:
all1:
$(eval X:=1)
$(eval Y:=1)
#echo "[$#] X: $(X), Y: $(Y)"
all2:
$(eval Y:=2);
$(eval X:=2);
#echo "[$#] X: $(X), Y: $(Y)"
and then ran
make all all2 -j
Are X and Y guaranteed to be the same values for the same target, or can one instance of the target expansion potentially scribble on the other as it's expanding?
(Background -- I'm debugging an inconsistent make bug on some makefiles which use evals to set some common variables across many targets, and I'm wondering if this could be a contributing factor)
make -j runs each job in a separate process. Variable changes don't transfer between make processes.
Related
So I have a really nice make macro that creates me a pretty message during compilation that is done in such a way that the destination of the message can be implemented externally.
It uses the $(shell ) make function something like this.
send_msg = $(shell $do_send_msg "$1")
Where do_send_msg can be expanded to be echo, wall, email, etc.
I use this lots in the makefiles.
I'd like to be able to use the same macro in recipes as the result of the recipe may change what is needed to be sent, dependent on some test made to the result of an external program.
However variables in recipes are expanded by make entirely before the lines in the recipe are called one at a time.
So if I write (e.g.)
if test_prog; then \
$(call send_msg,PASS);\
else \
$(call send_msg,FAIL);\
fi
Effectively this becomes (not strictly true syntax, but close enough):
(bash -c 'echo "PASS"') &
(bash -c 'echo "FAIL"') &
(bash -c 'if test_prog; then \
\
else \
\
fi)'
This of course will not work, it will run test_prog and, separately, regardless of the exit status both send_msg lines will expand and be executed.
So it will print both PASS and FAIL as the two subshells are run independently and in parallel by make.
I don't want to 'cheat' and use another variable in the $(call ) function or even worse a global that, ok, would allow an if some_var ... in the send_msg implementation but would reduce it's flexability as the implementation would have to understand that variable in all cases.
Another way would be to just have two different send_msg macros, one with and one without the $(shell ) function. Simple, but not elegant.
Right now I am using a 'hack' and calling the same makefile with a variable. If that variable is set then it sends it's contents otherwise does nothing. This works fine but to me it seems clunky and wrong, there must be a better way.....
e.g.
if test_prog; then \
MESSAGE=PASS $(MAKE) message;\
else \
MESSAGE=FAIL $(MAKE) message;\
fi
Where the Makefile says (and this is an abreviated version to convey the idea)
ifneq ($(MESSAGE),)
message:
$(call send_msg,$(MESSAGE))
else:
message:
endif
Question:
How would I make (make) detect if the macro is being expanded inside a Makefile recipe or inside the Makefile body and effectively keep or remove the $(shell ) call that wraps how the work is done?
e.g. (if make_or_shell existed)
ifdef some_test
$(make_or_shell send_msg,"Message from Make")
endif
goal:
$(make_or_shell send_msg,"Message from Recipe")
I have a complicated set of rules I need to write to generate a rather large number of "parameterised" output files and thought that, rather than expand them all out by hand, I could repeatedly "include" a template file with sets of rules and use (GNU)make's facility for allowing "simply expanded" variables to avoid the pain.
(In the past I've always been using the "recursively expanded" variable approach, so this is new to me)
As a trivial example of what I thought would work, I tried putting the following in a Makefile
Targ:=A
Param1:=Pa
Param2:=Qa
$(Targ):
#echo expect A, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
Targ:=B
Param1:=Pb
Param2:=Qb
$(Targ):
#echo expect B, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
Targ:=C
Param1:=Pc
Param2:=Qc
$(Targ):
#echo expect C, get $(Targ), Target is $#. Params are $(Param1) and $(Param2)
The eventual plan was to replace the rules with an include file containing dozens of different rules, each referencing the various "parameter" variables.
However, what I get is...
prompt> make A
expect A, get C, Target is A. Params are Pc and Qc
prompt> make B
expect B, get C, Target is B. Params are Pc and Qc
Essentially, unlike each rule's target, which is picking up the intended definition, the $(Targ), $(Param1), and $(Param2) in each rule's command is instead being run with the final definition.
Does anyone know how to prevent this, i.e. how do you force the command to use the definition at the time it is encountered in the Makefile?
Simple vs recursive expansion makes no difference here; regardless of which you use you'll see the same behavior. A GNU make variable is global and obviously can have only one value.
You have to understand when variables are expanded. The documentation provides a detailed description of this. Targets and prerequisites are expanded when the makefile is read in, so the value of Targ as the makefile is being parsed is used.
Recipes are expanded when the recipe is to be invoked, which is not until after all makefiles are parsed and make starts to build targets. At that time of course the variable Targ has its last set value.
Without knowing what your makefile really does it's hard to suggest an alternative. One option is to use target-specific variables:
Targ := A
$(Targ): LocalTarg := $(Targ)
$(Targ):
#echo expect A, get $(LocalTarg), Target is $#
Another option is to use constructed variable names:
Targ := A
Targ_$(Targ) := $(Targ)
$(Targ):
#echo expect A, get $(Targ_$#), Target is $#
Apologies for answering my own question, but I now realised it is possible to solve the issue I was having by running make recursively.
E.g. if the parameter variables for the rules are Targ, Param1 and Param2 then
#Set up "default" values for the parameters (As #madscientist points out,
#these will safely be overridden by the defs on the #(make) commands below
Targ=XXXXXXXXXX
Param=XXXXXXXXXX
Param2=XXXXXXXXXX
Recursing=
#
# define N (==3) templated rule(s)
#
$(Targ)%a:
#echo Run Combo_a $(Targ) $(Param1) $(Param2) $#
$(Targ)%b:
#echo Run Combo_b $(Targ) $(Param2) $(Param1) reversed $#
$(Targ)%c:
#echo Run Combo_c $(Param1) $(Targ) $(Param2) mixed again $#
#
#Enumerate "M" (==2) sets of parameters,
# (Except if we are already recursing because unrecognised targets may cause
# it to descend forever)
#
ifneq ($(Recursing), Yes)
Set1%:
#$(MAKE) Targ=Set1 Param1=foo Param2=bar Recursing=Yes $#
Set2%:
#$(MAKE) Targ=Set2 Param1=ray Param2=tracing Recursing=Yes $#
endif
This then allows N*M different combos for N+M typing cost.
eg. (removing messages from make re recursion)
>make Set1.a
Run Combo_a Set1 foo bar Set1.a
>make Set2.c
Run Combo_c ray Set2 tracing mixed again Set2.c
I have a rule something, that works on the variable VAR. I also have another rule something-all, that needs to run something, with VAR set to each value in vars.
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach VAR,$(vars),something)
This doesn't quite work, I get
noob#work:~/Desktop$ make something-all
something something
make: something: No such file or directory
make: *** [something-all] Error 1
It should probably print hello\nworld.
I used to do this with wildcard rules by retrieving VAR from %, but got the feeling that was the wrong way to do it. This looked like this:
vars = hello world
all: $(foreach VAR,$(vars),something-$(VAR))
something-%:
echo $*
The below should fix your problem
Using foreach (Tried on GNU Make 3.80 on sparc-solaris 2.8 and windows)
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach i, $(vars), $(MAKE) something VAR=$i || exit 1;)
Using shell for-loop (Tried on GNU Make 3.80 and cc make on sparc-solaris 2.8)
vars = hello world
something:
echo $(VAR)
something-all:
for i in $(vars); do $(MAKE) something VAR=$$i || exit 1; done
TL;DR: If you want to program make, drop GNU Make in favor of BSD Make.
This is a personal recommendation. While BSD Make seems more limited than GNU Make, as it offers less programming facilities, it is much easier to program and has a few unique killer features. This is why I propose a solution with GNU Make and another solution for BSD Make:
Doing it in GNU Make
Using GNU Make, you can write a macro to define a target. The canonical way to define a sequence in a Makefile is to add the steps of the sequence as dependencies to a target, as reflected by the snippet below:
vars= hello world
define something_t =
something: something-$(1)
something-$(1):
#echo $(1)
endef
$(foreach _,$(vars),$(eval $(call something_t,$_)))
It is recommended to use this organisation (rather than defining just one target), because you can work on it to make the task easily resumable if you interrupt the sequence. A Makefile describes a job whose advancement is entirely described by the state of the file system. A task is then easily resumable, if each step is associated to a file, usually a compilation object but sometimes also an empty file which is touch'ed to indicate that important checkpoints have been passed.
Using an auxiliary macro is a flexible solution that can be adapted to more complicated tasks than just echoing a name. Note that this does work with newest versions of GNU Make (4.1). On GNU Make 3.81, you should remove the equal sign from the macro definition.
Adapting your example for BSD Make
If this is an option for you, I recommand dropping the use of GNU Make and replace it by BSD Make, which is way easier to program: it has a short and to the point documentation, while the documentation of GNU Make is very verbose and somewhat unclear, BSD Make has industrial-strength examples of complex rulesets (FreeBSD Build system or BSD Owl), and it has a simple and predictable macro language.
vars= hello world
something:
.for _var in ${vars}
echo ${_var}
.endfor
This can evolve to support more complicated tasks, just by replacing the echo by the adapted commands, or using intermediary steps.
Allow the user to override some tasks, also in BSD Make
In this slightly more advanced variation, we allow the user to override our own recipes for building targets something-hello and something-world.
For each item in our list, a target something-* is created it if it does not already exist, and added to the dependencies of something. The whole operation of defining these targets only happens if something has been left undefined. Therefore, users of these macros can:
Override the recipes for something-hello and something-world
Override the full procedure bound to something.
Implementing such customisation possibilities is mandatory if we want to write useful, reusable, macros for Make. Unluckily, customisation of this sort is nearly impossible in GNU Make.
vars = hello world
.if!target(depend)
.for _var in ${vars}
.if!target(something-${_var})
something-${_var}:
echo ${_var}
.endif
something: something-${_var}
.endfor
.endif
Here's one way to do it:
VARS := hello world
THINGS := $(addprefix something-, $(VARS))
allthings: $(THINGS)
something-%:
echo $*
It should be no surprise that
vars := hello world
something-all:
$(foreach VAR,$(vars),something)
tries to run something something. That's exactly what the foreach expands to, since you don't reference VAR in the third expression.
All you need to do is reference VAR and use a command such as echo:
vars := hello world
something-all:
$(foreach VAR,$(vars),echo $(VAR);)
$ make
echo hello; echo world;
hello
world
Note how chaining the commands with a semicolon avoids forking several shells or -- GASP! -- recursive make invocations. It doesn't get more performant than that.
Alternatively, if your command accepts several somethings as arguments,
vars := hello world
something-all:
echo $(foreach VAR,$(vars),$(VAR))
$ make
echo hello world
hello world
But that is equivalent to the super simple echo $(vars). So it might pay off to think outside the box trying to change your requirements to make this simple solution work.
I have a file "ORIGINAL", which, if updated, I would like to copy, modify, and distribute to a few places on the drive. The modification is made by a little bash script which takes one parameter, a parameter unique for each spawned remote file.
In my Makefile, I can do this with a separate rule/recipe for each parameter, like so:
parameters = AWK BAT CAT DOG
$(DEST_FILE_AWK) : $(ORIGINAL)
./copyAndModify "AWK" ## Creates $(ORIGINAL)_AWK, substed copy of ORIGINAL
mv - f $(ORIGINAL)_AWK $(DEST_FILE_AWK)
$(DEST_FILE_BAT) : $(ORIGINAL)
./copyAndModify "BAT" ## Creates $(ORIGINAL)_BAT, substed copy of ORIGINAL
mv - f $(ORIGINAL)_BAT $(DEST_FILE_BAT)
The dereferenced values of DEST_FILE_AWK and DEST_FILE_BAT have nothing to do with each other, but other than that, the two recipes above are exactly the same with the only difference the parameter, so I can't help but want to merge them into one super rule/recipe with a multiple target rule line.
But I just can't make it happen. I've tried all kinds of foreach() and other stuff in the target section of the rule, but the problem is that no matter what, I can't get the value of the parameter into the recipe part.
Is there a way?
With the information provided here the best you can do (assuming you're using GNU make) is an eval/call combination. As anishsane suggests, depending on the value of the DEST_FILE_* variables it might be possible to do something simpler.
But this should work:
define COPY_TO_DEST
$$(DEST_FILE_$1) : $$(ORIGINAL)
./copyAndModify "$1"
mv - f $$(ORIGINAL)_$1 $$#
endef
parameters = AWK BAT CAT DOG
$(foreach P,$(parameters),$(eval $(call COPY_TO_DEST,$P)))
It can be done without $(eval), at least in gnu make :)
Start with one recipe that specifies all of the targets, i.e. the list of targets is on the left side of the recipe. Let's assume we have a variable that holds the names of all these targets.
Now observe that both functions and variables will be evaluated separately for a given recipe as it gets expanded for each of the targets. Recall that, say $# is just a variable, and will be substituted separately for each target. Function calls behave the same.
Provide a list of types, and a list of type:target pairs. I presume that there's no need to put the targets into separate variables like you did ($(DEST_FILE_AWK) etc).
The TARGET_FOR_TYPE function takes the pairs and the types and generates a list of destination files.
The TYPE variable is assigned once for each target, computed by the TYPE_FOR_TARGET function. That way the repeated function call doesn't pollute the recipe :)
Note that the DESTINATIONS list contains plain filenames, without any further indirection.
types = AWK BAT
ORIGINAL = an_original
DESTINATIONS = \
AWK:dest_for_awk \
BAT:dest_for_bat
TARGET_FOR_TYPE = $(patsubst $(1):%,%,$(filter $(1):%,$(DESTINATIONS)))
TYPE_FOR_TARGET = $(patsubst %:$(1),%,$(filter %:$(1),$(DESTINATIONS)))
# Usage example for the functions above:
$(info type: $(call TYPE_FOR_TARGET,dest_for_awk))
$(info target: $(call TARGET_FOR_TYPE,AWK))
$(info $())
DEST_FILES = $(foreach type,$(types),$(call TARGET_FOR_TYPE,$(type)))
all: $(DEST_FILES)
$(DEST_FILES) : TYPE=$(call TYPE_FOR_TARGET,$#)
$(DEST_FILES) : $(ORIGINAL)
#echo ./copyAndModify $(TYPE)
#echo mv - f $(ORIGINAL)_$(TYPE) $#
I have a compile job where linking is taking a lot of IO work. We have around a dozen of cores so we run make -j13, but when it comes to linking the 6 targets, I'd like those to be done in a round robin way. I thought about making one depend on the next but I think this would break the individual targets. Any ideas how to solve this small issue?
make itself doesn't provide a mechanism to request "N of these, but no more than M of those at a time".
You might try using the sem command from the GNU parallel package in the recipe of your linker rules. Its documentation has an example of ensuring only one instance of a tool runs at once. In your example, you would allow make to start up to 13 sems at a time, but only one of those at a time will run the linker, while the others block.
The downside is that you could get into a situation where 5 of your make's 13 job slots are tied up with instances of sem that are all waiting for a linker process to finish. Depending on the structure of your build, that might mean some wasted CPU time. Still beats 6 linkers thrashing the disk at once, though :-)
You should specify that your six targets cannot be built in parallel. Add a line like this to your makefile:
.NOTPARALLEL: target1 target2 target3 target4 target5 target6
For more information look here https://www.gnu.org/software/make/manual/html_node/Parallel-Disable.html.
I've stumbled upon a hacky solution:
For each recipe it runs, Make does two things: it expands variables/functions in the recipe, and then runs the shell commands.
Since the first step can read/write the global variables, it seems to be done synchronously.
So if you run all your shell commands during the first step (using $(shell )), no other recipe will be able to start while they're running.
E.g. consider this makefile:
all: a b
a:
sleep 1
b:
sleep 1
time make -j2 reports 1 second.
But if you rewrite it to this:
# A string of all single-letter Make flags, without spaces.
override single_letter_makeflags = $(filter-out -%,$(firstword $(MAKEFLAGS)))
ifneq ($(findstring n,$(single_letter_makeflags)),)
# See below.
override safe_shell = $(info Would run shell command: $1)
else ifeq ($(filter --trace,$(MAKEFLAGS)),)
# Same as `$(shell ...)`, but triggers a error on failure.
override safe_shell = $(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(.SHELLSTATUS)))
else
# Same functions but with logging.
override safe_shell = $(info Shell command: $1)$(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(>
endif
# Same as `safe_shell`, but discards the output and expands to nothing.
override safe_shell_exec = $(call,$(call safe_shell,$1))
all: a b
a:
$(call safe_shell_exec,sleep 1)
#true
b:
$(call safe_shell_exec,sleep 1)
#true
time make -j2 now reports 2 seconds.
Here, #true does nothing, and suppresses Nothing to be done for ?? output.
There are some problems with this approach though. One is that all output is discarded unless redirected to file or stderr...
It won't break individual targets.
You can create any number of (:) rules for a target, as long as only one of them has an actual recipe for building it. This appears to be a good use case for that.