I have
$(foreach ___project___, $(UNIT_TEST_STUBS),$(MAKE) -C ../../$(___project___) $(UT_CMD) || exit 1;)
I want make to be parallel INSIDE each submake but I don't want the submakes to be executed in parallel.
How do I do this?
As #Michael Livshin points out, you already seem to have answered your own question.
Might be neater to rely on make rather than shell syntax to tie a load of commands together though.
Basically you want make to see something like this:
.PHONY: all
all:
${MAKE} -C ../../foo/ -j9 ut-cmd
${MAKE} -C ../../bar/ -j9 ut-cmd
${MAKE} -C ../../bum/ -j9 ut-cmd
Auto generation from a list is fairly straight-forward.
submakes := foo bar bum
define generate-submake
${MAKE} -C ../../$1/ ut-cmd
endef
.PHONY: all
all: ; $(foreach _,${submakes},$(call generate-submake,$1))
Note the blank line in the definition of generate-submake. It's important.
Run this with make -j9.
If you really want everything in this makefile to run serially (but the sub-makes to be parallel), then just introduce a .NOTPARALLEL target.
.NOTPARALLEL:
Related
GNU make allows 1) parallel execution and 2) specifying several goals in the same invocation:
make -j4 clean all
But, as GNU make parallelizes the goals, some race conditions can occur. Illustration:
$ cat Makefile
clean:
#sleep 1 && rm -f foo
all: foo
#sleep 2 && cat foo
foo:
#echo '$#' > $#
$ make -j4 clean ; make -j4 all
foo
$ make -j4 clean all
cat: foo: No such file or directory
Makefile:5: recipe for target 'all' failed
make: *** [all] Error 1
Is there a nice way to force an order between the goals, but still benefit from the parallel acceleration for each goal? In the above example it would be nice to wait until clean completes before all starts in order to avoid race conditions.
As shown, the separate make invocations work as expected but this is not 100% satisfactory:
Some goals can be invoked simultaneously, some others cannot. Completely forbidding multiple goals can thus be considered as too restrictive. But identifying all valid and invalid combinations is tricky and error prone.
To completely avoid the problem, one could warn all potential users of such a Makefile that multiple goals invocations are not supported in parallel mode, but this warning will inevitably be overlooked by some users.
Race conditions do not always cause errors. Some could apparently work seamlessly but produce erroneous results.
IMHO this seems to root in the problem that in a programming language (in this case the shell) we are able to formulate dependencies which are of fundamentally different nature than the ones that make can handle. In your example there is a dependency of clean on the non-existence of foo, while all has the inverse dependency. If you make both targets active at the same time, this seems to surpass make's theoretical foundation - I don't know if there exists a sensible theory that can handle such relations. All that I could come up with is the explicit formulation:
.PHONY: all clean
clean:
#sleep 1
rm -f foo
all: foo $(filter clean,$(MAKECMDGOALS))
#sleep 2
cat foo
foo: $(filter clean,$(MAKECMDGOALS))
#echo Creating $#
#echo '$#' > $#
I think this is an interesting problem for sure.
I found a workaround (but I am not 100% convinced that it is the best solution and that it has no hidden drawbacks). The idea is to use the MAKECMDGOALS GNU make variable and the conditionals to force the serialization of multiple goals:
ifeq ($(words $(MAKECMDGOALS)),1)
.PHONY: all clean
clean:
#sleep 1 && rm -f foo
all: foo
#sleep 2 && cat foo
foo:
#echo '$#' > $#
else
.NOTPARALLEL:
%:
#$(MAKE) $#
endif
Of course, the condition of the conditional could be more sophisticated, like, for instance, testing if one of the goals matches clean...
In my project, I have a set of programs that are build from sources:
SRC_FILES = $(wildcard $(SRC_DIR)/*.cpp)
TARGETS = $(patsubst $(SRC_DIR)/%.cpp,$(BIN_DIR)/%,$(SRC_FILES))
My build target is simple, and works fine:
all: $(TARGETS)
#echo "- Done target $#"
Now, I want a run target so that all these programs are run from the shell on request. Say, if I have 3 files, I want make to run automatically:
>$ ./test1
>$ ./test2
>$ ./test3
Or
>$ ./test1 && ./test2 && ./test3
I tried this:
run: $(TARGETS)
$(addsuffix && ,$(TARGETS))
That generates the following command:
./test1&& ./test2&&
but it fails, due to the trailing &&
(Of course, I want these to be generated automatically as there can be 3... or 30.)
Edit: actually, the && separator is not required, so something like this:
>$ ./test1; ./test2; ./test3;
will be fine too.
Have some .PHONY line near start of Makefile with
.PHONY: all run
You might have
run: $(TARGETS)
$(addsuffix && ,$(TARGETS)) true
but it is a dirty trick.
Maybe you want to produce the output of test2 into test2.out then you might have
TESTSCRIPTS= $(wildcard test*[0-9])
run: $(patsubst %, %.out, $(TESTSCRIPTS))
test%.out: test%
# here some command to run the test%
As alternatives to Basile Starynkevitch's entirely correct answer here there are (at least) two other options as well.
You can avoid the need to run an unnecessary command (builtin though it might be) to end the list by manually pulling off the first entry (this may in fact be more costly then the shell builtin though).
run: $(TARGETS)
$< $(addprefix &&,$(wordlist 2,$(words $^),$^))
A better option I think, assuming that connecting the commands with && isn't a necessity would be to use $(foreach) to generate the command to be run.
run: $(TARGETS)
$(foreach t,$^,$t;)
The trailing ; in that is crucial as the output from $(foreach) is a single line and you need ; to terminate each shell command (or it is seen as one long command with arguments).
In this project, a top level rule looks like this:
all:
for l in $(LIBDIRS); do $(MAKE) -C $$l all; done
If I run make -jx, the files in each directory are compiled in parallel, but because of this bash loop, make doesn't move onto files in the next library until everything is done in the current one. What's the best way to change the loop so that it call truly be done in parallel? I only use gnumake, so gnu extensions are fine.
* Edit * as an aside, I was trying something like this
ALL_DEPENDENCY = $(foreach l, $(LIBDIRS), $(l).PHONY_LIB_RULE)
.PHONY: $(ALL_DEPENDENCY)
%.PHONY_LIB_RULE:
$(MAKE) --no-print-directory -C $(patsubst %.PHONY_LIB_RULE,%,$#) all
all: $(ALL_DEPENDENCY)
but these rules aren't tripped.
You can check this answer for an example of how to do this more correctly: makefile: foreach "make -C" call
I'd like to run my makefile without the -w flag turned on by the recursive make calls.
The flag to do that is --no-print-directory on the make command line.
Is it possible to specify that flag within the makefile itself?
I plan to make this flag dependent on a VERBOSE mode, perhaps something like
$(if $(VERBOSE),,MAKEFLAGS += no-print-directory))
Thanks,
Dan
Yes, just appending --no-print-directory to MAKEFLAGS should be enough, but you have to do that with conditional directives, not with conditional functions:
ifndef VERBOSE
MAKEFLAGS += --no-print-directory
endif
You can include the .SILENT: special target in the calling makefile. For example, here's your toplevel makefile:
all:
$(MAKE) -f sub.mk foo
.SILENT:
and the submake makefile, sub.mk:
foo:
#echo done
Note that .SILENT is considered obsolete, so it may not be around forever, and also note that including that in your makefile also has the effect of suppressing command echo, just as if you had put # before every command in the makefile.
I need to work on a system that uses automake tools and makes recursively.
'make -n' only traces the top level of make.
Is there a way to cause make to execute a make -n whenever he encounters a make command?
Use $(MAKE) to call your submakefiles, instead of using make. That should work. Check out How the MAKE variable works in the manual. Here's a quick example:
Makefile:
all:
#$(MAKE) -f Makefile2
Makefile2:
all:
#echo Makefile2
Command line:
$ make
Makefile2
$ make -n
make -f Makefile2
echo Makefile2
$
Does your recursive makefile look like this:
foo:
make -C src1
make -C src2
Or like this:
foo:
${MAKE} -C src1
${MAKE} -C src2
I think you need to use the second style if you want flags passed to child make processes. Could be your problem.
Setting the environment variable "MAKEFLAGS" to "n" may do what you need.
There are some more advanced tricks for tracing make commands here:
http://www.cmcrossroads.com/ask-mr-make/6535-tracing-rule-execution-in-gnu-make
The simplest of these tricks comes down to adding SHELL="sh -x" to your make command (running without "-n" in that case).