GNU make allows 1) parallel execution and 2) specifying several goals in the same invocation:
make -j4 clean all
But, as GNU make parallelizes the goals, some race conditions can occur. Illustration:
$ cat Makefile
clean:
#sleep 1 && rm -f foo
all: foo
#sleep 2 && cat foo
foo:
#echo '$#' > $#
$ make -j4 clean ; make -j4 all
foo
$ make -j4 clean all
cat: foo: No such file or directory
Makefile:5: recipe for target 'all' failed
make: *** [all] Error 1
Is there a nice way to force an order between the goals, but still benefit from the parallel acceleration for each goal? In the above example it would be nice to wait until clean completes before all starts in order to avoid race conditions.
As shown, the separate make invocations work as expected but this is not 100% satisfactory:
Some goals can be invoked simultaneously, some others cannot. Completely forbidding multiple goals can thus be considered as too restrictive. But identifying all valid and invalid combinations is tricky and error prone.
To completely avoid the problem, one could warn all potential users of such a Makefile that multiple goals invocations are not supported in parallel mode, but this warning will inevitably be overlooked by some users.
Race conditions do not always cause errors. Some could apparently work seamlessly but produce erroneous results.
IMHO this seems to root in the problem that in a programming language (in this case the shell) we are able to formulate dependencies which are of fundamentally different nature than the ones that make can handle. In your example there is a dependency of clean on the non-existence of foo, while all has the inverse dependency. If you make both targets active at the same time, this seems to surpass make's theoretical foundation - I don't know if there exists a sensible theory that can handle such relations. All that I could come up with is the explicit formulation:
.PHONY: all clean
clean:
#sleep 1
rm -f foo
all: foo $(filter clean,$(MAKECMDGOALS))
#sleep 2
cat foo
foo: $(filter clean,$(MAKECMDGOALS))
#echo Creating $#
#echo '$#' > $#
I think this is an interesting problem for sure.
I found a workaround (but I am not 100% convinced that it is the best solution and that it has no hidden drawbacks). The idea is to use the MAKECMDGOALS GNU make variable and the conditionals to force the serialization of multiple goals:
ifeq ($(words $(MAKECMDGOALS)),1)
.PHONY: all clean
clean:
#sleep 1 && rm -f foo
all: foo
#sleep 2 && cat foo
foo:
#echo '$#' > $#
else
.NOTPARALLEL:
%:
#$(MAKE) $#
endif
Of course, the condition of the conditional could be more sophisticated, like, for instance, testing if one of the goals matches clean...
Related
Hi i have a makefile that compiles my library and then compiles the program. What i want to do is that the makefile recompile alway i modify my library's files for that i thought in this
ifneq ("$(wildcard $(PATH_LIB.A)","")
FILE_EXIST = 1
else
FILE_EXIST = 0
endif
$(MAIN_PROCESS): $(PATH_LIB.A) check_lib
...thing to do...
$(PATH_LIB.a):
FILE_EXIST = 0
check_lib:
ifeq("$(FILE_EXIST)","0")
$(MAKE) -C $(PATH_MAKEFILE_LIB.A)
endif
My problem es that when i compile it relinks all time "...thins to do..." because is checking all time check_lib as updateable what do you suggest for do what i want to do?
Make is not a scripting language like bash or python. What it needs is a description of inter-dependencies between targets and prerequisites, plus recipes to build them. In your case (but I am not sure I understood all details) you could try:
$(MAIN_PROCESS): $(PATH_LIB.A)
...thing to do...
$(PATH_LIB.A):
$(MAKE) -C $(PATH_MAKEFILE_LIB.A)
And that's all (but continue reading, there is more to understand). This tells make that:
$(MAIN_PROCESS) depends on $(PATH_LIB.A), plus the things to do to build $(MAIN_PROCESS) if it does not exist or if it is older than $(PATH_LIB.A).
$(PATH_LIB.A) depends on nothing, plus what to do if it does not exist.
It almost works. Almost only because if $(PATH_LIB.A) already exists but is out of date (with respect to its own source files) it will not be rebuilt. A solution is to declare it as phony:
.PHONY: $(PATH_LIB.A)
$(MAIN_PROCESS): $(PATH_LIB.A)
...thing to do...
$(PATH_LIB.A):
$(MAKE) -C $(PATH_MAKEFILE_LIB.A)
This way make will always try to rebuild it, even if it already exists. The sub-make will do it if needed, else it will just tell you that it was up to date. But it is not the whole story: as make always tries to rebuild $(PATH_LIB.A), it will consider that $(MAIN_PROCESS) must also be rebuilt, even if the sub-make didn't do anything because $(PATH_LIB.A) was up-to-date.
If this is a problem, more tricky solutions can be used, like using one more sub-make. The idea is the following:
Use make conditionals to create two different contexts of invocation with two different rules for your $(MAIN_PROCESS) target.
On the first invocation of make, the first context is used where $(MAIN_PROCESS) depends on the phony $(PATH_LIB.A) but its recipe, instead of ...thing to do... is a second invocation of make, in the other context.
For this second invocation $(MAIN_PROCESS) depends on the non-phony $(PATH_LIB.A) and will have its normal recipe.
The two contexts are distinguished thanks to a dedicated make variable (SECONDPASS in the code below).
Example:
host> cat lib/Makefile
foo.a: foo.c
touch $#
host> cat Makefile
ifeq ($(SECONDPASS),)
$(MAIN_PROCESS): $(PATH_LIB.A)
$(MAKE) SECONDPASS=1
.PHONY: $(PATH_LIB.A)
$(PATH_LIB.A):
$(MAKE) -C $(dir $#)
else
$(MAIN_PROCESS): $(PATH_LIB.A)
touch $#
endif
host> make --no-print-directory
make -C lib/
touch foo.a
make SECONDPASS=1
touch bar
host> make --no-print-directory
make[1]: 'foo.a' is up to date.
make SECONDPASS=1
make[1]: 'bar' is up to date.
host> touch lib/foo.c
host> make --no-print-directory
make -C lib/
touch foo.a
make SECONDPASS=1
touch bar
host> touch lib/foo.a
host> make --no-print-directory
make -C lib/
make[1]: 'foo.a' is up to date.
make SECONDPASS=1
touch bar
I have a make target that depends on a variable, which contains both PHONY and real targets.
This target needs to depend only on the real targets in the variable.
How can I test a variable to determine if it is a PHONY or not, so I can filter them out?
(I can test for a file's existence inside the recipe, but I don't want my target to be triggered by execution of any of the PHONY targets.)
Thanks!
There is a way to do it, but I would strongly recommend against it. First of, phony targets can be also file targets. And there is no way to tell a phony file target from a non-phony file target.
It looks like the question implies that the phony targets the author wants to ignore are all non-file targets. In this case see the example below.
.PHONY: phony_target .FORCE
.FORCE:
ALL_TARGETS = phony_target file_target undetermined_target
-include detect_phony.inc
all: final_target
# All done
final_target: $(REAL_TARGETS)
# create $# triggered by $?
#touch $#
ifeq (,$(MAKE_RESTARTS))
# Generate the list of real file targets in make include file
detect_phony.inc: .FORCE
#echo 'REAL_TARGETS = ' `ls $(ALL_TARGETS) 2>/dev/null` > $# |:
endif
file_target:
touch $#
undetermined_target phony_target:
# process $#
clean:
rm -f file_target final_target
Here are the test results:
$make clean
rm -f file_target final_target
$ make
# create final_target triggered by
# All done
$ touch file_target
$ make
# create final_target triggered by file_target
# All done
$ make
# All done
As you can see it only triggers the final target when the file target is updated.
Before you criticize - Here are the flaws of this implementation:
make is always called twice, updating the generated detect_phony.inc include file at every run
if detect_phony.inc gets corrupted somehow, make execution will be locked by syntax errors, until you manually delete it.
it can't handle phony file targets as I mentioned before
if another generated include is added in this makefile that requires another restart before detect_phony.inc this functionality will break.
So it this method is hacky and has several gotchas. I would not use it in production environment. I would insist on changing the top level Makefile first.
I have
$(foreach ___project___, $(UNIT_TEST_STUBS),$(MAKE) -C ../../$(___project___) $(UT_CMD) || exit 1;)
I want make to be parallel INSIDE each submake but I don't want the submakes to be executed in parallel.
How do I do this?
As #Michael Livshin points out, you already seem to have answered your own question.
Might be neater to rely on make rather than shell syntax to tie a load of commands together though.
Basically you want make to see something like this:
.PHONY: all
all:
${MAKE} -C ../../foo/ -j9 ut-cmd
${MAKE} -C ../../bar/ -j9 ut-cmd
${MAKE} -C ../../bum/ -j9 ut-cmd
Auto generation from a list is fairly straight-forward.
submakes := foo bar bum
define generate-submake
${MAKE} -C ../../$1/ ut-cmd
endef
.PHONY: all
all: ; $(foreach _,${submakes},$(call generate-submake,$1))
Note the blank line in the definition of generate-submake. It's important.
Run this with make -j9.
If you really want everything in this makefile to run serially (but the sub-makes to be parallel), then just introduce a .NOTPARALLEL target.
.NOTPARALLEL:
In my project, I have a set of programs that are build from sources:
SRC_FILES = $(wildcard $(SRC_DIR)/*.cpp)
TARGETS = $(patsubst $(SRC_DIR)/%.cpp,$(BIN_DIR)/%,$(SRC_FILES))
My build target is simple, and works fine:
all: $(TARGETS)
#echo "- Done target $#"
Now, I want a run target so that all these programs are run from the shell on request. Say, if I have 3 files, I want make to run automatically:
>$ ./test1
>$ ./test2
>$ ./test3
Or
>$ ./test1 && ./test2 && ./test3
I tried this:
run: $(TARGETS)
$(addsuffix && ,$(TARGETS))
That generates the following command:
./test1&& ./test2&&
but it fails, due to the trailing &&
(Of course, I want these to be generated automatically as there can be 3... or 30.)
Edit: actually, the && separator is not required, so something like this:
>$ ./test1; ./test2; ./test3;
will be fine too.
Have some .PHONY line near start of Makefile with
.PHONY: all run
You might have
run: $(TARGETS)
$(addsuffix && ,$(TARGETS)) true
but it is a dirty trick.
Maybe you want to produce the output of test2 into test2.out then you might have
TESTSCRIPTS= $(wildcard test*[0-9])
run: $(patsubst %, %.out, $(TESTSCRIPTS))
test%.out: test%
# here some command to run the test%
As alternatives to Basile Starynkevitch's entirely correct answer here there are (at least) two other options as well.
You can avoid the need to run an unnecessary command (builtin though it might be) to end the list by manually pulling off the first entry (this may in fact be more costly then the shell builtin though).
run: $(TARGETS)
$< $(addprefix &&,$(wordlist 2,$(words $^),$^))
A better option I think, assuming that connecting the commands with && isn't a necessity would be to use $(foreach) to generate the command to be run.
run: $(TARGETS)
$(foreach t,$^,$t;)
The trailing ; in that is crucial as the output from $(foreach) is a single line and you need ; to terminate each shell command (or it is seen as one long command with arguments).
In the GNU-Make manual the How make Reads a Makefile https://www.gnu.org/software/make/manual/make.html#Reading-Makefiles sections says
GNU make does its work in two distinct phases. During the first phase it reads all the makefiles, included makefiles, etc. and internalizes all the variables and their values, implicit and explicit rules, and constructs a dependency graph of all the targets and their prerequisites. During the second phase, make uses these internal structures to determine what targets will need to be rebuilt and to invoke the rules necessary to do so.
I am not able to understand the difference b/w the two phases clearly. May be looking at an example will help understand. Are there any links or tutorials which clarifies what exactly happens in the first phase and second phase.
Take this trivial makefile:
var := some_other_file
some_file: $(var)
some_command $^ $#
After phase one the file will look like the following
var := some_other_file
some_file: some_other_file
some_command $^ $#
Notice how $^ and $# haven't been expanded yet, recipes are only expanded and invoked as part of phase 2.
In phase 2, make will use the rules resulting from phase 1 and determine which targets need to be remade, you can see how make "thinks" by running make with the -d flag (warning: lots of output).
During phase 2 of the above case, after having checked all of some_other_file's dependencies and remade it if necessary, it then considers whether some_other_file is newer then some_file.
If this is the case then (and only then) are the recipes' variables expanded, and the contents of each line is passed to the shell, which here would be some_command some_other_file some_file.
It follows that you can't use $# etc. anywhere except as part of a recipe because automatic variables are only set during phase 2.
foo: $#bar
some_command $^
This will be expanded in phase 1 to:
foo: bar
some_command $^
Which during phase 2 will result in:
foo: bar
some_command bar
Probably not what you wanted.
Some makes have ways of working around this limitation. GNU make for instance has .SECONDEXPANSION, and the following will work as expected:
.SECONDEXPANSION:
foo: $$#bar
some_command $^
Anything after .SECONDEXPANSION will be expanded both during phase 1:
.SECONDEXPANSION:
foo: $#bar
some_command $^
and phase 2:
.SECONDEXPANSION:
foo: foobar
some_command foobar