Let's say I have a couple of dirs, each of that contains a makefile. Now let's say there is a lib dir (with the makefile too) among of this dirs and there are a references to it in some of the makefiles in other dirs. I.e., there are some amount of the projects which depend on another one. Now, if I run the make utility with the -j option, it may to try to make the lib in two or more parallel threads.
Question is pretty simple: how to prevent this behaviour?
I think the best solution is to use .NOTPARALLEL:
a:
#echo a
#sleep 1
#echo end a
b:
#echo b
#sleep 1
#echo end b
.NOTPARALLEL:
Without .NOTPARALLEL:
$ gmake a b -j 2
a
b
end a
end b
With .NOTPARALLEL:
$ gmake a b -j 2
a
end a
b
end b
GNU make's documentation:
.NOTPARALLEL
If .NOTPARALLEL is mentioned as a target, then this invocation of make
will be run serially, even if the ‘-j’ option is given. Any
recursively invoked make command will still run recipes in parallel
(unless its makefile also contains this target). Any prerequisites on
this target are ignored.
FreeBSD's make documentation:
.NOTPARALLEL
Disable parallel mode.
.NO_PARALLEL
Synonym for .NOTPARALLEL, for compatibility with other pmake variants.
Related
I’ve the following makefile, with two rules.
Is there a way to run these two rules in parallel, I mean to maximize
the core’s capabilities? I see this section but not sure that I got how to use it for my purpose, since I want to handle it within the makefile and not from the command line.
I.e. run module1 & 2 targets in parallel.
This is the makefile:
all: module1 module2
.PHONY: module1
module1:
#echo "run module 1"
DIR=$(PWD)
#echo $(DIR)
.PHONY: module2
module2:
#echo "run module2”
cleanup:
fzr clean $(DIR)
You can set make options that you usually pass to make via its command line invokation in the makefile itself. Add this line to your makefile
MAKEFLAGS += -j2
and you can invoke make without the -j flag, it will still spawn two processes to build targets in parallel, when they aren't dependent on each other. To automatically determine the number of jobs to spawn, you can use this on linux
NPROCS = $(shell grep -c 'processor' /proc/cpuinfo)
MAKEFLAGS += -j$(NPROCS)
and on MacOS
NPROCS = $(shell sysctl hw.ncpu | grep -o '[0-9]\+')
MAKEFLAGS += -j$(NPROCS)
Even a MAKEFLAGS += -j$(NPROCS) would not work if you have '::' instead of ':' for your rule, as illustrated by that recent fix in Git 2.25.2 (March 2020)
See commit 2607d39 (18 Feb 2020) by Jeff King (peff).
(Merged by Junio C Hamano -- gitster -- in commit 29b09c5, 02 Mar 2020)
doc-diff: use single-colon rule in rendering Makefile
Signed-off-by: Jeff King
When rendering the troff manpages to text via "man", we create an ad-hoc Makefile and feed it to "make".
The purpose here is two-fold:
reuse results from a prior interrupted render of the same tree
use make's -j option to build in parallel
But the second part doesn't seem to work (at least with my version of GNU make, 4.2.1). It just runs one render at a time.
We use a double-colon "all" rule for each file, like:
all:: foo
foo:
...actual render recipe...
all:: bar
bar:
...actual render recipe...
...and so on...
And it's this double-colon that seems to inhibit the parallelism.
We can just switch to a regular single-colon rule.
Even though we do have multiple rules for "all" here, we don't have any recipe to execute for "all" (we only care about triggering its dependencies), so the distinction is irrelevant.
I have a makefile which calls multiple other makefiles.
I'd like to pass the -j param along to the other makefile calls.
Something like (make -j8):
all:
make -f libpng_linux.mk -j$(J)
Where $(J) is the value 8 from -j8. I absolutely swear I've done this before but I cannot locate my example.
$(MAKEFLAGS) seems to contain --jobserver-fds=3,4 -j regardless of what -j2 or -j8
Edit: Possible Solution:
Will post this as an answer soon.
It appears one solution to not worry about it. Include -j8 when you call the main makefile. The sub calls to make should look like this:
all:
+make -f libpng_linux.mk -j$(J)
Notice the "+" in front of make. I noticed make tossing a warning when I tried parallel builds: make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
Only certain flags go into $(MAKEFLAGS). -j isn't included because the sub-makes communicate with each other to ensure the appropriate number of jobs are occuring
Also, you should use $(MAKE) instead of make, since $(MAKE) will always evaluate to the correct executable name (which might not be make).
"Do not do that" is not always the answer, but in this case it is, at least for GNU make.
GNU make parent process has an internal jobserver. If top-level Makefile is run with -j, subprocess makes will talk to the jobserver and read a parallelism level from it, without an explicit -j.
Ongoing coordination with parent's jobserver is much better for core utilization. For example, during the same build with -j6, parent could be running 2 jobs and the child 4 more, next moment both could be running 3 jobs each, then a parent would run 1 and the child 5.
I have a compile job where linking is taking a lot of IO work. We have around a dozen of cores so we run make -j13, but when it comes to linking the 6 targets, I'd like those to be done in a round robin way. I thought about making one depend on the next but I think this would break the individual targets. Any ideas how to solve this small issue?
make itself doesn't provide a mechanism to request "N of these, but no more than M of those at a time".
You might try using the sem command from the GNU parallel package in the recipe of your linker rules. Its documentation has an example of ensuring only one instance of a tool runs at once. In your example, you would allow make to start up to 13 sems at a time, but only one of those at a time will run the linker, while the others block.
The downside is that you could get into a situation where 5 of your make's 13 job slots are tied up with instances of sem that are all waiting for a linker process to finish. Depending on the structure of your build, that might mean some wasted CPU time. Still beats 6 linkers thrashing the disk at once, though :-)
You should specify that your six targets cannot be built in parallel. Add a line like this to your makefile:
.NOTPARALLEL: target1 target2 target3 target4 target5 target6
For more information look here https://www.gnu.org/software/make/manual/html_node/Parallel-Disable.html.
I've stumbled upon a hacky solution:
For each recipe it runs, Make does two things: it expands variables/functions in the recipe, and then runs the shell commands.
Since the first step can read/write the global variables, it seems to be done synchronously.
So if you run all your shell commands during the first step (using $(shell )), no other recipe will be able to start while they're running.
E.g. consider this makefile:
all: a b
a:
sleep 1
b:
sleep 1
time make -j2 reports 1 second.
But if you rewrite it to this:
# A string of all single-letter Make flags, without spaces.
override single_letter_makeflags = $(filter-out -%,$(firstword $(MAKEFLAGS)))
ifneq ($(findstring n,$(single_letter_makeflags)),)
# See below.
override safe_shell = $(info Would run shell command: $1)
else ifeq ($(filter --trace,$(MAKEFLAGS)),)
# Same as `$(shell ...)`, but triggers a error on failure.
override safe_shell = $(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(.SHELLSTATUS)))
else
# Same functions but with logging.
override safe_shell = $(info Shell command: $1)$(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(>
endif
# Same as `safe_shell`, but discards the output and expands to nothing.
override safe_shell_exec = $(call,$(call safe_shell,$1))
all: a b
a:
$(call safe_shell_exec,sleep 1)
#true
b:
$(call safe_shell_exec,sleep 1)
#true
time make -j2 now reports 2 seconds.
Here, #true does nothing, and suppresses Nothing to be done for ?? output.
There are some problems with this approach though. One is that all output is discarded unless redirected to file or stderr...
It won't break individual targets.
You can create any number of (:) rules for a target, as long as only one of them has an actual recipe for building it. This appears to be a good use case for that.
I trying to make parallel make work on our build server. I am running into a very frequent problem here that two instances of make trying to make two different targets, say A and B, nearly simultaneously try to make a target which is required by both, say C.
As both instances try to make C together in different instances, C make fails for either of them since one making of C requires some files to be moved here and there and either one of these instances ends up moving or deleting an already created file.
Is there is common construct that I can use to prevent a re-entry into a makefile if the target is already being made ?
Update:
Ok let me put it this way :
My application requires A.lo and B.lo to be present. These A.lo and B.lo are libraries which also link against C.lo.
So the rules look like
app.d : A.lo B.lo (other lo s)
(do linking)
In some other directory say A (which will house A.lo) :
A.lo : C.lo (other .o s and .lo s)
(do linking)
In some other directory say B (which will house B.lo) :
B.lo : C.lo (other .o s and .lo s)
(do linking)
So in effect while making app.d make forks off two parallel makes for targets A.lo and B.lo.
Upon entering directories A and B make forks off another two threads for target C.lo independently and at times both of these are linking C.lo at the same time, which causes one of them to fail with some weird errors like file not recognized (since it may be written onto by other linker instance)
How should I go about solving this? It is not possible to create A.lo and B.lo without C.lo linked against them.
This may sound a little, well, obvious, but the simplest solution is just to arrange for C to be built explicitly before either A or B. That way when the recursive makes for A and B run, C will already be up-to-date, and neither will bother trying to build it.
If your toplevel makefile looks like this:
all: buildA buildB
buildA:
$(MAKE) -C A
buildB:
$(MAKE) -C B
You can extend it like this:
all: buildA buildB
buildA: buildC
$(MAKE) -C A
buildB: buildC
$(MAKE) -C B
buildC:
$(MAKE) -C C
You didn't give a lot of detail about your specific build, so probably this won't just drop-in and work, but hopefully you get the idea.
I solve this problem using the "mkdir" technique:
SHELL = /bin/bash
all: targetC targetD
targetC: targetA
............
targetD: targetB
............
targetA targetB: source
-mkdir source.lock && ( $(command) < source ; rm -r source.lock )
$(SHELL) -c "while [ -d source.lock ] ; do sleep 0.1 ; done"
I would be happy to see a more elegant solution, though.
I am using Makefiles.
However, there is a command (zsh script) I want executed before any targets is executed.
How do I do this?
Thanks!
There are several techniques to have code executed before targets are built. Which one you should choose depends a little on exactly what you want to do, and why you want to do it. (What does the zsh script do? Why do you have to execute it?)
You can either do like #John suggests; placing the zsh script as the first dependency. You should then mark the zsh target as .PHONY unless it actually generates a file named zsh.
Another solution (in GNU make, at least) is to invoke the $(shell ...) function as part of a variable assignment:
ZSH_RESULT:=$(shell zsh myscript.zsh)
This will execute the script as soon as the makefile is parsed, and before any targets are executed. It will also execute the script if you invoke the makefile recursively.
Just make that a dependancy of one of the other targets
foo.obj : zsh foo.c
rule for compileing foo.c
zsh:
rule for running zsh script.
or alternatively, make your first target depend on it
goal: zsh foo.exe
Solution for both preprocessing and postprocessing in makefiles using MAKECMDGOALS and double colon rules.
MAKECMDGOALS are the targets listed on the command line.
First step is to get the first and last targets from the command line,
or if there are no targets listed, use the default target.
ifneq ($(MAKECMDGOALS),)
FIRST_GOAL := $(word 1, $(MAKECMDGOALS))
LAST_GOAL := $(word $(words $(MAKECMDGOALS)), $(MAKECMDGOALS))
else
FIRST_GOAL := all
LAST_GOAL := all
endif
Double colon rules allow multiple recipes for the same target executed in order. You'll have to change all command line targets to double colon rules.
#Dummy rule to set the default
.PHONY: all
all ::
#Preprocessing
$(FIRST_GOAL) ::
echo "Starting make..."
all :: normal_prerequistes
normal_recipe
other_stuff
#Postprocessing
$(LAST_GOAL) ::
echo "All done..."
There is a solution without modifying your existing Makefile (main difference with the accepted answer). Just create a makefile containing:
.PHONY: all
all:
pre-script
#$(MAKE) -f Makefile --no-print-directory $(MAKECMDGOALS) MAKE='$(MAKE) -f Makefile'
post-script
$(MAKECMDGOALS): all ;
The only drawback is that the pre- and post- scripts will always be run, even if there is nothing else to do. But they will not be run if you invoke make with one of the --dry-run options (other difference with the accepted answer).