I have a compile job where linking is taking a lot of IO work. We have around a dozen of cores so we run make -j13, but when it comes to linking the 6 targets, I'd like those to be done in a round robin way. I thought about making one depend on the next but I think this would break the individual targets. Any ideas how to solve this small issue?
make itself doesn't provide a mechanism to request "N of these, but no more than M of those at a time".
You might try using the sem command from the GNU parallel package in the recipe of your linker rules. Its documentation has an example of ensuring only one instance of a tool runs at once. In your example, you would allow make to start up to 13 sems at a time, but only one of those at a time will run the linker, while the others block.
The downside is that you could get into a situation where 5 of your make's 13 job slots are tied up with instances of sem that are all waiting for a linker process to finish. Depending on the structure of your build, that might mean some wasted CPU time. Still beats 6 linkers thrashing the disk at once, though :-)
You should specify that your six targets cannot be built in parallel. Add a line like this to your makefile:
.NOTPARALLEL: target1 target2 target3 target4 target5 target6
For more information look here https://www.gnu.org/software/make/manual/html_node/Parallel-Disable.html.
I've stumbled upon a hacky solution:
For each recipe it runs, Make does two things: it expands variables/functions in the recipe, and then runs the shell commands.
Since the first step can read/write the global variables, it seems to be done synchronously.
So if you run all your shell commands during the first step (using $(shell )), no other recipe will be able to start while they're running.
E.g. consider this makefile:
all: a b
a:
sleep 1
b:
sleep 1
time make -j2 reports 1 second.
But if you rewrite it to this:
# A string of all single-letter Make flags, without spaces.
override single_letter_makeflags = $(filter-out -%,$(firstword $(MAKEFLAGS)))
ifneq ($(findstring n,$(single_letter_makeflags)),)
# See below.
override safe_shell = $(info Would run shell command: $1)
else ifeq ($(filter --trace,$(MAKEFLAGS)),)
# Same as `$(shell ...)`, but triggers a error on failure.
override safe_shell = $(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(.SHELLSTATUS)))
else
# Same functions but with logging.
override safe_shell = $(info Shell command: $1)$(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(>
endif
# Same as `safe_shell`, but discards the output and expands to nothing.
override safe_shell_exec = $(call,$(call safe_shell,$1))
all: a b
a:
$(call safe_shell_exec,sleep 1)
#true
b:
$(call safe_shell_exec,sleep 1)
#true
time make -j2 now reports 2 seconds.
Here, #true does nothing, and suppresses Nothing to be done for ?? output.
There are some problems with this approach though. One is that all output is discarded unless redirected to file or stderr...
It won't break individual targets.
You can create any number of (:) rules for a target, as long as only one of them has an actual recipe for building it. This appears to be a good use case for that.
Related
I'm trying to use find_j=$(findstring j,$(filter-out --%,$(MAKEFLAGS))) to find if there is -j option, so when I echo $(find_j) the value is j
but when I compare it ifeq (j, $(find_j)) this returnes false
I cant understand where is the problem
my version of make is make-3.99.90
find_j=$(findstring -j,$(filter-out --%,$(MAKEFLAGS)))
ifneq ( , $(find_j))
PARALLEL_ENABLED=true
endif
.PHONY: PRINT
PRINT:
$(info $(PARALLEL_ENABLED))
$(info $(MAKEFLAGS))
$(info $(find_j))
---empty line---
--warn-undefined-variables -ws --jobserver-fds=5,6 -j
-j
One thing to note is that the release of GNU make you're using is a beta release of GNU make 4.0, which itself was released in 2013... so you're using a beta of a release that itself is 8.5 years old.
However, that's not related to this problem.
The issue is that the MAKEFLAGS variable's final value is not set until after all makefiles are parsed. If you try to examine it before all makefiles are parsed, it will contain only a subset of the total set of options.
When you expand that variable as part of an ifeq or ifneq statement, that happens as the makefile is being parsed and so (as per the above) only the simple options (ones that don't take an argument: -j accepts an argument so is not "simple") are available.
When you expand the variable as part of a recipe, that happens after all makefiles are parsed: at that time the final value of MAKEFLAGS is set. So your $(info ...) functions inside the recipe do the right thing.
This is easy to see:
$(info no recipe MAKEFLAGS is '$(MAKEFLAGS)')
all: ; $(info in recipe MAKEFLAGS is '$(MAKEFLAGS)')
If you run with -j10 you'll get:
no recipe MAKEFLAGS is ''
in recipe MAKEFLAGS is '-j10 --jobserver-auth=3,4'
(your "in recipe" flags might look different because you're using such an old version of GNU make).
In the next release of GNU make, the value of MAKEFLAGS is kept up-to-date constantly so you can check it at any time and it will be accurate. But that release is not available yet.
This seems to be imprecisely documented. While MAKEFLAGS has the flags like e.g. -s and -k as ks in it, the -j flag gets processed in another way: it is not stripped of the leading dash - AND it is not visible in the first pass of processing the makefile. Only when rules are executed, MAKEFLAGS receives a value, albeit a processed form of the one you gave. -j3 elicits a -j3 --jobserver-auth=3,4 response from the command line transcriber of make, while -j stays -j. So what does this mean for us? Obviously the feature to detect the requested parallelism at runtime is not stable or there are some good reasons not to access them (which is the case most of the time when you encounter exceptional behaviour in GNU tools), so maybe you can give us more information on what you are trying to achieve - maybe there is a way to circumvent accessing the command line.
Our project uses Makefiles with the following type of rule for each multi-directory sub-make:
DIRS = lib audio conf parser control
all: $(DIRS)
#for DIR in $(DIRS); \
do \
( cd $$DIR; $(MAKE) $(MFLAGS) all; ) \
done
If any file fails to compile in one of the leaf makes, the build stops in that directory - but the rest of the make continues. How do I set up these Makefiles so the first error at any level will stop the entire make?
Thanks
From the for loops section of the bash manual:
The return status is the exit status of the last command that
executes.
So, you do not need to capture return statuses. You need your recipe to fail if any sub-make fails:
DIRS = lib audio conf parser control
all: $(DIRS)
#for DIR in $(DIRS); do \
$(MAKE) -C $$DIR $(MFLAGS) all || exit 1; \
done
But it would be much better to have individual recipes per directory, instead of a single for loop:
DIRS = lib audio conf parser control
all: $(DIRS)
.PHONY: all $(DIRS)
$(DIRS):
$(MAKE) -C $# $(MFLAGS) all
This way, if a sub-make fails, it is the complete rule's recipe that fails and make stops. Note the .PHONY special target, in this case it is needed because you want to run the recipe, even if the directory already exists.
There is another advantage with this structure: if you run make in parallel mode (make -j N) it will launch several sub-makes simultaneously instead of just one with the for loop. And each sub-make, in turn, will launch several recipes in parallel, up to N jobs. On a multi-processor or multi-core architecture the speed-up factor can be significant.
But this advantage can become a drawback if your project is not parallel safe, that is, if the order of processing of your directories matters and is not properly defined in the makefiles. If you are in this situation you can add a:
.NOTPARALLEL:
special target at the beginning of your main makefile to tell make. But it would be better to explicitly define the inter-directories dependencies. And if you do not know how to do this, please ask another question.
I found the answer in this question's answer: I have to rewrite to capture the return status of each submake.
Let's say I have a couple of dirs, each of that contains a makefile. Now let's say there is a lib dir (with the makefile too) among of this dirs and there are a references to it in some of the makefiles in other dirs. I.e., there are some amount of the projects which depend on another one. Now, if I run the make utility with the -j option, it may to try to make the lib in two or more parallel threads.
Question is pretty simple: how to prevent this behaviour?
I think the best solution is to use .NOTPARALLEL:
a:
#echo a
#sleep 1
#echo end a
b:
#echo b
#sleep 1
#echo end b
.NOTPARALLEL:
Without .NOTPARALLEL:
$ gmake a b -j 2
a
b
end a
end b
With .NOTPARALLEL:
$ gmake a b -j 2
a
end a
b
end b
GNU make's documentation:
.NOTPARALLEL
If .NOTPARALLEL is mentioned as a target, then this invocation of make
will be run serially, even if the ‘-j’ option is given. Any
recursively invoked make command will still run recipes in parallel
(unless its makefile also contains this target). Any prerequisites on
this target are ignored.
FreeBSD's make documentation:
.NOTPARALLEL
Disable parallel mode.
.NO_PARALLEL
Synonym for .NOTPARALLEL, for compatibility with other pmake variants.
Is there a way how to ask gmake to never run two targets from a set in parallel?
I don't want to use .NOTPARALLEL, because it forces the whole Makefile to be run sequentially, not just the required part.
I could also add dependencies so that one depends on another, but then (apart from being ugly) I'd need to build all of them in order to build the last one, which isn't necessary.
The reason why I need this is that (only a) part of my Makefile invokes ghc --make, which takes care of its dependencies itself. And it's not possible to run it in parallel on two different targets, because if the two targets share some dependency, they can rewrite each other's .o file. (But ghc is fine with being called sequentially.)
Update: To give a specific example. Let's say I need to compile two programs in my Makefile:
prog1 depends on prog1.hs and mylib.hs;
prog2 depends on prog2.hs and mylib.hs.
Now if I invoke ghc --make prog1.hs, it checks its dependencies, compiles both prog1.hs and mylib.hs into their respective object and interface files, and links prog1. The same happens when I call ghc --make prog2.hs. So if they the two commands get to run in parallel, one will overwrite mylib.o of the other one, causing it to fail badly.
However, I need that neither prog1 depends on prog2 nor vice versa, because they should be compilable separately. (In reality they're very large with a lot of modules and requiring to compile them all slows development considerably.)
Hmmm, could do with a bit more information, so this is just a stab in the dark.
Make doesn't really support this, but you can sequential-ise two targets in a couple of ways. First off, a real use for recursive make:
targ1: ; recipe1...
targ2: ; recipe2...
both-targets:
${MAKE} targ1
${MAKE} targ2
So here you can just make -j both-targets and all is fine. Fragile though, because make -j targ1 targ2 still runs in parallel. You can use dependencies instead:
targ1: ; recipe1...
targ2: | targ1 ; recipe2...
Now make -j targ1 targ2 does what you want. Disadvantage? make targ2 will always try to build targ1 first (sequentially). This may (or may not) be a show-stopper for you.
EDIT
Another unsatisfactory strategy is to explicitly look at $MAKECMDGOALS, which lists the targets you specified on the command-line. Still a fragile solution as it is broken when someone uses dependencies inside the Makefile to get things built (a not unreasonable action).
Let's say your makefile contains two independent targets targ1 and targ2. Basically they remain independent until someone specifies on the command-line that they must both be built. In this particular case you break this independence. Consider this snippet:
$(and $(filter targ1,${MAKECMDGOALS)),$(filter targ2,${MAKECMDGOALS}),$(eval targ1: | targ2))
Urk! What's going on here?
Make evaluates the $(and)
It first has to expand $(filter targ1,${MAKECMDGOALS})
Iff targ1 was specified, it goes on to expand $(filter targ2,${MAKECMDGOALS})
Iff targ2 was also specified, it goes on to expand the $(eval), forcing the serialization of targ1 and targ2.
Note that the $(eval) expands to nothing (all its work was done as a side-effect), so that the original $(and) always expands to nothing at all, causing no syntax error.
Ugh!
[Now that I've typed that out, the considerably simpler prog2: | $(filter prog1,${MAKECMDGOALS})
occurs to me. Oh well.]
YMMV and all that.
I'm not familiar with ghc, but the correct solution would be to get the two runs of ghc to use different build folders, then they can happily run in parallel.
Since I got stuck at the same problem, here is another pointer in the direction that make does not provide the functionality you describe:
From the GNU Make Manual:
It is important to be careful when using parallel execution (the -j switch; see Parallel Execution) and archives. If multiple ar commands run at the same time on the same archive file, they will not know about each other and can corrupt the file.
Possibly a future version of make will provide a mechanism to circumvent this problem by serializing all recipes that operate on the same archive file. But for the time being, you must either write your makefiles to avoid this problem in some other way, or not use -j.
What you are attempting, and what I was attempting (using make to insert data in a SQLite3 database) suffers from the exact same problem.
I needed to separate the compilation from other steps (cleaning, building dirs and linking), as I wanted to run the compilation with more core processes and the -j flag.
I managed to solve this, with different makefiles including and calling each other. Only the "compile" make file is running in parallel with all the cores, the rest of the process is syncronous.
I divided my makefile in 3 separate scripts:
settings.mk: contains all the variables and flag definitions
makefile: has all the targets except the compilation one (It has .NOTPARALLEL directive). It calls compile.mk with -j flag
compile.mk: contains only the compile operation (without .NOTPARALLEL)
In settings.mk I have:
CC = g++
DB = gdb
RM = rm
MD = mkdir
CP = cp
MAKE = mingw32-make
BUILD = Debug
DEBUG = true
[... all other variables and flags needed, directories etc ...]
In makefile I have Link and compilation target as these:
include .makefiles/settings.mk
[... OTHER TARGETS (clean, directories etc)]
compilation:
#echo Compilation
#$(MAKE) -f .makefiles/compile.mk --silent -j 8 -Oline
#Link
$(TARGET): compilation
#echo -e Linking $(TARGET)
#$(CC) $(LNKFLAGS) -o $(TARGETDIR)/$(TARGET) $(OBJECTS) $(LIBDIRS) $(LIB)
#Non-File Targets
.PHONY: all prebuild release rebuild clean resources directories run debug
.NOTPARALLEL: all
# include dependency files (*.d) if available
-include $(DEPENDS)
And this is my compile.mk:
include .makefiles/settings.mk
#Defauilt
all: $(OBJECTS)
#Compile
$(BUILDDIR)/%.$(OBJEXT): $(SRCDIR)/%.$(SRCEXT)
#echo -e Compiling: $<
#$(MD) -p $(dir $#)
#$(CC) $(COMFLAGS) $(INCDIRS) -c $< -o $#
#Non-File Targets
.PHONY: all
# include dependency files (*.d) if available
-include $(DEPENDS)
Until now, it's working.
Note that I'm calling compile.mk with -j flag AND -Oline so that parallel processing doesn't mess up with the output.
Any syntax color can be setted in the makefile main script, since the -O flag invalidates escape color codes.
I hope it can help.
I had a similar problem so ended up solving it on the command line, like so:
make target1; make target2
to force it to do the targets sequentially.
I have a makefile which calls multiple other makefiles.
I'd like to pass the -j param along to the other makefile calls.
Something like (make -j8):
all:
make -f libpng_linux.mk -j$(J)
Where $(J) is the value 8 from -j8. I absolutely swear I've done this before but I cannot locate my example.
$(MAKEFLAGS) seems to contain --jobserver-fds=3,4 -j regardless of what -j2 or -j8
Edit: Possible Solution:
Will post this as an answer soon.
It appears one solution to not worry about it. Include -j8 when you call the main makefile. The sub calls to make should look like this:
all:
+make -f libpng_linux.mk -j$(J)
Notice the "+" in front of make. I noticed make tossing a warning when I tried parallel builds: make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
Only certain flags go into $(MAKEFLAGS). -j isn't included because the sub-makes communicate with each other to ensure the appropriate number of jobs are occuring
Also, you should use $(MAKE) instead of make, since $(MAKE) will always evaluate to the correct executable name (which might not be make).
"Do not do that" is not always the answer, but in this case it is, at least for GNU make.
GNU make parent process has an internal jobserver. If top-level Makefile is run with -j, subprocess makes will talk to the jobserver and read a parallelism level from it, without an explicit -j.
Ongoing coordination with parent's jobserver is much better for core utilization. For example, during the same build with -j6, parent could be running 2 jobs and the child 4 more, next moment both could be running 3 jobs each, then a parent would run 1 and the child 5.