For a project spread over several sub-directories we use GNU make for the builds. Developers can use the -j <number> flag to parallelize the build tasks, and choose a number that suits the hardware of their machines.
However, the Makefiles of a third-party library that we use are not safe to parallelize - they apparently rely on implicit order of targets instead of explicit dependency rules between all dependent targets.
Since I have no desire to fix third-party Makefiles, we currently invoke their Makefiles with an explicit -j 1 parameter to restrict the number of jobs to 1 for building that library. The rule looks like this:
third_party_lib:
$(MAKE) -j 1 -C $#
This works as desired, however, make emits a warning for this:
make[1]: warning: -jN forced in submake: disabling jobserver mode.
which leads me to ask here if there is a better way to restrict the number of parallel jobs in one sub-make.
You can add the .NOTPARALLEL: special target to the makefiles which should not be parallelized.
If you don't want to modify those makefiles you can use the --eval option on the command line (note --eval was added in GNU make 3.82):
third_party_lib:
$(MAKE) --eval .NOTPARALLEL: -C $#
Related
I have a Makefile as below, and I am concerned with the dependency order for up-clean:
.PHONY: up
up: down
docker-compose up -d
.PHONY: up-clean
up-clean: down-clean up
.PHONY: down
down:
docker-compose down
.PHONY: down-clean
down-clean:
docker-compose down -v
Obviously it is important that in up-clean: down-clean up, down-clean must be executed before up. For ordinary make targets, the solution would be to add an entry, up: down-clean, but as these are PHONY targets, that would make up functionally equivalent to up-clean, removing volumes every time. Obviously, this is unacceptable.
In practice, GNU make respects the order of dependencies, but it does not guarantee it, and so is not entirely trustworthy, or with -j not trustworthy at all. What methods, if any, exist to ensure the order of execution of dependencies in this situation without changing the result of building other targets?
EDIT: Attempting to use order-only prerequisites does not appear to work, possibly because of an interaction with .PHONY. Adding
up: | down-clean
Causes the execution log to be:
$ make up
docker-compose down
<...>
docker-compose down -v
Removing volume <...>
Which is what is supposed to happen for normal prerequisites, not order-only ones.
The simplest answer is to use recursive invocations of make:
up-clean:
$(MAKE) down-clean
$(MAKE) up
Another alternative would be to model up-clean on up instead of making the latter a prerequisite for the former:
.PHONY: up
up: down
docker-compose up -d
.PHONY: up-clean
up-clean: down-clean
docker-compose up -d
If you want to make that a little DRYer, you could factor out the docker-compose command to a variable:
UP_COMMAND = docker-compose up -d
.PHONY: up
up: down
$(UP_COMMAND)
.PHONY: up-clean
up-clean: down-clean
$(UP_COMMAND)
Starting with GNU make 4.4 you can explicitly serialize your prerequisites with the .WAIT pseudo-target, e.g.,
.PHONY: up-clean
up-clean: down-clean .WAIT up
Before 4.4, the GNU make documentation was avoiding making any commitments about the order or execution to enable parallel execution. However, any POSIX-compliant implementation of make (emphasis mine)
... shall treat all prerequisites as targets themselves and recursively ensure that they are up-to-date, processing them in the order in which they appear in the rule. The make utility shall use the modification times of files to determine whether the corresponding targets are out-of-date.
Of course, in the parallel mode, GNU make can't be fully compliant with this requirement.
But, the 4.4 release also adds the --shuffle option, and from the research made by the implementor of this option, it is evident that the only source of non-determinism is the -j option and the parallel mode of execution, which is also witnessed by the contents of the patch and the tests, which were reviewed by other members of the GNU make project.
Therefore, in version prior to 4.4, we can safely assume that the prerequisites are executed strictly in the order in which they are specified, from left to right, as long as make is executed in non-parallel mode. In versions before 4.4, we can disable parallelism with the .NOTPARALLEL pseudo target, just add it to your file, e.g.,
.NOTPARALLEL: # ensures that all deps are executed strictly in order
.PHONY: up
up: down
docker-compose up -d
.PHONY: up-clean
up-clean: down-clean up
.PHONY: down
down:
docker-compose down
.PHONY: down-clean
down-clean:
docker-compose down -v
Notice also, that --shuffle respects the presence of the .NOTPARALLEL target, which corroborates our hypothesis the order could different from the syntactic order only because of the parallel execution.
Just out of curiosity, what is the order of executing targets in a makefile with
${OBJ_DIR}/%.o: ${SRC_DIR}/%.cpp
I noticed it is not lexicographic (like ls -l).
Is it just random?
They are built in the order in which make walks the prerequisite graph.
In the simple case where you don't have parallel jobs (no -j option), then if you have a target like:
prog: foo.o bar.o. baz.o
make will first try to build foo.o, then bar.o, then baz.o, then finally prog.
If you do enable parallel jobs, then make will still try to start builds in the same order but because some builds finish faster than others, you may get different targets building at the same time.
I am not aware of any way to define programatically targets in GNU Make. How is this possible?
Sometimes one can go away with alternate methods. The ability to define programatically targets in Makefiles is however a very important to write and organise complex production rules with make. Examples of complex production rules are found in the build system of FreeBSD or in Makefile libraries such as BSD Owl
The main differences between shell scripts and Makefiles are:
In a Makefile, the state of the program is given by the command line and the filesystem, so it is possible to resume a job after it has been interrupted. Of course, this requires to properly write the Makefiles, but even if this is rather hard, it is considerably easier than to achieve a similar effect with a shell script.
In a Makefile, it is ridiculously easy to decorate a procedure with advises or decorate it with hooks, while this is essentially impossible in shell scripts.
For instance, a very simple and useful pattern is the following:
build: pre-build
build: do-build
build: post-build
This presents the build target as a composite of three targets, one containing the actual instructions do-build and two other that are hooks, executed before and after do-build. This pattern is used by many build systems written for BSD Make, which incidentally allows programmatic definition of targets, so that one can write in a batch:
.for _target in configure build test install
.if !target(${_target})
${_target}: pre-${_target}
${_target}: do-${_target}
${_target}: post-${_target}
.endif
.endfor
The condition introduced by the .if/.endif block enables the user to use its own definition of any ${_target}.
What would be the translation of that snippet for GNU Make?
FWIW here is the make equivalent syntax for
.for _target in configure build test install
.if !target(${_target})
${_target}: pre-${_target}
${_target}: do-${_target}
${_target}: post-${_target}
.endif
.endfor
Basically, you want make to see something like this snippet:
build: pre-build
build: do-build
build: post-build
and similarly for configure, test and install. This suggests a loop with an eval somewhere:
define makerule =
$1: pre-$1
$1: do-$1
$1: post-$1
endef
targets := configure build test install
$(foreach _,${targets},$(eval $(call makerule,$_)))
(to play with this, change eval to info). Careful with those closures!
FWIW, here's the expansion of the foreach:
make expands the list to be iterated over
${targets} becomes configure, build, test and install
We have $(foreach _,configure build test install,$(eval $(call makerule,$_)))
_ is set to the first value, configure.
make expands $(eval $(call makerule,configure))
To evaluate the eval, make expands $(call makerule,configure)
It does this by setting 1 to configure, and expanding ${makerule} which produces 3 lines of text:
configure: pre-configure
configure: do-configure
configure: post-configure
$(eval) goes to work, reading this text as make syntax
Note that the expansion of the $(eval) is empty! All its work is done as a side effect.
Wash, lather, rinse, repeat.
Please note: I have to agree with all the other commenters: your pattern is bad make. If your makefile is not -j safe, then it is broken (missing dependencies).
First this structure is invalid if you ever want to support parallel builds; if you invoke make with the -j option it will run all three prerequisite rules at the same time, because while all of them must be complete before build, none of them depend on each other so there's no ordering defined (that is, you don't say that pre-build must be complete before do-build can run).
Second, GNU make has a number of facilities for programmatically defining rules. One thing GNU make does not have, currently, is the ability to search the targets which are already defined, so there's no direct analogy to .if !target(...).
However, you CAN search whether a variable has been defined or not using the .VARIABLES variable. So one workaround would be to define a variable if you want your own target and then have your rule generator check that.
Is there a way how to ask gmake to never run two targets from a set in parallel?
I don't want to use .NOTPARALLEL, because it forces the whole Makefile to be run sequentially, not just the required part.
I could also add dependencies so that one depends on another, but then (apart from being ugly) I'd need to build all of them in order to build the last one, which isn't necessary.
The reason why I need this is that (only a) part of my Makefile invokes ghc --make, which takes care of its dependencies itself. And it's not possible to run it in parallel on two different targets, because if the two targets share some dependency, they can rewrite each other's .o file. (But ghc is fine with being called sequentially.)
Update: To give a specific example. Let's say I need to compile two programs in my Makefile:
prog1 depends on prog1.hs and mylib.hs;
prog2 depends on prog2.hs and mylib.hs.
Now if I invoke ghc --make prog1.hs, it checks its dependencies, compiles both prog1.hs and mylib.hs into their respective object and interface files, and links prog1. The same happens when I call ghc --make prog2.hs. So if they the two commands get to run in parallel, one will overwrite mylib.o of the other one, causing it to fail badly.
However, I need that neither prog1 depends on prog2 nor vice versa, because they should be compilable separately. (In reality they're very large with a lot of modules and requiring to compile them all slows development considerably.)
Hmmm, could do with a bit more information, so this is just a stab in the dark.
Make doesn't really support this, but you can sequential-ise two targets in a couple of ways. First off, a real use for recursive make:
targ1: ; recipe1...
targ2: ; recipe2...
both-targets:
${MAKE} targ1
${MAKE} targ2
So here you can just make -j both-targets and all is fine. Fragile though, because make -j targ1 targ2 still runs in parallel. You can use dependencies instead:
targ1: ; recipe1...
targ2: | targ1 ; recipe2...
Now make -j targ1 targ2 does what you want. Disadvantage? make targ2 will always try to build targ1 first (sequentially). This may (or may not) be a show-stopper for you.
EDIT
Another unsatisfactory strategy is to explicitly look at $MAKECMDGOALS, which lists the targets you specified on the command-line. Still a fragile solution as it is broken when someone uses dependencies inside the Makefile to get things built (a not unreasonable action).
Let's say your makefile contains two independent targets targ1 and targ2. Basically they remain independent until someone specifies on the command-line that they must both be built. In this particular case you break this independence. Consider this snippet:
$(and $(filter targ1,${MAKECMDGOALS)),$(filter targ2,${MAKECMDGOALS}),$(eval targ1: | targ2))
Urk! What's going on here?
Make evaluates the $(and)
It first has to expand $(filter targ1,${MAKECMDGOALS})
Iff targ1 was specified, it goes on to expand $(filter targ2,${MAKECMDGOALS})
Iff targ2 was also specified, it goes on to expand the $(eval), forcing the serialization of targ1 and targ2.
Note that the $(eval) expands to nothing (all its work was done as a side-effect), so that the original $(and) always expands to nothing at all, causing no syntax error.
Ugh!
[Now that I've typed that out, the considerably simpler prog2: | $(filter prog1,${MAKECMDGOALS})
occurs to me. Oh well.]
YMMV and all that.
I'm not familiar with ghc, but the correct solution would be to get the two runs of ghc to use different build folders, then they can happily run in parallel.
Since I got stuck at the same problem, here is another pointer in the direction that make does not provide the functionality you describe:
From the GNU Make Manual:
It is important to be careful when using parallel execution (the -j switch; see Parallel Execution) and archives. If multiple ar commands run at the same time on the same archive file, they will not know about each other and can corrupt the file.
Possibly a future version of make will provide a mechanism to circumvent this problem by serializing all recipes that operate on the same archive file. But for the time being, you must either write your makefiles to avoid this problem in some other way, or not use -j.
What you are attempting, and what I was attempting (using make to insert data in a SQLite3 database) suffers from the exact same problem.
I needed to separate the compilation from other steps (cleaning, building dirs and linking), as I wanted to run the compilation with more core processes and the -j flag.
I managed to solve this, with different makefiles including and calling each other. Only the "compile" make file is running in parallel with all the cores, the rest of the process is syncronous.
I divided my makefile in 3 separate scripts:
settings.mk: contains all the variables and flag definitions
makefile: has all the targets except the compilation one (It has .NOTPARALLEL directive). It calls compile.mk with -j flag
compile.mk: contains only the compile operation (without .NOTPARALLEL)
In settings.mk I have:
CC = g++
DB = gdb
RM = rm
MD = mkdir
CP = cp
MAKE = mingw32-make
BUILD = Debug
DEBUG = true
[... all other variables and flags needed, directories etc ...]
In makefile I have Link and compilation target as these:
include .makefiles/settings.mk
[... OTHER TARGETS (clean, directories etc)]
compilation:
#echo Compilation
#$(MAKE) -f .makefiles/compile.mk --silent -j 8 -Oline
#Link
$(TARGET): compilation
#echo -e Linking $(TARGET)
#$(CC) $(LNKFLAGS) -o $(TARGETDIR)/$(TARGET) $(OBJECTS) $(LIBDIRS) $(LIB)
#Non-File Targets
.PHONY: all prebuild release rebuild clean resources directories run debug
.NOTPARALLEL: all
# include dependency files (*.d) if available
-include $(DEPENDS)
And this is my compile.mk:
include .makefiles/settings.mk
#Defauilt
all: $(OBJECTS)
#Compile
$(BUILDDIR)/%.$(OBJEXT): $(SRCDIR)/%.$(SRCEXT)
#echo -e Compiling: $<
#$(MD) -p $(dir $#)
#$(CC) $(COMFLAGS) $(INCDIRS) -c $< -o $#
#Non-File Targets
.PHONY: all
# include dependency files (*.d) if available
-include $(DEPENDS)
Until now, it's working.
Note that I'm calling compile.mk with -j flag AND -Oline so that parallel processing doesn't mess up with the output.
Any syntax color can be setted in the makefile main script, since the -O flag invalidates escape color codes.
I hope it can help.
I had a similar problem so ended up solving it on the command line, like so:
make target1; make target2
to force it to do the targets sequentially.
I have a makefile which calls multiple other makefiles.
I'd like to pass the -j param along to the other makefile calls.
Something like (make -j8):
all:
make -f libpng_linux.mk -j$(J)
Where $(J) is the value 8 from -j8. I absolutely swear I've done this before but I cannot locate my example.
$(MAKEFLAGS) seems to contain --jobserver-fds=3,4 -j regardless of what -j2 or -j8
Edit: Possible Solution:
Will post this as an answer soon.
It appears one solution to not worry about it. Include -j8 when you call the main makefile. The sub calls to make should look like this:
all:
+make -f libpng_linux.mk -j$(J)
Notice the "+" in front of make. I noticed make tossing a warning when I tried parallel builds: make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
Only certain flags go into $(MAKEFLAGS). -j isn't included because the sub-makes communicate with each other to ensure the appropriate number of jobs are occuring
Also, you should use $(MAKE) instead of make, since $(MAKE) will always evaluate to the correct executable name (which might not be make).
"Do not do that" is not always the answer, but in this case it is, at least for GNU make.
GNU make parent process has an internal jobserver. If top-level Makefile is run with -j, subprocess makes will talk to the jobserver and read a parallelism level from it, without an explicit -j.
Ongoing coordination with parent's jobserver is much better for core utilization. For example, during the same build with -j6, parent could be running 2 jobs and the child 4 more, next moment both could be running 3 jobs each, then a parent would run 1 and the child 5.