Just out of curiosity, what is the order of executing targets in a makefile with
${OBJ_DIR}/%.o: ${SRC_DIR}/%.cpp
I noticed it is not lexicographic (like ls -l).
Is it just random?
They are built in the order in which make walks the prerequisite graph.
In the simple case where you don't have parallel jobs (no -j option), then if you have a target like:
prog: foo.o bar.o. baz.o
make will first try to build foo.o, then bar.o, then baz.o, then finally prog.
If you do enable parallel jobs, then make will still try to start builds in the same order but because some builds finish faster than others, you may get different targets building at the same time.
Related
I have a makefile with many independent targets, each of which depends on one intermediate target. Here is an example simplified makefile:
.PHONY: %.world %.hello
all: 0.world 1.world 2.world 3.world 4.world 5.world 6.world
%.world: %.hello
#echo $#
%.hello:
#echo $#
.INTERMEDIATE: %.hello
When I run make serially, it processes in the following order:
0.hello
0.world
1.hello
1.world
2.hello
2.world
3.hello
3.world
4.hello
4.world
5.hello
5.world
6.hello
6.world
But when I run with -j2, it runs all the intermediate files first, then the final ones:
0.hello
1.hello
2.hello
3.hello
4.hello
5.hello
6.hello
0.world
1.world
2.world
3.world
4.world
5.world
6.world
Both orderings are correct, but when processing many 100's of targets, I'd rather have the final files (*.world) sooner rather than waiting for all the intermediate files. Is there any way to tweak the execution ordering, such that it would execute more like this:
0.hello
1.hello
0.world
1.world
2.hello
3.hello
2.world
3.world
4.hello
5.hello
4.world
5.world
6.hello
6.world
I know I could artificially enforce any generic ordering with dependencies, but I need a more general solution since I'm working with pattern rules and 100's of auto-generated targets.
No, there's no way you can change this behavior. Make walks the graph in one direction and it doesn't start over until it's done. Even if it sees that some prerequisite has finished building it will keep going through the graph until it's done, before restarting at the beginning and building things which are now ready to be built.
I'm having a weird parallel build problem. My makefile looks something like this (not the actual Makefile, just to illustrate the point, so don't hate):
parser.c parser.h : parser.y
lexer.c : lexer.l parser.h
app : lexer.o parser.o
Because both the parser.c and parser.h file both depend on parser.y, bison eventually gets invoked twice, once on behalf of lexer.c (which includes parser.h and therefore depends on it), and once on behalf of app (which depends on parser.o, which depends on parser.c).
When I make with -d to see the debugging output, I see that bison get invoked twice in two different threads, relatively close together. I'm worried that when all the stars are aligned just right, I could have a race condition and corrupt bison output. I tried a few things with the dependencies in an attempt to force them to be serial, but I just starting getting stranger and stranger make behaviour.
So, the million dollar question is this: Is there a more sane way around the fact that when a program like bison actually outputs two files, satisfying two dependencies, and therefore when built with -j might be invoked from two different unsynced threads?
Thanks.
You should use a pattern rule; in pattern rules multiple targets mean that one invocation of the recipe generates both targets:
%.c %.h : %.y
...
Is there a way how to ask gmake to never run two targets from a set in parallel?
I don't want to use .NOTPARALLEL, because it forces the whole Makefile to be run sequentially, not just the required part.
I could also add dependencies so that one depends on another, but then (apart from being ugly) I'd need to build all of them in order to build the last one, which isn't necessary.
The reason why I need this is that (only a) part of my Makefile invokes ghc --make, which takes care of its dependencies itself. And it's not possible to run it in parallel on two different targets, because if the two targets share some dependency, they can rewrite each other's .o file. (But ghc is fine with being called sequentially.)
Update: To give a specific example. Let's say I need to compile two programs in my Makefile:
prog1 depends on prog1.hs and mylib.hs;
prog2 depends on prog2.hs and mylib.hs.
Now if I invoke ghc --make prog1.hs, it checks its dependencies, compiles both prog1.hs and mylib.hs into their respective object and interface files, and links prog1. The same happens when I call ghc --make prog2.hs. So if they the two commands get to run in parallel, one will overwrite mylib.o of the other one, causing it to fail badly.
However, I need that neither prog1 depends on prog2 nor vice versa, because they should be compilable separately. (In reality they're very large with a lot of modules and requiring to compile them all slows development considerably.)
Hmmm, could do with a bit more information, so this is just a stab in the dark.
Make doesn't really support this, but you can sequential-ise two targets in a couple of ways. First off, a real use for recursive make:
targ1: ; recipe1...
targ2: ; recipe2...
both-targets:
${MAKE} targ1
${MAKE} targ2
So here you can just make -j both-targets and all is fine. Fragile though, because make -j targ1 targ2 still runs in parallel. You can use dependencies instead:
targ1: ; recipe1...
targ2: | targ1 ; recipe2...
Now make -j targ1 targ2 does what you want. Disadvantage? make targ2 will always try to build targ1 first (sequentially). This may (or may not) be a show-stopper for you.
EDIT
Another unsatisfactory strategy is to explicitly look at $MAKECMDGOALS, which lists the targets you specified on the command-line. Still a fragile solution as it is broken when someone uses dependencies inside the Makefile to get things built (a not unreasonable action).
Let's say your makefile contains two independent targets targ1 and targ2. Basically they remain independent until someone specifies on the command-line that they must both be built. In this particular case you break this independence. Consider this snippet:
$(and $(filter targ1,${MAKECMDGOALS)),$(filter targ2,${MAKECMDGOALS}),$(eval targ1: | targ2))
Urk! What's going on here?
Make evaluates the $(and)
It first has to expand $(filter targ1,${MAKECMDGOALS})
Iff targ1 was specified, it goes on to expand $(filter targ2,${MAKECMDGOALS})
Iff targ2 was also specified, it goes on to expand the $(eval), forcing the serialization of targ1 and targ2.
Note that the $(eval) expands to nothing (all its work was done as a side-effect), so that the original $(and) always expands to nothing at all, causing no syntax error.
Ugh!
[Now that I've typed that out, the considerably simpler prog2: | $(filter prog1,${MAKECMDGOALS})
occurs to me. Oh well.]
YMMV and all that.
I'm not familiar with ghc, but the correct solution would be to get the two runs of ghc to use different build folders, then they can happily run in parallel.
Since I got stuck at the same problem, here is another pointer in the direction that make does not provide the functionality you describe:
From the GNU Make Manual:
It is important to be careful when using parallel execution (the -j switch; see Parallel Execution) and archives. If multiple ar commands run at the same time on the same archive file, they will not know about each other and can corrupt the file.
Possibly a future version of make will provide a mechanism to circumvent this problem by serializing all recipes that operate on the same archive file. But for the time being, you must either write your makefiles to avoid this problem in some other way, or not use -j.
What you are attempting, and what I was attempting (using make to insert data in a SQLite3 database) suffers from the exact same problem.
I needed to separate the compilation from other steps (cleaning, building dirs and linking), as I wanted to run the compilation with more core processes and the -j flag.
I managed to solve this, with different makefiles including and calling each other. Only the "compile" make file is running in parallel with all the cores, the rest of the process is syncronous.
I divided my makefile in 3 separate scripts:
settings.mk: contains all the variables and flag definitions
makefile: has all the targets except the compilation one (It has .NOTPARALLEL directive). It calls compile.mk with -j flag
compile.mk: contains only the compile operation (without .NOTPARALLEL)
In settings.mk I have:
CC = g++
DB = gdb
RM = rm
MD = mkdir
CP = cp
MAKE = mingw32-make
BUILD = Debug
DEBUG = true
[... all other variables and flags needed, directories etc ...]
In makefile I have Link and compilation target as these:
include .makefiles/settings.mk
[... OTHER TARGETS (clean, directories etc)]
compilation:
#echo Compilation
#$(MAKE) -f .makefiles/compile.mk --silent -j 8 -Oline
#Link
$(TARGET): compilation
#echo -e Linking $(TARGET)
#$(CC) $(LNKFLAGS) -o $(TARGETDIR)/$(TARGET) $(OBJECTS) $(LIBDIRS) $(LIB)
#Non-File Targets
.PHONY: all prebuild release rebuild clean resources directories run debug
.NOTPARALLEL: all
# include dependency files (*.d) if available
-include $(DEPENDS)
And this is my compile.mk:
include .makefiles/settings.mk
#Defauilt
all: $(OBJECTS)
#Compile
$(BUILDDIR)/%.$(OBJEXT): $(SRCDIR)/%.$(SRCEXT)
#echo -e Compiling: $<
#$(MD) -p $(dir $#)
#$(CC) $(COMFLAGS) $(INCDIRS) -c $< -o $#
#Non-File Targets
.PHONY: all
# include dependency files (*.d) if available
-include $(DEPENDS)
Until now, it's working.
Note that I'm calling compile.mk with -j flag AND -Oline so that parallel processing doesn't mess up with the output.
Any syntax color can be setted in the makefile main script, since the -O flag invalidates escape color codes.
I hope it can help.
I had a similar problem so ended up solving it on the command line, like so:
make target1; make target2
to force it to do the targets sequentially.
I have a simple example makefile that shows my problem:
.PHONY: a.out b.out
all: a.out b.out
common:
echo building common
sleep 1
touch common
a.out: common
echo building a.out
b.out: common
echo building b.out
a.out & b.out depend on common, so there can be a race condition (common being generated twice) when doing a parallel build.
I did make -j4 and didn't experience common being generated twice. I even put a sleep statement in the generation of common to make things more deterministic.
So is it safe to say that GNU make properly synchronizes when building common dependencies?
Yes, GNU make's parallelization support handles this properly (it would not be very useful if it didn't!) In general, if your makefile environment completely and correctly declares your dependency graph to make, then make will always build it correctly regardless of the number of parallel jobs.
The trick is that for some kinds of complicated environments, it can be difficult to completely and correctly declare the dependency graph. But for simple situations like above, there's no problem.
I have a makefile which calls multiple other makefiles.
I'd like to pass the -j param along to the other makefile calls.
Something like (make -j8):
all:
make -f libpng_linux.mk -j$(J)
Where $(J) is the value 8 from -j8. I absolutely swear I've done this before but I cannot locate my example.
$(MAKEFLAGS) seems to contain --jobserver-fds=3,4 -j regardless of what -j2 or -j8
Edit: Possible Solution:
Will post this as an answer soon.
It appears one solution to not worry about it. Include -j8 when you call the main makefile. The sub calls to make should look like this:
all:
+make -f libpng_linux.mk -j$(J)
Notice the "+" in front of make. I noticed make tossing a warning when I tried parallel builds: make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
Only certain flags go into $(MAKEFLAGS). -j isn't included because the sub-makes communicate with each other to ensure the appropriate number of jobs are occuring
Also, you should use $(MAKE) instead of make, since $(MAKE) will always evaluate to the correct executable name (which might not be make).
"Do not do that" is not always the answer, but in this case it is, at least for GNU make.
GNU make parent process has an internal jobserver. If top-level Makefile is run with -j, subprocess makes will talk to the jobserver and read a parallelism level from it, without an explicit -j.
Ongoing coordination with parent's jobserver is much better for core utilization. For example, during the same build with -j6, parent could be running 2 jobs and the child 4 more, next moment both could be running 3 jobs each, then a parent would run 1 and the child 5.