I trying to make parallel make work on our build server. I am running into a very frequent problem here that two instances of make trying to make two different targets, say A and B, nearly simultaneously try to make a target which is required by both, say C.
As both instances try to make C together in different instances, C make fails for either of them since one making of C requires some files to be moved here and there and either one of these instances ends up moving or deleting an already created file.
Is there is common construct that I can use to prevent a re-entry into a makefile if the target is already being made ?
Update:
Ok let me put it this way :
My application requires A.lo and B.lo to be present. These A.lo and B.lo are libraries which also link against C.lo.
So the rules look like
app.d : A.lo B.lo (other lo s)
(do linking)
In some other directory say A (which will house A.lo) :
A.lo : C.lo (other .o s and .lo s)
(do linking)
In some other directory say B (which will house B.lo) :
B.lo : C.lo (other .o s and .lo s)
(do linking)
So in effect while making app.d make forks off two parallel makes for targets A.lo and B.lo.
Upon entering directories A and B make forks off another two threads for target C.lo independently and at times both of these are linking C.lo at the same time, which causes one of them to fail with some weird errors like file not recognized (since it may be written onto by other linker instance)
How should I go about solving this? It is not possible to create A.lo and B.lo without C.lo linked against them.
This may sound a little, well, obvious, but the simplest solution is just to arrange for C to be built explicitly before either A or B. That way when the recursive makes for A and B run, C will already be up-to-date, and neither will bother trying to build it.
If your toplevel makefile looks like this:
all: buildA buildB
buildA:
$(MAKE) -C A
buildB:
$(MAKE) -C B
You can extend it like this:
all: buildA buildB
buildA: buildC
$(MAKE) -C A
buildB: buildC
$(MAKE) -C B
buildC:
$(MAKE) -C C
You didn't give a lot of detail about your specific build, so probably this won't just drop-in and work, but hopefully you get the idea.
I solve this problem using the "mkdir" technique:
SHELL = /bin/bash
all: targetC targetD
targetC: targetA
............
targetD: targetB
............
targetA targetB: source
-mkdir source.lock && ( $(command) < source ; rm -r source.lock )
$(SHELL) -c "while [ -d source.lock ] ; do sleep 0.1 ; done"
I would be happy to see a more elegant solution, though.
Related
I have some software built using parallel multi-level makefiles and I see that when my main Makefile runs two separate targets from a submakefile that have the same dependency, this dependency is run twice simultaneously and an error is created.
Consider the following main Makefile in the project root folder:
TARGETS = t1 t2 t3 t4 t5 t6 t7 t8
.PHONY: all $(TARGETS) clean
all: $(TARGETS)
$(TARGETS):
#echo Making $#
#sleep 1
$(MAKE) -C folder s$#
clean:
#echo Making $#
$(MAKE) -C folder clean
and the sub-makefile folder/Makefile:
SUBTARGETS = st1 st2 st3 st4 st5 st6 st7 st8
$(SUBTARGETS): dep
#echo Making $#
#sleep 1
#touch $#
dep:
#echo Making $#
#sleep 1
#echo bla >> dep
clean:
rm -f $(SUBTARGETS)
rm -f dep
rm -f dep2dump
Then running make -j8 in the root folder will run targets t1...t8 in parallel, which will then run subtargets st1...st8, which all depend on dependency dep. From the shell output and the contents of the dep file (8 lines) it is obvious that the dep rule is run 8 times, as if the 8 implications of folder/Makefile are completely independent.
I thought submakes coordinated when running in parallel and that they would avoid running the same target twice, but it seems this is not the case.
Can anyone suggest a correct way to solve such a case?
If eventually this is an unavoidable weakness of make, what alternative build tools should I look into?
Thanks
EDIT: The answers by MadScientist and Renaud Pacalet are useful but don't exactly solve my problem because they both require that the author of the top-level makefile has knowledge about the internals of the sub-makefile. I have not explained this requirement explicitly in my original post though.
So to give more details, the use case I am trying to solve is that where the source code in path folder/ is a separate project, eg. a collection of utilities st1...st8 where all (or some) of them have a dependency on library dep, internal to the utilities project in folder. Then I want to be able to use this sub-project (as seamlessly as possible) in various master projects, each of them using just a (possible different) subset of the utilities st1...st8. Additionally, the master project may contain many targets t1...t8, each depending on a different subset of st1...st8, as shown in my example above. Targets t1...t8 need to be able to run separately, building only the required dependencies from subproject (so make t1 only builds st1, etc), thus having to build all st1...st8 for each one of t1...t8 is not desired. On the other hand they also need to be able to run in parallel, eg. by running make all.
Ideally I would not want the author of each master makefile to have to know about internals of sub-project, nor have to include in the sub-makefile all the possible combinations of st1...st8 so that each master project can call just ONE of these to avoid the parallel build issue.
So far I have in mind but not tested the following imperfect solutions:
As Renaud suggested, use something like flock to at least ensure that the multiple runs of dep (by separate sub-make instances) won't happen simultaneously. Cons: requires extra tool (flock or similar) to be installed + dep runs multiple times, so extra work is needed to avoid doing the actual compilation over and over again, otherwise just eat the performance cost.
Include the sub-makefile in the master makefile so that everything runs in one make instance. This requires makes the sub-makefile able to work regardless of the path of the master makefile that includes it. No big issue. Cons: merging / including two makefile from different authors can open a can of worms, i.e. variables with same name, etc.
Modify sub-makefile as described in (2) + In the main project create another makefile, eg. utils.make, that contains a rule for the targets of sub-makefile needed and includes the sub-makefile. So utils.make will be (assuming this master project only needs st1, st5 and st7:
utils: st1 st5 st7
include foldes/Makefile
Then the master makefile will have a utils-ext rule as dependency of each of t1...t8 that will be:
utils-ext:
$(MAKE) -f rules.make utils
to build all the utils needed. This keeps the two main makefiles separate but has all utils / subtargets built when building any single one of t1...t8, which is suboptimal.
You could try to move the dep dependency to your top Makefile:
.PHONY: all $(TARGETS) clean dep
all: $(TARGETS)
$(TARGETS): dep
#echo Making $#
#sleep 1
$(MAKE) -C folder s$#
dep:
$(MAKE) -C folder s#
The only decent solution to your problem is to have ONE instance of make build all the sub-directory targets you want. Having the parent make invoke multiple sub-makes in parallel in the same directory, unless every invocation uses a completely disjoint set of targets, is a guaranteed fail situation. So if you have multiple things you want to do in the submake you should collect them all in one invocation of the sub-make and let the sub-make's parallelism handle it for you.
You could do something like this:
TARGETS = t1 t2 t3 t4 t5 t6 t7 t8
.PHONY: all $(TARGETS) clean
all: $(TARGETS)
$(TARGETS): .submake ;
.submake:
$(MAKE) -C folder $(addprefix s,$(MAKECMDGOALS))
Then in the sub-make add this so that when invoked with no arguments it builds everything:
all: $(SUBTARGETS)
Here, if you run make then the sub-make is invoked with no arguments and builds all the things in parallel. If you invoke make t1 t2 then the submake is invoked with the arguments st1 st2.
Alternatively, you can re-architect your makefiles so that you don't use recursive make at all, and one instance of make knows all the different rules and dependency relationships.
I have a project that involves sub-directories with sub-makefiles. I'm aware that I can pass variables from a parent makefile to a sub-makefile through the environment using the export command. Is there a way to pass variables from a sub-makefile to its calling makefile? I.e. can export work in the reverse? I've attempted this with no success. I'm guessing once the sub-make finishes its shell is destroyed along with its environment variables. Is there another standard way of passing variables upward?
The short answer to your question is: no, you can't [directly] do what you want for a recursive build (see below for a non-recursive build).
Make executes a sub-make process as a recipe line like any other command. Its stdout/stderr get printed to the terminal like any other process. In general, a sub-process cannot affect the parent's environment (obviously we're not talking about environment here, but the same principle applies) -- unless you intentionally build something like that into the parent process, but then you'd be using IPC mechanisms to pull it off.
There are a number of ways I could imagine for pulling this off, all of which sound like an awful thing to do. For example you could write to a file and source it with an include directive (note: untested) inside an eval:
some_target:
${MAKE} ${MFLAGS} -f /path/to/makefile
some_other_target : some_target
$(eval include /path/to/new/file)
... though it has to be in a separate target as written above because all $(macro statements) are evaluated before the recipe begins execution, even if the macro is on a later line of the recipe.
gmake v4.x has a new feature that allows you to write out to a file directly from a makefile directive. An example from the documentation:
If the command required each argument to be on a separate line of the
input file, you might write your recipe like this:
program: $(OBJECTS)
$(file >$#.in) $(foreach O,$^,$(file >>$#.in,$O))
$(CMD) $(CMDFLAGS) #$#.in
#rm $#.in
(gnu.org)
... but you'd still need an $(eval include ...) macro in a separate recipe to consume the file contents.
I'm very leery of using $(eval include ...) in a recipe; in a parallel build, the included file can affect make variables and the timing for when the inclusion occurs could be non-deterministic w/respect to other targets being built in parallel.
You'd be much better off finding a more natural solution to your problem. I would start by taking a step back and asking yourself "what problem am I trying to solve, and how have other people solved that problem?" If you aren't finding people trying to solve that problem, there's a good chance it's because they didn't start down a path you're on.
edit You can do what you want for a non-recursive build. For example:
# makefile1
include makefile2
my_tool: ${OBJS}
# makefile2
OBJS := some.o list.o of.o objects.o
... though I caution you to be very careful with this. The build I maintain is extremely large (around 250 makefiles). Each level includes with a statement like the following:
include ${SOME_DIRECTORY}/*/makefile
The danger here is you don't want people in one tree depending on variables from another tree. There are a few spots where for the short term I've had to do something like what you want: sub-makefiles append to a variable, then that variable gets used in the parent makefile. In the long term that's going away because it's brittle/unsafe, but for the time being I've had to use it.
I suggest you read the paper Recursive Make Considered Harmful (if that link doesn't work, just google the name of the paper).
Your directory structure probably looks like this:
my_proj
|-- Makefile
|-- dir1
| `-- Makefile
`-- dir2
`-- Makefile
And what you are doing in your parent Makefile is probably this:
make -C ./dir1
make -C ./dir2
This actually spawns/forks a new child process for every make call.
You are asking for updating the environment of the parent process from its children, but that's not possible by design (1, 2).
You still could work around this by:
using a file as shared memory between two processes (see Brian's answer)
using the child's exit error code as a trigger for different actions [ugly trick]
I think the simplest solution is using standard out from a sub Makefile.
Parent Makefile
VAR := $(shell $(MAKE) -s -C child-directory)
all:
echo $(VAR)
Child Makefile
all:
#echo "MessageToTheParent"
Is there a way how to ask gmake to never run two targets from a set in parallel?
I don't want to use .NOTPARALLEL, because it forces the whole Makefile to be run sequentially, not just the required part.
I could also add dependencies so that one depends on another, but then (apart from being ugly) I'd need to build all of them in order to build the last one, which isn't necessary.
The reason why I need this is that (only a) part of my Makefile invokes ghc --make, which takes care of its dependencies itself. And it's not possible to run it in parallel on two different targets, because if the two targets share some dependency, they can rewrite each other's .o file. (But ghc is fine with being called sequentially.)
Update: To give a specific example. Let's say I need to compile two programs in my Makefile:
prog1 depends on prog1.hs and mylib.hs;
prog2 depends on prog2.hs and mylib.hs.
Now if I invoke ghc --make prog1.hs, it checks its dependencies, compiles both prog1.hs and mylib.hs into their respective object and interface files, and links prog1. The same happens when I call ghc --make prog2.hs. So if they the two commands get to run in parallel, one will overwrite mylib.o of the other one, causing it to fail badly.
However, I need that neither prog1 depends on prog2 nor vice versa, because they should be compilable separately. (In reality they're very large with a lot of modules and requiring to compile them all slows development considerably.)
Hmmm, could do with a bit more information, so this is just a stab in the dark.
Make doesn't really support this, but you can sequential-ise two targets in a couple of ways. First off, a real use for recursive make:
targ1: ; recipe1...
targ2: ; recipe2...
both-targets:
${MAKE} targ1
${MAKE} targ2
So here you can just make -j both-targets and all is fine. Fragile though, because make -j targ1 targ2 still runs in parallel. You can use dependencies instead:
targ1: ; recipe1...
targ2: | targ1 ; recipe2...
Now make -j targ1 targ2 does what you want. Disadvantage? make targ2 will always try to build targ1 first (sequentially). This may (or may not) be a show-stopper for you.
EDIT
Another unsatisfactory strategy is to explicitly look at $MAKECMDGOALS, which lists the targets you specified on the command-line. Still a fragile solution as it is broken when someone uses dependencies inside the Makefile to get things built (a not unreasonable action).
Let's say your makefile contains two independent targets targ1 and targ2. Basically they remain independent until someone specifies on the command-line that they must both be built. In this particular case you break this independence. Consider this snippet:
$(and $(filter targ1,${MAKECMDGOALS)),$(filter targ2,${MAKECMDGOALS}),$(eval targ1: | targ2))
Urk! What's going on here?
Make evaluates the $(and)
It first has to expand $(filter targ1,${MAKECMDGOALS})
Iff targ1 was specified, it goes on to expand $(filter targ2,${MAKECMDGOALS})
Iff targ2 was also specified, it goes on to expand the $(eval), forcing the serialization of targ1 and targ2.
Note that the $(eval) expands to nothing (all its work was done as a side-effect), so that the original $(and) always expands to nothing at all, causing no syntax error.
Ugh!
[Now that I've typed that out, the considerably simpler prog2: | $(filter prog1,${MAKECMDGOALS})
occurs to me. Oh well.]
YMMV and all that.
I'm not familiar with ghc, but the correct solution would be to get the two runs of ghc to use different build folders, then they can happily run in parallel.
Since I got stuck at the same problem, here is another pointer in the direction that make does not provide the functionality you describe:
From the GNU Make Manual:
It is important to be careful when using parallel execution (the -j switch; see Parallel Execution) and archives. If multiple ar commands run at the same time on the same archive file, they will not know about each other and can corrupt the file.
Possibly a future version of make will provide a mechanism to circumvent this problem by serializing all recipes that operate on the same archive file. But for the time being, you must either write your makefiles to avoid this problem in some other way, or not use -j.
What you are attempting, and what I was attempting (using make to insert data in a SQLite3 database) suffers from the exact same problem.
I needed to separate the compilation from other steps (cleaning, building dirs and linking), as I wanted to run the compilation with more core processes and the -j flag.
I managed to solve this, with different makefiles including and calling each other. Only the "compile" make file is running in parallel with all the cores, the rest of the process is syncronous.
I divided my makefile in 3 separate scripts:
settings.mk: contains all the variables and flag definitions
makefile: has all the targets except the compilation one (It has .NOTPARALLEL directive). It calls compile.mk with -j flag
compile.mk: contains only the compile operation (without .NOTPARALLEL)
In settings.mk I have:
CC = g++
DB = gdb
RM = rm
MD = mkdir
CP = cp
MAKE = mingw32-make
BUILD = Debug
DEBUG = true
[... all other variables and flags needed, directories etc ...]
In makefile I have Link and compilation target as these:
include .makefiles/settings.mk
[... OTHER TARGETS (clean, directories etc)]
compilation:
#echo Compilation
#$(MAKE) -f .makefiles/compile.mk --silent -j 8 -Oline
#Link
$(TARGET): compilation
#echo -e Linking $(TARGET)
#$(CC) $(LNKFLAGS) -o $(TARGETDIR)/$(TARGET) $(OBJECTS) $(LIBDIRS) $(LIB)
#Non-File Targets
.PHONY: all prebuild release rebuild clean resources directories run debug
.NOTPARALLEL: all
# include dependency files (*.d) if available
-include $(DEPENDS)
And this is my compile.mk:
include .makefiles/settings.mk
#Defauilt
all: $(OBJECTS)
#Compile
$(BUILDDIR)/%.$(OBJEXT): $(SRCDIR)/%.$(SRCEXT)
#echo -e Compiling: $<
#$(MD) -p $(dir $#)
#$(CC) $(COMFLAGS) $(INCDIRS) -c $< -o $#
#Non-File Targets
.PHONY: all
# include dependency files (*.d) if available
-include $(DEPENDS)
Until now, it's working.
Note that I'm calling compile.mk with -j flag AND -Oline so that parallel processing doesn't mess up with the output.
Any syntax color can be setted in the makefile main script, since the -O flag invalidates escape color codes.
I hope it can help.
I had a similar problem so ended up solving it on the command line, like so:
make target1; make target2
to force it to do the targets sequentially.
I have a makefile which calls multiple other makefiles.
I'd like to pass the -j param along to the other makefile calls.
Something like (make -j8):
all:
make -f libpng_linux.mk -j$(J)
Where $(J) is the value 8 from -j8. I absolutely swear I've done this before but I cannot locate my example.
$(MAKEFLAGS) seems to contain --jobserver-fds=3,4 -j regardless of what -j2 or -j8
Edit: Possible Solution:
Will post this as an answer soon.
It appears one solution to not worry about it. Include -j8 when you call the main makefile. The sub calls to make should look like this:
all:
+make -f libpng_linux.mk -j$(J)
Notice the "+" in front of make. I noticed make tossing a warning when I tried parallel builds: make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
Only certain flags go into $(MAKEFLAGS). -j isn't included because the sub-makes communicate with each other to ensure the appropriate number of jobs are occuring
Also, you should use $(MAKE) instead of make, since $(MAKE) will always evaluate to the correct executable name (which might not be make).
"Do not do that" is not always the answer, but in this case it is, at least for GNU make.
GNU make parent process has an internal jobserver. If top-level Makefile is run with -j, subprocess makes will talk to the jobserver and read a parallelism level from it, without an explicit -j.
Ongoing coordination with parent's jobserver is much better for core utilization. For example, during the same build with -j6, parent could be running 2 jobs and the child 4 more, next moment both could be running 3 jobs each, then a parent would run 1 and the child 5.
I have a project that I am working to release that actually contains 3 subprojects, all of which need to be compiled in one go. My makefile looks roughly like this:
all: a b c
a:
#cd a && make
b:
#cd b && make
c:
#cd c && make
Projects A and B both compile fine but for the 3rd project, it tells me there is nothing to be done although switching to the C directory and running make does in fact compile code.
To be a little more specific: Project C in the example above is actually Mozilla's SpiderMonkey. Whereas A and B are code/makefiles that I have written, C is just a raw copy of SpiderMonkey from the Mozilla website. The actually compile command for it is:
make JS_DIST=/usr JS_THREADSAFE=1 JS_HAS_FILE_OBJECT=1
In my master Makefile, I have:
spidermonkey:
#cd spidermonkey/src && $(MAKE) JS_DIST=/usr JS_THREADSAFE=1 JS_HAS_FILE_OBJECT=1
Running "make spidermonkey" outputs "make: Nothing to be done for `spidermonkey'." How do I get make to run the command?
EDIT:
I've tried adding the following lines to my makefile:
.PHONY: spidermonkey
As well as renaming the spidermonkey rule to sm, but still no change.
EDIT:
My bad! I had spaces when I should have had a tab. doh!
You probably have a file or directory at the toplevel called "spidermonkey". Make thinks this is what its supposed to create, and since it is already there, make stops.
One of the most important rules to follow when writing makefiles is each target should create one file with the same name as the target. In other words, if you have
a:
<some command>
That command should produce a single file called "a".
Rules which do not produce files but are only there as placeholders are called phony targets, and they should be declared like this:
.PHONY: a
Make will then always assume that a has to be remade.
Also, as a general rule do not use "make" to invoke make recursively, use $(MAKE) instead.
EDIT: changed "pseudo" to "phony"
Make only checks for the existance of a file (or directory) named the same as the rule target, and if there is (and it is newer than the dependencies) then from make's point of view there is nothing more to do.
So your problem is that you have a spidermonkey rule (with no dependencies) as well as a directory called spidermonkey, and then make thinks "the target is already made, nothing for me to do". To get make to do what you want, rename the spidermonkey rule (or the directory).
Speaking of recursive make by the way, this is not neccessarily a good idea,
see Recursive Make Considered Harmful.