I use the following code to execute targets in parallel. Now what I want that it will work in parallel but there is some targets that need to be executed at the end, like wait for 3 processes that run in parallel to finish and then execute another 2 processes. How can I do that?
For example here is pack and cleanup to run after module1 and module2 will run in parallel.
NPROCS = $(shell sysctl hw.ncpu | grep -o '[0-9]\+')
MAKEFLAGS += -j$(NPROCS)
all: module1 module2
.PHONY: module1
module1:
#echo "run module 1"
DIR=$(PWD)
#echo $(DIR)
.PHONY: module2
module2:
#echo "run module2”
.PHONY:
pack:
pack $(DIR)
cleanup:
gbt clean $(DIR)
This is what prerequisites, one of make’s great strengths, are for.
Change:
pack:
↓
pack: module1 module2
Which tells make that the pack target has two prerequisites; module1 and module2.
This means make won’t run pack until both module1 and module2 have finished running. Whether they run in parallel won’t make a difference.
If you have a lot of prerequisites, you can put them in a macro, e.g:
modules = module1 module2 …
pack: $(modules)
Related
I have some software built using parallel multi-level makefiles and I see that when my main Makefile runs two separate targets from a submakefile that have the same dependency, this dependency is run twice simultaneously and an error is created.
Consider the following main Makefile in the project root folder:
TARGETS = t1 t2 t3 t4 t5 t6 t7 t8
.PHONY: all $(TARGETS) clean
all: $(TARGETS)
$(TARGETS):
#echo Making $#
#sleep 1
$(MAKE) -C folder s$#
clean:
#echo Making $#
$(MAKE) -C folder clean
and the sub-makefile folder/Makefile:
SUBTARGETS = st1 st2 st3 st4 st5 st6 st7 st8
$(SUBTARGETS): dep
#echo Making $#
#sleep 1
#touch $#
dep:
#echo Making $#
#sleep 1
#echo bla >> dep
clean:
rm -f $(SUBTARGETS)
rm -f dep
rm -f dep2dump
Then running make -j8 in the root folder will run targets t1...t8 in parallel, which will then run subtargets st1...st8, which all depend on dependency dep. From the shell output and the contents of the dep file (8 lines) it is obvious that the dep rule is run 8 times, as if the 8 implications of folder/Makefile are completely independent.
I thought submakes coordinated when running in parallel and that they would avoid running the same target twice, but it seems this is not the case.
Can anyone suggest a correct way to solve such a case?
If eventually this is an unavoidable weakness of make, what alternative build tools should I look into?
Thanks
EDIT: The answers by MadScientist and Renaud Pacalet are useful but don't exactly solve my problem because they both require that the author of the top-level makefile has knowledge about the internals of the sub-makefile. I have not explained this requirement explicitly in my original post though.
So to give more details, the use case I am trying to solve is that where the source code in path folder/ is a separate project, eg. a collection of utilities st1...st8 where all (or some) of them have a dependency on library dep, internal to the utilities project in folder. Then I want to be able to use this sub-project (as seamlessly as possible) in various master projects, each of them using just a (possible different) subset of the utilities st1...st8. Additionally, the master project may contain many targets t1...t8, each depending on a different subset of st1...st8, as shown in my example above. Targets t1...t8 need to be able to run separately, building only the required dependencies from subproject (so make t1 only builds st1, etc), thus having to build all st1...st8 for each one of t1...t8 is not desired. On the other hand they also need to be able to run in parallel, eg. by running make all.
Ideally I would not want the author of each master makefile to have to know about internals of sub-project, nor have to include in the sub-makefile all the possible combinations of st1...st8 so that each master project can call just ONE of these to avoid the parallel build issue.
So far I have in mind but not tested the following imperfect solutions:
As Renaud suggested, use something like flock to at least ensure that the multiple runs of dep (by separate sub-make instances) won't happen simultaneously. Cons: requires extra tool (flock or similar) to be installed + dep runs multiple times, so extra work is needed to avoid doing the actual compilation over and over again, otherwise just eat the performance cost.
Include the sub-makefile in the master makefile so that everything runs in one make instance. This requires makes the sub-makefile able to work regardless of the path of the master makefile that includes it. No big issue. Cons: merging / including two makefile from different authors can open a can of worms, i.e. variables with same name, etc.
Modify sub-makefile as described in (2) + In the main project create another makefile, eg. utils.make, that contains a rule for the targets of sub-makefile needed and includes the sub-makefile. So utils.make will be (assuming this master project only needs st1, st5 and st7:
utils: st1 st5 st7
include foldes/Makefile
Then the master makefile will have a utils-ext rule as dependency of each of t1...t8 that will be:
utils-ext:
$(MAKE) -f rules.make utils
to build all the utils needed. This keeps the two main makefiles separate but has all utils / subtargets built when building any single one of t1...t8, which is suboptimal.
You could try to move the dep dependency to your top Makefile:
.PHONY: all $(TARGETS) clean dep
all: $(TARGETS)
$(TARGETS): dep
#echo Making $#
#sleep 1
$(MAKE) -C folder s$#
dep:
$(MAKE) -C folder s#
The only decent solution to your problem is to have ONE instance of make build all the sub-directory targets you want. Having the parent make invoke multiple sub-makes in parallel in the same directory, unless every invocation uses a completely disjoint set of targets, is a guaranteed fail situation. So if you have multiple things you want to do in the submake you should collect them all in one invocation of the sub-make and let the sub-make's parallelism handle it for you.
You could do something like this:
TARGETS = t1 t2 t3 t4 t5 t6 t7 t8
.PHONY: all $(TARGETS) clean
all: $(TARGETS)
$(TARGETS): .submake ;
.submake:
$(MAKE) -C folder $(addprefix s,$(MAKECMDGOALS))
Then in the sub-make add this so that when invoked with no arguments it builds everything:
all: $(SUBTARGETS)
Here, if you run make then the sub-make is invoked with no arguments and builds all the things in parallel. If you invoke make t1 t2 then the submake is invoked with the arguments st1 st2.
Alternatively, you can re-architect your makefiles so that you don't use recursive make at all, and one instance of make knows all the different rules and dependency relationships.
Most makefiles have a structure such as this:
.PHONY: prebuild
all: $(TARGET)
prebuild: Makefile
$(shell DEPDIR=$(DEPDIR) mkdir -p $(DEPDIR)/../common >/dev/null)
# do other work related to preparing for the object files to be built such as run a script to modify a header file included by $(TARGET).c
$(TARGET): $(TARGET).c prebuild
$(CC) $(CFLAGS) -o $(TARGET) $(TARGET).c
The implicit rules know how to build $(TARGET).o from $(TARGET).c, and doesn't do any work if $(TARGET).o is already newer than $(TARGET).c. This happens when make is run multiple times without changing the source file.
However, building the all target above will seemingly always rerun the $(CC) $(CFLAGS) -o $(TARGET) $(TARGET).c link to link the application and create the application binary. This happens even if that binary already exists and doesn't need to be recreated. In some larger projects, this process can take a long time (tens of seconds), which is sometimes not desirable.
Edit #1: The issue has to do something with an extra phony target that I do want to run ONCE before the object files are built. In my case, I'm running a script which takes Makefile variables and possibly updates a header file that is included in the C file. But, if the Makefile doesn't change, the prebuild target isn't run. However, the $(TARGET) target is still run even if prebuild doesn't do anything (for instance, because Makefile wasn't changed). FYI: because of the structure of my build system, I have prebuild run always because my build system is used for a variety of applications that can dynamically redefine prebuild.
How can this Makefile be restructured to not do the linking again when not necessary?
Edit #2:
Here's a simplified example that seems to illustrate my issue:
Before running, create a new directory and touch a b
.PHONY: prebuild main all
all: main
prebuild: a Makefile
#echo prebuild ran
main: prebuild
#echo main ran
When I run, I get this output:
prebuild ran
main ran
This is what happens no matter how many times I run make, even though the prerequisite a nor Makefile doesn't change. What I expect to happen is prebuild doesn't run (because a and Makefile don't change) and main also doesn't run because prebuild doesn't run. Clearly, I'm misunderstanding something.
The problem is that extra dependency triggering your rebuild.
Try this:
.PHONY: all
OUTPUTDIR=common/
TARGET=finalexe
all: $(OUTPUTDIR)/$(TARGET)
$(OUTPUTDIR)/$(TARGET): $(TARGET).c | $(OUTPUTDIR)
$(CC) $(CFLAGS) -o $# $(TARGET).c
$(OUTPUTDIR):
mkdir -p $#
In this above example, 'finalexe' will be created if A. it doesn't yet exist or B. if finalexe.c was modified. The timestamp on the OUTPUTDIR is not checked.
I've create this simple makefile and I want to run all the process witn make
while do that it run only the first section module1 and not module2,
what am I missing here ?
.PHONY: module1
module1:
#echo "run module 1"
DIR=$(PWD)
#echo $(DIR)
.PHONY: module2
module2:
#echo "run module2"
if I run make module2 the module2 is executed successfully but I want all to be run in the make command and as far as I read in the net this is how it should work, what was wrong here ?
From the documentation...
By default, the goal is the first target in the makefile (not counting
targets that start with a period). Therefore, makefiles are usually
written so that the first target is for compiling the entire program
or programs they describe. If the first rule in the makefile has
several targets, only the first target in the rule becomes the default
goal, not the whole list. You can manage the selection of the default
goal from within your makefile using the .DEFAULT_GOAL variable (see
Other Special Variables).
So you just need to provide a suitable default target -- all is `traditional'...
all: module1 module2
So the complete makefile would be...
all: module1 module2
.PHONY: module1
module1:
#echo "run module 1"
DIR=$(PWD)
#echo $(DIR)
.PHONY: module2
module2:
#echo "run module2"
The question is about parallel making w/ GNU makefile.
Given a folder structure as below, the goal is to deliver a makefile that it supports make release/debug/clean in parallel.
project folder structure:
foo
+-foo1
+-foo2
+-foo3
The makefile may be sth like:
SUBDIR = foo1 foo2 foo3
.PHONY $(SUBDIR) release debug clean
release: $(SUBDIR)
$(SUBDIR):
$(MAKE) -C $# release
debug: $(SUBDIR)
#below is incorrect. $(SUBDIR) is overriden.
$(SUBDIR):
$(MAKE) -C $# debug
..
Sub directory list are set as phony targets for parallel making. but it lost the information of original target (release, debug, clean etc).
One method is to suffix the names for the directories and recover it in commands, but it is weird. another method might be to use variables, but not sure how to work it out.
The questions is:
How to write the rules for directories, that supports parallel making w/ different targets (release/debug/clean)?
Any hints are greatly appreciated.
Setting variables on the command line certainly works. You can also use MAKECMDGOALS (see the GNU make manual):
$(SUBDIR):
$(MAKE) -C $# $(MAKECMDGOALS)
I want to create a Makefile (in a parent dir) to call several other Makefiles (in sub dirs) such that I can build several binaries (one per project sub dir) by invoking just the one parent Makefile.
My research has been hampered by finding loads of stuff on recursive Makefiles, but I think this is where you are trying to build several directories Makefiles into a single binary?
Maybe what I want to do is better handled by a shell script perhaps invoking make in each sub directory in turn, but I thought a Makefile might be a more elegant solution?
any pointers gratefully received
PS using linux and the GNU tool chain
The for loop solution given in the first answer above actually shouldn't be used, as-is. In that method, if one of your sub-makes fails the build will not fail (as it should) but continue on with the other directories. Not only that, but the final result of the build will be whatever the exit code of the last subdirectory make was, so if that succeeded the build succeeds even if some other subdirectory failed. Not good!!
You could fix it by doing something like this:
all:
#for dir in $(SUBDIRS); \
do \
$(MAKE) -C $${dir} $# || exit $$?; \
done
However now you have the opposite problem: if you run "make -k" (continue even if there are errors) then this won't be obeyed in this situation. It'll still exit on failure.
An additional issue with both of the above methods is that they serialize the building of all subdirectories, so if you enable parallel builds (with make's -j option) that will only happen within a single subdirectory, instead of across all subdirectories.
Eregrith and sinsedrix have solutions that are closer to what you want, although FYI you should never, ever use "make" when you are invoking a recursive make invocation. As in johfel's example you should ALWAYS use $(MAKE).
Something like this is what you want:
SUBDIRS = subdir1 subdir1 subdir3 ...
all: $(addprefix all.,$(SUBDIRS))
all.%:
# $(MAKE) -C '$*' '$(basename $#)'
.PHONY: $(addprefix all.,$(SUBDIRS))
And of course you can add more stanzas like this for other targets such as "install" or whatever. There are even more fancy ways to handle building subdirectories with any generic target, but this requires a bit more detail.
If you want to support parallel builds you may need to declare dependencies at this level to avoid parallel builds of directories which depend on each other. For example in the above if you cannot build subdir3 until after both subdir1 and subdir2 are finished (but it's OK for subdir1 and subdir2 to build in parallel) then you can add something like this to your makefile:
all.subdir3 : all.subdir1 all.subdir2
You can call targets in subdirectory makefiles via
all:
$(MAKE) -C subdirectory1 $#
$(MAKE) -C subdirectory2 $#
...
or better
SUBDIRS=subd1 subd2 subd3
all:
#for dir in $(SUBDIRS); \
do \
$(MAKE) -C $${dir} $#; \
done
you should indeed use cmake to generate the Makefile automatically from a given CMakeLists.txt configuration file.
Here's a random link to get you started. Here you can find a simple sample project, including multiple subdirectories, executables, and a shared library.
Each makefile can have several target, it's still true with recursive makefiles, usually it's written:
all: target1 target2 target3
target1 :
make -C subdir
Then make all