Suppose I have a make file, and I have many target MyTarget1,MyTarget2,MyTarget3,...,MyTarget100.
If I want compile all targets using 12 thread, I can simply using make -j12 all.
Now I want compile a subset of all target, suppose MyTarget1, MyTarget2, MyTarget3, MyTarget4.
I know that compile each target one by one must work. In this way, 12 thread work on MyTarget1, wait, work on MyTarget2, wait, ... If MyTarget do not have a high parallism, for example it is a small target like helloworld, some thread's time is wasted. I do not like it for its low parallism.
I want a high parallism solution, like make -j12 all, which 12 thread can work on different target at a certain moment.
How can I implement that?
I want something like
make -j12 MyTarget1,MyTarget2,MyTarget3,MyTarget4
Reference
follow link already given the CMake's solution, now I am wondering can it be implement directly using make.
How can I build multiple targets using cmake --build
Thanks for your time.
This is a limitation of CMake. Generated Makefile is explicitly listed to not run in parallel. For example:
$ cat CMakeLists.txt
project(foo C)
add_custom_target(target1 ALL
COMMAND python3 -c "import time; time.sleep(5)"
VERBATIM
)
add_custom_target(target2 ALL
COMMAND python3 -c "import time; time.sleep(5)"
VERBATIM
)
Relevant parts of generated Makefile are:
$ cat Makefile
...
# Allow only one "make -f Makefile2" at a time, but pass parallelism.
.NOTPARALLEL:
...
# The main all target
all: cmake_check_build_system
$(CMAKE_COMMAND) -E cmake_progress_start /home/raspy/so-62013595/CMakeFiles /home/raspy/so-62013595/CMakeFiles/progress.marks
$(MAKE) -f CMakeFiles/Makefile2 all
$(CMAKE_COMMAND) -E cmake_progress_start /home/raspy/so-62013595/CMakeFiles 0
.PHONY : all
...
# Build rule for target.
target2: cmake_check_build_system
$(MAKE) -f CMakeFiles/Makefile2 target2
.PHONY : target2
...
# Build rule for target.
target1: cmake_check_build_system
$(MAKE) -f CMakeFiles/Makefile2 target1
.PHONY : target1
So as you can see every target is propagated to a sub-makefile, but since this top Makefile is listed as not parallel, it will not allow to build multiple targets at the same time.
$ make -j8 target1 target2 | ts
May 26 15:45:06 Built target target1
May 26 15:45:13 Built target target2 # <--- Built after target1 completed
For arbitrary targets, you might be successful with calling sub-makefile directly:
$ make -j8 -f CMakeFiles/Makefile2 target1 target2 | ts
May 26 15:45:42 Built target target2
May 26 15:45:42 Built target target1 # <--- Built simultaneously with target2
YMMV though.
Related
I have a makefile structured something like this:
all :
compile executable
clean :
rm -f *.o $(EXEC)
I realized that I was consistently running "make clean" followed by "clear" in my terminal before running "make all". I like to have a clean terminal before I try and sift through nasty C++ compilation errors. So I tried to add a 3rd target:
fresh :
rm -f *.o $(EXEC)
clear
make all
This works, however this runs a second instance of make (I believe). Is there a right way to get the same functionality without running a 2nd instance of make?
Actually you are right: it runs another instance of make.
A possible solution would be:
.PHONY : clearscr fresh clean all
all :
compile executable
clean :
rm -f *.o $(EXEC)
fresh : clean clearscr all
clearscr:
clear
By calling make fresh you get first the clean target, then the clearscreen which runs clear and finally all which does the job.
EDIT Aug 4
What happens in the case of parallel builds with make’s -j option?
There's a way of fixing the order. From the make manual, section 4.2:
Occasionally, however, you have a situation where you want to impose a specific ordering on the rules to be invoked without forcing the target to be updated if one of those rules is executed. In that case, you want to define order-only prerequisites. Order-only prerequisites can be specified by placing a pipe symbol (|) in the prerequisites list: any prerequisites to the left of the pipe symbol are normal; any prerequisites to the right are order-only: targets : normal-prerequisites | order-only-prerequisites
The normal prerequisites section may of course be empty. Also, you may still declare multiple lines of prerequisites for the same target: they are appended appropriately. Note that if you declare the same file to be both a normal and an order-only prerequisite, the normal prerequisite takes precedence (since they are a strict superset of the behavior of an order-only prerequisite).
Hence the makefile becomes
.PHONY : clearscr fresh clean all
all :
compile executable
clean :
rm -f *.o $(EXEC)
fresh : | clean clearscr all
clearscr:
clear
EDIT Dec 5
It is not a big deal to run more than one makefile instance since each command inside the task will be a sub-shell anyways. But you can have reusable methods using the call function.
log_success = (echo "\x1B[32m>> $1\x1B[39m")
log_error = (>&2 echo "\x1B[31m>> $1\x1B[39m" && exit 1)
install:
#[ "$(AWS_PROFILE)" ] || $(call log_error, "AWS_PROFILE not set!")
command1 # this line will be a subshell
command2 # this line will be another subshell
#command3 # Use `#` to hide the command line
$(call log_error, "It works, yey!")
uninstall:
#[ "$(AWS_PROFILE)" ] || $(call log_error, "AWS_PROFILE not set!")
....
$(call log_error, "Nuked!")
You already have a sequential solution which could be rewritten as:
fresh:
$(MAKE) clean
clear
$(MAKE) all
This is correct and a very safe approach.
Sequential target execution is possible in GNU make with a proper dependency graph:
fresh: _all
_all: _clear
Recipe for all
_clear: _clean
Recipe for clear
_clean:
Recipe for clean
The above rules define the following dependency graph: fresh <- _all <- _clear <- _clean which guarantees the following recipe execution order: Recipe for clean, Recipe for clear, Recipe for all.
Recipes can be shared with multiple targets using:
target1 target2 target…:
recipe1
Merging your script with the above concepts results in:
all _all :
compile executable
clean _clean :
rm -f *.o $(EXEC)
clear _clear :
clear
fresh: _all
_all: _clear
_clear: _clean
With syntactic sugar using chains.mk from https://github.com/pkoper/mk/ you can write:
all all#fresh :
compile executable
clean clean#fresh :
rm -f *.o $(EXEC)
clear clear#fresh :
clear
#fresh = clean clear all
include chains.mk
fresh: #fresh
Or better:
all: compile
#fresh = clean clear compile
include chains.mk
fresh: #fresh
compile compile#fresh:
compile executable
clear clear#fresh:
clear
clean clean#fresh:
rm -f *.o $(EXEC)
If you removed the make all line from your "fresh" target:
fresh :
rm -f *.o $(EXEC)
clear
You could simply run the command make fresh all, which will execute as make fresh; make all.
Some might consider this as a second instance of make, but it's certainly not a sub-instance of make (a make inside of a make), which is what your attempt seemed to result in.
I have a make target that depends on a variable, which contains both PHONY and real targets.
This target needs to depend only on the real targets in the variable.
How can I test a variable to determine if it is a PHONY or not, so I can filter them out?
(I can test for a file's existence inside the recipe, but I don't want my target to be triggered by execution of any of the PHONY targets.)
Thanks!
There is a way to do it, but I would strongly recommend against it. First of, phony targets can be also file targets. And there is no way to tell a phony file target from a non-phony file target.
It looks like the question implies that the phony targets the author wants to ignore are all non-file targets. In this case see the example below.
.PHONY: phony_target .FORCE
.FORCE:
ALL_TARGETS = phony_target file_target undetermined_target
-include detect_phony.inc
all: final_target
# All done
final_target: $(REAL_TARGETS)
# create $# triggered by $?
#touch $#
ifeq (,$(MAKE_RESTARTS))
# Generate the list of real file targets in make include file
detect_phony.inc: .FORCE
#echo 'REAL_TARGETS = ' `ls $(ALL_TARGETS) 2>/dev/null` > $# |:
endif
file_target:
touch $#
undetermined_target phony_target:
# process $#
clean:
rm -f file_target final_target
Here are the test results:
$make clean
rm -f file_target final_target
$ make
# create final_target triggered by
# All done
$ touch file_target
$ make
# create final_target triggered by file_target
# All done
$ make
# All done
As you can see it only triggers the final target when the file target is updated.
Before you criticize - Here are the flaws of this implementation:
make is always called twice, updating the generated detect_phony.inc include file at every run
if detect_phony.inc gets corrupted somehow, make execution will be locked by syntax errors, until you manually delete it.
it can't handle phony file targets as I mentioned before
if another generated include is added in this makefile that requires another restart before detect_phony.inc this functionality will break.
So it this method is hacky and has several gotchas. I would not use it in production environment. I would insist on changing the top level Makefile first.
Hopefully this is a very simple question. I have a makefile pattern rule that looks like this:
%.so : %.f %.pyf
f2py -c -L${LAPACK_DIR} ${GRASPLIBS} -m $* $^ ${SOURCES} --opt='-02' --f77flags='-fcray-pointer' >> silent.txt
I want the makefile to build a number of .so files, so I tried to get it to build two files (radgrd_py.so and lodiso_py.so) by doing this:
radgrd_py.so lodiso_py.so:
%.so : %.f %.pyf
f2py -c -L${LAPACK_DIR} ${GRASPLIBS} -m $* $^ ${SOURCES} --opt='-02' --f77flags='-fcray-pointer' >> silent.txt
and then tried this:
radgrd_py.so:
lodiso_py.so:
%.so : %.f %.pyf
f2py -c -L${LAPACK_DIR} ${GRASPLIBS} -m $* $^ ${SOURCES} --opt='-02' --f77flags='-fcray-pointer' >> silent.txt
But in each case, it only builds the first target that I specify. If I run 'make radgrd_py.so' it works fine, I'm just not sure how to specify a list of files that need to be built so I can just run 'make'.
The usual trick is to add a 'dummy' target as the first that depends on all targets you want to build when running a plain make:
all: radgrd_py.so lodiso_py.so
It is a convention to call this target 'all' or 'default'. For extra correctness, let make know that this is not a real file by adding this line to your Makefile:
.PHONY: all
Best way is to add:
.PHONY: all
.DEFAULT: all
all: radgrd_py.so lodiso_py.so
Explanations:
make uses the first target appearing when no .DEFAULT is specified.
.PHONY informs make that the targets (a coma-separated list, in fact) don't create any file or folder.
all: as proposed by schot
I encountered a software that had three makefiles, one in the root directory and two in immediate subdirectories. Experimenting a bit it turned out that there is a good order of them:
$ make -C wordn all
$ make -C buses all
$ make all
from the root directory. What is an elegant way to rewrite the main makefile so that this sequence happens?
In the target all of the main Makefile you can add:
all:
$(MAKE) -C wordn all
$(MAKE) -C buses all
...
This will execute the submake before doing the all commands.
You can also create a dependency of the main all target:
all: suball
...
suball:
$(MAKE) -C wordn all
$(MAKE) -C buses all
.PHONY: suball
The suball target must be a PHONY target as it is a virtual target
I have a complex makefile with a lot of recipes. I would like run them with no parallel execution except for the generation of my objects files. I noticed that the .NOTPARALLEL target cannot take any prerequisites otherwise it would have been much easier to solve my issue.
My first guess was to use a nonexistent target named ".PARALLEL" with which I would have mentioned the objects files as dependancies like this:
SRC=$(wildcard *.c)
OBJ=$(SRC:.c=.o)
.PARALLEL: $(OBJ)
%.o: %.c
gcc –c –o$# $< -M
a.out: $(OBJ)
gcc –o$# $^
A more functional solution I have found is to use an intermediate target. However, since MyObjects has no dependancies, make will always call MyObjects and recreate a.out.
%.o: %.c
$(CC) –c –o$# $< -M
MyObjects:
$(MAKE) -j $(OBJ)
a.out: MyObjects
$(CC) –o$# $(OBJ)
To avoid this I've found nothing better than using dummy files. I wrote this example to illustrate it:
NAMES = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
SRC = $(addsuffix .c, $(NAMES))
OBJ = $(patsubst %.c,%.o,$(SRC))
DUM = $(addsuffix .dummy,$(OBJ))
all: a.out
$(SRC):%.c:
touch $#
$(OBJ):%.o: %.c
cp $< $#
touch $(addsuffix .dummy, $#)
$(DUM):
$(MAKE) -j8 $(OBJ)
a.out: $(DUM) $(OBJ)
zip $# $(OBJ)
clean:
-rm *.o
-rm *.out
-rm *.c
-rm *.dummy
I'm sure this is not the best solution I can get. I would be glad to get some help.
P.S. MadScientist, thank you for your advices.
This is really not right:
MyObjects: $(OBJ)
$(MAKE) -j $(OBJ)
This means that before make tries to build the MyObjects target, it will first try to update all the $(OBJ) files. Once that's all done, then it will try to build the MyObjects target by recursively invoking make to rebuild them again. Obviously that's not what you want. Plus you're using -j which is basically "infinitely parallel" and is likely (if you have enough object files) to bring your system to its knees.
You want something like this:
MyObjects:
$(MAKE) -j5 $(OBJ)
As for your second question about trying to rebuild targets, there's no way we can help without some kind of specific example. Typically this happens because your rules are written incorrectly, and they don't actually update the target you told make they would. So for example, you have a target recipe_a but the rule for recipe_a updates some other target, not recipe_a.
I'll add a few notes based on your second question. Probably if you don't get it after this you should take this off of StackOverflow and ask on the help-make#gnu.org mailing list, or else consider breaking this up and asking several specific StackOverflow questions.
First, why you see make[1]: '15.o' is up to date. for every file in your recursive make: because make always prints that message for every target on the command line, so if you run make 1.o 2.o 3.o ... (doesn't matter whether you use -j or not or what value of -j you use) you'll get that message for every target which doesn't need to be rebuilt. Just as if you ran that same make command from the command line yourself.
Second, why you don't get a.out is up to date, because a.out is NOT up to date. It depends on the build target, and the file build doesn't exist, and thus it's out of date, and so it must be rebuilt every time. And that means anything that depends on the build target, like a.out, must be rebuilt every time. Which explains why it always re-runs the zip command.
Third, the behavior with all.c is because if you create a pattern rule like %.c: with no prerequisites, that tells make that it can create ANY file with a .c extension by running that command. Well, one of the targets you asked make to build is the all target. Since you didn't declare that as a .PHONY target, make tries to build it. Normally that attempt fails because make can't find any rules that know how to build all so nothing happens, but after you tell make how to build a .c file out of nothing (no prerequisites), then when make wants to build all it looks in its internal database of predefined rules and sees a pattern rule % : %.c, which tells make how to build an executable from a source file with the same name (on UNIX systems executables don't have any suffix like .exe: they're just make or cc etc.) So, make tries to run those rules and they fail.
For any target which you don't expect to actually be created, like all, clean, etc. you should declare them to be .PHONY so make won't try to build them.
As for your problem. I think the simplest thing to do is push the entire build of the zip file down into the recursive make, rather than trying to build the objects only in the recursive make. Something like this:
NAMES = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
SRC = $(addsuffix .c,$(NAMES))
OBJ = $(patsubst %.c,%.o,$(SRC))
all: recurse
recurse: non-parallel-targets
$(MAKE) -j8 a.out PARALLEL=true
ifneq($(PARALLEL),true)
.NOTPARALLEL:
endif
%.o: %.c
cp $< $#
a.out: $(OBJ)
zip $# $(OBJ)
init: $(SRC)
clean:
-rm *.o
-rm *.out
.PHONY: all clean init