There are three source files. Each source files are corresponding targets. For example, the target of a.cpp is a, b.cpp is b, and c.cpp is c.
src/a.cpp
b.cpp
c.cpp
build/
I can build the targets parallelly using -j option.
For example,
cd build
make -j3
target a, b, and c build parallelly.
Is there any way to specify some of targets and parallel build ?
For example
make -j2 a b
Unfortunately, it works sequentially. First, build a and then build b.
I want to build only a and b parallelly.
I tried the following approach.
make a&
make b&
wait
However, the return code of the wait is the last finished waiting target. That means if make a& finished with failure and then make b& successfully finished, the return code of wait is 0.
I want to stop building process if any of make are failure.
Is there any good way ?
Unfortunately, it works sequentially
That is not true, as you could see from writing a small test:
$ cat Makefile
all: one two
one two:
#echo start $#
#sleep 2
#echo stop $#
$ make
start one
stop one
start two
stop two
$ make -j2
start one
start two
stop one
stop two
$ make -j2 one two
start one
start two
stop one
stop two
As you can see even when providing specific targets on the command line, they are run in parallel. If you are not seeing this behavior then there's something about your makefiles that is materially different than what you've described in your question.
Related
I'm using this makefile as an example:
https://gist.github.com/mihaitodor/bfb8e7ad908489fdf3ceb496573f306a
Before compiling/linking I need to do a pre-stage, consisting of cd to a directory and run a script. Without the pre-stage it compiles & links fine.
I think I need to change the all rule:
all : $(TESTS)
I have tried this:
all : cd /bla/bla ./my_script $(TESTS)
and I have tried this:
all :
cd /bla/bla ./my_script
$(TESTS)
but it stops the compile/linking stage.
Given the URL above, where should I insert my pre-stage?
The short answer is you should likely create a new recipe, and make any bottom level target that is dependent on your script running be dependent on it. Something like:
.ran_my_script: $(GTEST_SRCS_)
cd /bla/bla ./my_script
touch $#
test_mihai.o gtest_main.o gtest.o : .ran_my_script
This way it will run your script before it attempts to generate any of the listed targets. It then touches a file $# (.ran_my_script). This will then be newer than any of the .o files meaning the script will not rerun unless someone modifies one of the scripts dependencies (i.e. $(GTEST_SRCS_)). You'll have to figure out what artifcats are actually dependent on your script, and what artifacts your script is dependent on.
Notice that if any of the sources change, then .ran_my_script will be considered out of date, as will anything that depends on it. This means that if you modify any source, it will rebuild all .o files. I'm assuming this is what you want.
Because I'm assuming you're new to makefiles, I will point out two things: first $# resolves to the target name (.ran_my_script in the above example), and second, that the last line of this causes .ran_my_script to be a dependency of test_mihai.o and friends -- but because it does not have any recipes associated with it, this line does not override any other recipes against the same targets specified prior to or later on in the makefile.
As to why what you were doing doesn't work:
all: cd /bla/blah ./my_script $(TESTS)
indicates that the target all is dependent on the targets cd, /bla/bla, ./my_script, and $(TESTS). It will attempt to build all of these before it builds all. Of course, you likely do not have a rule to build cd, etc, so this will fail.
Your next example,
all :
cd /bla/bla ./my_script
$(TESTS)
You create a target all, with two recipes. First, it will attempt to run your script, and then it will attempt to run whatever $(TESTS) resolves to. Notice that $(TESTS) in this case is not a bash command, so this would fail as well.
You might be tempted to do something like this:
all : $(TESTS)
cd /bla/bla ./my_script
but this would cause your script to be run as part of the all target, which runs after everything in $(TESTS) has already been completed.
I'm running make (GNU Make 3.82) with a recursive Makefile.
I'm running make -j2 in order to spawn only 2 processes in parallel.
The internal Makefile is called with $(MAKE).
However, it looks like the internal Makefile (which was started by the main Makefile) spawns processes infinitely as if it was given -j and not -j2.
Trying to verify this, I dumped the environment variables of the child "make":
# pgrep -a make
17218 make -j2
17227 make -C obj_dir/ -f Vf1_package.mk ...
# strings /proc/17227/environ
...
MAKEFLAGS= --jobserver-fds=3,4 -j
...
MAKEFLAGS is not set explicitly anywhere, and -j is only provided in the command line and doesn't appear anywhere in the makefiles. So it seems like "make" itself decided to strip the "2" from the -j argument when composing the MAKEFLAGS for the child "make".
Any idea what could cause "make" to set MAKEFLAGS to -j instead of -j2?
Update 1
I've identified the problem, but I still don't understand why it happens and how to fix that.
The problem is that the job server doesn't work well when the sub-make is running under SCL context.
This is required because I need the sub-make to use specific gcc toolchain.
SCL = scl enable devtoolset-8
...
sub_make:
$(SCL) "$(MAKE) -C $(SUB_MAKE_DIR) ... "
When running like this, the sub-make spawns infinite number of jobs. When SCL is removed, it works as expected.
Why does SCL interfere with make's job server?
How can I solve this? I know I can enable SCL before running the external Makefile, but I would like to control the toolset from within the Makefile.
Update 2
It seems to be related to the fact that SCL changes PATH environment variable. On the new PATH, "make" is newer ("GNU Make 4.2.1").
So it seems that make job server fails if the top level make is running old GNU Make 3.82 and the sub make is running newer 4.2.1 make, maybe something changed between these versions in the way make communicates with the sub-make.
There's nothing wrong here. The top-level make knows how many total jobs there are and it arranges for all the sub-makes to share those jobs through the jobserver (that's what the --jobserver-fds entry in MAKEFLAGS is for). The sub-makes don't need to know how many total jobs there are, they just need to know how to ask if they can start a new job.
In the very old version of GNU make you are using there is no way, from a sub-make, to know what the specific -j number for this build.
Starting with GNU make 4.2, make will add the specific -j value to MAKEFLAGS for informational purposes even though it's still not used.
EDIT
I don't know anything about scl or how it works. But, the GNU make jobserver works by sharing file descriptors across all the sub-makes. If this scl tool is interfering with that, say by forcing all file descriptors to be closed, or running the sub-make inside a docker image where obviously it can't access these shared file descriptors, or some similar thing, then it clearly cannot work with the jobserver feature and you'll have to run the entire make inside the scl.
An option is to not put the -j on the outer make but instead run a single inner make using -j, inside scl.
Can you run make --print-data-base and check if you get proper value of -j.
May be execute a simple test example as shown below where you can test to check if gnu make is able to compile multiple files in parallel to generate object files and is giving correct values of -j:
# .SILENT:
.PHONY:compile objs
TARGET = program.exe
CC=gcc
SOURCES = file_1.c file_2.c file_3.c
OBJ_FILES:= $(SOURCES:.c=.o)
objs: $(OBJ_FILES)
%.o: %.c
$(CC) $(FLAGS) -c $< -o $#
all: test
# Enable parallel compilation
compile:
make -j ${NUMBER_OF_PROCESSORS} objs
link : compile $(TARGET)
$(TARGET): $(OBJ_FILES)
$(CC) $(FLAGS) $(OBJ_FILES) -o $#
test: link
# Execute test script
echo "Executing test script"
Command to execute : make test
This will help you debug and also check if there is issue of gnu-make or some internal bug or make is unable to run in parallel as it did not find anything. I have use ${NUMBER_OF_PROCESSORS} to use all the available processors, you can change it's value and test different runs as per your need.
EDIT
Unfortunately I am not aware about sc1.If scl is the root cause identified, then option would be run the entire make inside sc1. or maye be would be good to test once by explicitly passing -j2 inside the sc1 as maybe global flags are not getting passed to SC1.
I have a project to compile with a lot of makefiles including a clean command. That's why i always have to start over again if there is an error (because the clean command is called in many Makefiles)
Question:
Is there any possibility to tell make to ignore the clean command? I would have to touch > 100 Makefiles otherwise. I would like make to start on the last error, not compiling all done stuff again
Example Makefile entries:
clean: cleansubdirs $(DIR) $(DIR1)
$(DIR2)
It's possible to redefine the recipe of an explicit target as simple as that:
noclean.mk
clean:;
cleansubdirs:;
# more stuff...
Now run make -f Makefile -f noclean.mk and it will work without actual cleaning files. However, make will issue several warnings about "overriding/ignoring old recipes".
When I run make -j3 to build in parallel, I get
warning: -jN forced in submake: disabling jobserver mode.
In the documentation I found the warning is emitted
if make detects error conditions related to parallel processing on
systems where sub-makes can communicate.
What are these error conditions? What can I do to heal them or suppress the error message?
The makefile is generated from CMake, so I cannot (=I don't want to) edit the makefile.
Usually this message occurres if you call make from your Makefile not by the variable $(MAKE)
Example:
Change
foo:
cd foo/ && make foo
to
foo:
cd foo && $(MAKE) foo
You call make -j3 for you top-level Makefile, which means that up to 3 targets can be built simultaneously (make uses 3 threads to build). Your top-level Makefile (generated by CMake) run another make (which is called sub-make). And in this invocation another -jN option is used. So instead of using common jobserver for both makes, sub-make uses another N threads to build its targets. So up to 3 + N threads in total are used for building. So this is what this warning about.
To enable common jobserver for parent make and sub-make, there are 2 requirements:
Explicit -jN option mustn't be passed to sub-make. Call sub-make without this argument.
Sub-make must be called using $(MAKE) expression (not just plain make). Alternatively the + prefix can be added before make command. See this answer for details.
If both requirements are satisfied, then, when you run make -jN for you top-level Makefile, parent make and child make use N threads in common for building.
This is related to multi-threaded make feature. It happens when there is yet another multi-threaded make inside a make process that is already multi-threaded. As there is second level of multi-threaded make, the internal synchronization is not available.
It is possible to fix this by -j parameter.
make -j 1
The code above will avoid multiple thread and it will remove the warnings.
Again, we can use -j parameter in the makefile as well.
target:
$(MAKE) -j 1 -C ${SUBDIR} $#
The above rule will not spawn threads for submake and it will avoid the warning.
With newer versions of GNU make (4.0) and CMake (2.8.6 and 3.3) I can no longer reproduce the warning. One of these got fixed in the meantime.
I have a tricky issue with gmake, when I build from the parent directory, something is different and the make does not build all the .o(s) it needs and fails, but if I cd to the directory and do a make it builds them fine.
How can I get GNUmake to tell me the difference between these two runs? There must be some make variables set in the parent that break the child, but I need help figuring out how to track them down.
If running make from the parent directory fails to build foo.o, then try make foo.o. If that fails then try running make -n foo.o in both directories (to print the commands instead of executing them) to see what it's doing differently. If it succeeds, then it's not even trying to build foo.o when run from the parent directory; make -n may shed some light, and as a last resort make -d will give you a torrent of information about the decision process, why it's doing what it's doing.
Here's a handy trick to see the value of variables. Put this rule in your makefile:
show_%:
#echo $# is $($*)
Now you can run make show_FOO and it will tell you the value of the variable FOO.
Finally, are you sure you know where you build your .o files? Make is very good at using things there to build files here, but not the other way around, so it can lose track of intermediate files if you're not careful.