I have a parallel test suite (perl prove -j XXX). If a user types make -j 8 all, I'd like the test suite to be run with the same parameter: prove -j XXX t. If not, then I'd like it to be run single-threaded. Since I know the test suite is top level and depends on all of the binary targets, I'd like to, very simply, pass-through the user's specified parallel argument.
Is there something in gnu make that allows for getting the command-line arguments used to run make? Or will the user have to do something like: make -j 8 PLL=8 all.
According to the manual -j is passed down to some sub-makes (via MAKEFLAGS) in some circumstances, but not most and is not (apparently) present in MAKEFLAGS in the toplevel make. So I don't see any way to get this information unfortunately.
You could, however, have the user pass the value only through a variable assignment PLL=8 and add it to MAKEFLAGS yourself manually (MAKEFLAGS += -j$(PLL)) with appropriate guarding for only doing that in the toplevel makefile and only when some other value of -j isn't in MAKEFLAGS (in case that can actually happen somehow). I believe this will work correctly as far as make jobserver behaviour is concerned.
Related
I'm running make (GNU Make 3.82) with a recursive Makefile.
I'm running make -j2 in order to spawn only 2 processes in parallel.
The internal Makefile is called with $(MAKE).
However, it looks like the internal Makefile (which was started by the main Makefile) spawns processes infinitely as if it was given -j and not -j2.
Trying to verify this, I dumped the environment variables of the child "make":
# pgrep -a make
17218 make -j2
17227 make -C obj_dir/ -f Vf1_package.mk ...
# strings /proc/17227/environ
...
MAKEFLAGS= --jobserver-fds=3,4 -j
...
MAKEFLAGS is not set explicitly anywhere, and -j is only provided in the command line and doesn't appear anywhere in the makefiles. So it seems like "make" itself decided to strip the "2" from the -j argument when composing the MAKEFLAGS for the child "make".
Any idea what could cause "make" to set MAKEFLAGS to -j instead of -j2?
Update 1
I've identified the problem, but I still don't understand why it happens and how to fix that.
The problem is that the job server doesn't work well when the sub-make is running under SCL context.
This is required because I need the sub-make to use specific gcc toolchain.
SCL = scl enable devtoolset-8
...
sub_make:
$(SCL) "$(MAKE) -C $(SUB_MAKE_DIR) ... "
When running like this, the sub-make spawns infinite number of jobs. When SCL is removed, it works as expected.
Why does SCL interfere with make's job server?
How can I solve this? I know I can enable SCL before running the external Makefile, but I would like to control the toolset from within the Makefile.
Update 2
It seems to be related to the fact that SCL changes PATH environment variable. On the new PATH, "make" is newer ("GNU Make 4.2.1").
So it seems that make job server fails if the top level make is running old GNU Make 3.82 and the sub make is running newer 4.2.1 make, maybe something changed between these versions in the way make communicates with the sub-make.
There's nothing wrong here. The top-level make knows how many total jobs there are and it arranges for all the sub-makes to share those jobs through the jobserver (that's what the --jobserver-fds entry in MAKEFLAGS is for). The sub-makes don't need to know how many total jobs there are, they just need to know how to ask if they can start a new job.
In the very old version of GNU make you are using there is no way, from a sub-make, to know what the specific -j number for this build.
Starting with GNU make 4.2, make will add the specific -j value to MAKEFLAGS for informational purposes even though it's still not used.
EDIT
I don't know anything about scl or how it works. But, the GNU make jobserver works by sharing file descriptors across all the sub-makes. If this scl tool is interfering with that, say by forcing all file descriptors to be closed, or running the sub-make inside a docker image where obviously it can't access these shared file descriptors, or some similar thing, then it clearly cannot work with the jobserver feature and you'll have to run the entire make inside the scl.
An option is to not put the -j on the outer make but instead run a single inner make using -j, inside scl.
Can you run make --print-data-base and check if you get proper value of -j.
May be execute a simple test example as shown below where you can test to check if gnu make is able to compile multiple files in parallel to generate object files and is giving correct values of -j:
# .SILENT:
.PHONY:compile objs
TARGET = program.exe
CC=gcc
SOURCES = file_1.c file_2.c file_3.c
OBJ_FILES:= $(SOURCES:.c=.o)
objs: $(OBJ_FILES)
%.o: %.c
$(CC) $(FLAGS) -c $< -o $#
all: test
# Enable parallel compilation
compile:
make -j ${NUMBER_OF_PROCESSORS} objs
link : compile $(TARGET)
$(TARGET): $(OBJ_FILES)
$(CC) $(FLAGS) $(OBJ_FILES) -o $#
test: link
# Execute test script
echo "Executing test script"
Command to execute : make test
This will help you debug and also check if there is issue of gnu-make or some internal bug or make is unable to run in parallel as it did not find anything. I have use ${NUMBER_OF_PROCESSORS} to use all the available processors, you can change it's value and test different runs as per your need.
EDIT
Unfortunately I am not aware about sc1.If scl is the root cause identified, then option would be run the entire make inside sc1. or maye be would be good to test once by explicitly passing -j2 inside the sc1 as maybe global flags are not getting passed to SC1.
I have a Makefile, which fails at some point, with a git error. How can I view the git command that causes the whole make process to fail? More precisely, I am looking for a list of commands (including the ones that start with #) that I can run on an identical setup, to achieve the same effect as what make does.
I know for a script, instead of #! /bin/bash you would add a flag -x to it, and that would display all the commands before their execution. How do I do the same thing for make?
I am looking for a list of commands (including the ones that start with #) that I can run on an identical setup, to achieve the same effect as what make does.
By default, make echoes all recipe commands it runs, except those prefixed with #. The POSIX specifications for make do not describe a way to override that effect of # (but see below). It is conceivable that your make has an extension for that, but the make implementations you are most likely to be using (GNU make or BSD make, since you seem to assume that your standard shell is bash) do not.
Additionally, in POSIX-conforming make implementations, including the two mentioned above, the special target .SILENT can be used to suppress echoing the commands of some or all targets, and the -s command-line option can be used to suppress echoing for all targets.
You can print recipe commands prefixed with # if you run make with the -n (no-op) flag. That will print the commands for out-of-date targets without running them, except that those prefixed with a + are run even in no-op mode. Commands prefixed with # are included among those printed. Under some circumstances, the fact that most commands are not actually run in this mode can affect the output, but all the cases I can think of at the moment involve recursive make, and I think they are fairly unlikely.
POSIX seems to indicate that -n does not override -s or .SILENT, so if you have to deal with those then you may have no alternative but to modify your makefile. If you happen to be using GNU make, however, you will find that -n does override .SILENT and -s in that implementation. The same may be true of other makes.
GNU make version 3.81, when I tab make -j20, there are so many errors and warnings. But when tab "make", then it will seccess ? Is there any relation with include "../build/Generic.mak"?
or when make -j20 will failed?
Sounds like your Makefile is simply not thread safe. That's why it doesn't work with -j parameter. The -j option tells make to run in parallel.
Unfortunately, the Makefile language is too complex to know exactly what is wrong with your build with the information you've given. It's a rather involved subject but you could try to read (a lot) about GNU make and automake to find some insight.
When I run make -j3 to build in parallel, I get
warning: -jN forced in submake: disabling jobserver mode.
In the documentation I found the warning is emitted
if make detects error conditions related to parallel processing on
systems where sub-makes can communicate.
What are these error conditions? What can I do to heal them or suppress the error message?
The makefile is generated from CMake, so I cannot (=I don't want to) edit the makefile.
Usually this message occurres if you call make from your Makefile not by the variable $(MAKE)
Example:
Change
foo:
cd foo/ && make foo
to
foo:
cd foo && $(MAKE) foo
You call make -j3 for you top-level Makefile, which means that up to 3 targets can be built simultaneously (make uses 3 threads to build). Your top-level Makefile (generated by CMake) run another make (which is called sub-make). And in this invocation another -jN option is used. So instead of using common jobserver for both makes, sub-make uses another N threads to build its targets. So up to 3 + N threads in total are used for building. So this is what this warning about.
To enable common jobserver for parent make and sub-make, there are 2 requirements:
Explicit -jN option mustn't be passed to sub-make. Call sub-make without this argument.
Sub-make must be called using $(MAKE) expression (not just plain make). Alternatively the + prefix can be added before make command. See this answer for details.
If both requirements are satisfied, then, when you run make -jN for you top-level Makefile, parent make and child make use N threads in common for building.
This is related to multi-threaded make feature. It happens when there is yet another multi-threaded make inside a make process that is already multi-threaded. As there is second level of multi-threaded make, the internal synchronization is not available.
It is possible to fix this by -j parameter.
make -j 1
The code above will avoid multiple thread and it will remove the warnings.
Again, we can use -j parameter in the makefile as well.
target:
$(MAKE) -j 1 -C ${SUBDIR} $#
The above rule will not spawn threads for submake and it will avoid the warning.
With newer versions of GNU make (4.0) and CMake (2.8.6 and 3.3) I can no longer reproduce the warning. One of these got fixed in the meantime.
I have a makefile which looks roughly like this
debug:
make -C build-debug
release:
make -C build-release
Now, I run "main" make
make -j4 debug
How do I forward -j4 to the collateral make? Note that I don't want to hardcode it, I want to forward whatever was passed to main make.
From the manual,:
If you set [-j] to some numeric value ‘N’ and your operating system
supports it (most any UNIX system will; others typically won't), the
parent make and all the sub-makes will communicate to ensure that
there are only ‘N’ jobs running at the same time between them all...
If your operating system doesn't support the above communication, then
‘-j 1’ is always put into MAKEFLAGS instead of the value you
specified.
If you really want to override this behavior, it's probably not too difficult...