I have a makefile in which one of the targets calls another makefile elsewhere.
target:
$(MAKE) -C /some/dir other_target
I am running my main makefile using the -j option at command line to enable parallel jobs. However, the other makefile is perhaps not written properly (it is auto-generated from a tool I am using) because it fails miserably when running with -j.
How can I stop the -j option from trickling down to the $(MAKE) command inside my main makefile?
Use -j 1 option for nested make calls.
Related
I am maintaining a cross-platform project (Linux/Windows/32/64) with hundreds of C and C ++ files. For this I am using a single makefile using GNU Make.
My problem happens when I compile in Windows because the Visual Studio (cl) compiler forces me to set an enviroment by calling vcvarsall.bat [platform_type] and this process is executed every time I compile a file with which the total process is extremely expensive in time.
Partial makefile
# C object file generation rule
../int/$(PROJECT)/%.o: ../%.c
ifeq ($(OS),Windows_NT)
#vcvarsall.bat x86 & cl $(CFLAGS) -c $(subst /,\,$<)
else
#gcc $(CFLAGS) -c $<
endif
The sentence #vcvarsall.bat x86 is executed whenever each file is compiled. Is there a way where this statement is only executed once and that all successive shells inherit this ?
Thanks !
This is one of the truly annoying things about development on Windows, for sure.
It's not possible for a shell to "pass on" its values after it exits, to the next shell. It's also not possible for make to start a single shell then run all the rules' recipes in that one shell.
All you can do is set the values once in the parent shell, then invoke make from there. The simplest way to do this is have a .bat file that does the proper thing, then invokes make. You can have a similarly-named shell script on POSIX systems that doesn't do anything except invoke make.
If you really don't want to do that, then the only option is to use recursive make invocations where the top-level make will run the vcvarsall script then invoke a sub-make which will compile all the things.
I'm running make (GNU Make 3.82) with a recursive Makefile.
I'm running make -j2 in order to spawn only 2 processes in parallel.
The internal Makefile is called with $(MAKE).
However, it looks like the internal Makefile (which was started by the main Makefile) spawns processes infinitely as if it was given -j and not -j2.
Trying to verify this, I dumped the environment variables of the child "make":
# pgrep -a make
17218 make -j2
17227 make -C obj_dir/ -f Vf1_package.mk ...
# strings /proc/17227/environ
...
MAKEFLAGS= --jobserver-fds=3,4 -j
...
MAKEFLAGS is not set explicitly anywhere, and -j is only provided in the command line and doesn't appear anywhere in the makefiles. So it seems like "make" itself decided to strip the "2" from the -j argument when composing the MAKEFLAGS for the child "make".
Any idea what could cause "make" to set MAKEFLAGS to -j instead of -j2?
Update 1
I've identified the problem, but I still don't understand why it happens and how to fix that.
The problem is that the job server doesn't work well when the sub-make is running under SCL context.
This is required because I need the sub-make to use specific gcc toolchain.
SCL = scl enable devtoolset-8
...
sub_make:
$(SCL) "$(MAKE) -C $(SUB_MAKE_DIR) ... "
When running like this, the sub-make spawns infinite number of jobs. When SCL is removed, it works as expected.
Why does SCL interfere with make's job server?
How can I solve this? I know I can enable SCL before running the external Makefile, but I would like to control the toolset from within the Makefile.
Update 2
It seems to be related to the fact that SCL changes PATH environment variable. On the new PATH, "make" is newer ("GNU Make 4.2.1").
So it seems that make job server fails if the top level make is running old GNU Make 3.82 and the sub make is running newer 4.2.1 make, maybe something changed between these versions in the way make communicates with the sub-make.
There's nothing wrong here. The top-level make knows how many total jobs there are and it arranges for all the sub-makes to share those jobs through the jobserver (that's what the --jobserver-fds entry in MAKEFLAGS is for). The sub-makes don't need to know how many total jobs there are, they just need to know how to ask if they can start a new job.
In the very old version of GNU make you are using there is no way, from a sub-make, to know what the specific -j number for this build.
Starting with GNU make 4.2, make will add the specific -j value to MAKEFLAGS for informational purposes even though it's still not used.
EDIT
I don't know anything about scl or how it works. But, the GNU make jobserver works by sharing file descriptors across all the sub-makes. If this scl tool is interfering with that, say by forcing all file descriptors to be closed, or running the sub-make inside a docker image where obviously it can't access these shared file descriptors, or some similar thing, then it clearly cannot work with the jobserver feature and you'll have to run the entire make inside the scl.
An option is to not put the -j on the outer make but instead run a single inner make using -j, inside scl.
Can you run make --print-data-base and check if you get proper value of -j.
May be execute a simple test example as shown below where you can test to check if gnu make is able to compile multiple files in parallel to generate object files and is giving correct values of -j:
# .SILENT:
.PHONY:compile objs
TARGET = program.exe
CC=gcc
SOURCES = file_1.c file_2.c file_3.c
OBJ_FILES:= $(SOURCES:.c=.o)
objs: $(OBJ_FILES)
%.o: %.c
$(CC) $(FLAGS) -c $< -o $#
all: test
# Enable parallel compilation
compile:
make -j ${NUMBER_OF_PROCESSORS} objs
link : compile $(TARGET)
$(TARGET): $(OBJ_FILES)
$(CC) $(FLAGS) $(OBJ_FILES) -o $#
test: link
# Execute test script
echo "Executing test script"
Command to execute : make test
This will help you debug and also check if there is issue of gnu-make or some internal bug or make is unable to run in parallel as it did not find anything. I have use ${NUMBER_OF_PROCESSORS} to use all the available processors, you can change it's value and test different runs as per your need.
EDIT
Unfortunately I am not aware about sc1.If scl is the root cause identified, then option would be run the entire make inside sc1. or maye be would be good to test once by explicitly passing -j2 inside the sc1 as maybe global flags are not getting passed to SC1.
I would like to check if a file exists in a Makefile and run a recipe if it doesn't exist and stop after creating it else continue working with the file that exists and execute more recipes. I think this is best illustrated with an example in my Makefile:
GEN_C_FILES: GEN_C_FLS
$(shell cat $(DESIGN_TOP).c.tmp c_template.c.tmp > $(DESIGN_TOP).c )
$(DESIGN_TOP).c:
$(MAKE) GEN_C_FILES
#(echo $(DESIGN_TOP).c created and modify it to suit the design before running COMPILE)
exit 1
COMPILE: $(DESIGN_TOP).c
#(echo bsub -I compile_script-batch $(DESIGN_TOP).c -log $(DESIGN_TOP).log)
#(bsub -I compile_script -batch $(DESIGN_TOP).c -log $(DESIGN_TOP).log)
I can accomplish what I want but the exit produces an error message. I want it to exit if the recipe of $(DESIGN_TOP).c is executed but if the recipe is not executed the file exist and so it need not be generated and the rest of COMPILE target should complete by running the compile on bsub. Is there a better way to accomplish this without generating the error on exit?. Thanks
You can't make it work as you want when running make COMPILE. The only way to get make to stop in the middle of a build process and not proceed any further is to fail. But, instead you can change what make wants to build in the first place, like this:
GOAL = $(if $(wildcard $(DESIGN_TOP).c),COMPILE,$(DESIGN_TOP).c)
all: $(GOAL)
$(DESIGN_TOP).c:
cat $(DESIGN_TOP).c.tmp c_template.c.tmp > $#
#echo $# created and modify it to suit the design
COMPILE: $(DESIGN_TOP).c
bsub -I compile_script -batch $< -log $(DESIGN_TOP).log
Now if you run make or make all and the source file does not exist, the prerequisite of all will be the source file and nothing else, so it will be built then make will stop.
If the source file does exist, then the prerequisite of all will be COMPILE and it will be built.
I'm using automake.
I'd like to have a script run each time I run 'make'.
This script does a git diff and generates an MD5 sum of the diff.
The hash is written as a #define in repos_version.h
e.g.:
#define REPOS_DIFF "-190886e9f895e80c42cf6b426dc85afd"
The script only rewrites this file if it doesn't exist or if the diff has is different to what is in repos_version.h already. But the script needs to be run for each make.
main.c includes repos_version.h and prints out the hash when the executable is run.
Here's Attempt 1 for Makefile.am
all: config.h
#chmod +x gen_diff_hash.sh
#./gen_diff_hash.sh
$(MAKE) $(AM_MAKEFLAGS) all-recursive
This work, but I get the following error
Makefile:1234: warning: overriding recipe for target all'
Makefile:734: warning: ignoring old recipe for targetall'
Here's Attempt 2 for Makefile.am
all-local:
#chmod +x gen_diff_hash.sh
#./gen_diff_hash.sh
main.c: repos_version.h
However, this doesn't work, as all-local seems to be run too late. A second run of 'make' does get the desired result, but that's not a runner.
So neither are great.
Any ideas?
I've been reading through the automake hooks documentation, but I can't see anything that suits my needs.
You could ensure the script is always run every time Make loads the Makefile, by executing it via $(shell ./gen_diff_hash.sh) and assigning it to a throwaway variable (or using it in some other construct like an ifeq or something).
Note, that this is not POSIX, and on Make implementations other than GNU this isn't valid syntax. GNU Make 4.x supports using VAR != ./gen_diff_hash.sh as well, which is compatible with BSD Make at least.
But maybe it would be a better idea to create a .PHONY: gendiff target that runs the script, and make the header depend on this gendiff. The target would then be re-evaluated every time Make checks if repos_version.h is up-to-date, rather than every time Make is run at all.
When I run make -j3 to build in parallel, I get
warning: -jN forced in submake: disabling jobserver mode.
In the documentation I found the warning is emitted
if make detects error conditions related to parallel processing on
systems where sub-makes can communicate.
What are these error conditions? What can I do to heal them or suppress the error message?
The makefile is generated from CMake, so I cannot (=I don't want to) edit the makefile.
Usually this message occurres if you call make from your Makefile not by the variable $(MAKE)
Example:
Change
foo:
cd foo/ && make foo
to
foo:
cd foo && $(MAKE) foo
You call make -j3 for you top-level Makefile, which means that up to 3 targets can be built simultaneously (make uses 3 threads to build). Your top-level Makefile (generated by CMake) run another make (which is called sub-make). And in this invocation another -jN option is used. So instead of using common jobserver for both makes, sub-make uses another N threads to build its targets. So up to 3 + N threads in total are used for building. So this is what this warning about.
To enable common jobserver for parent make and sub-make, there are 2 requirements:
Explicit -jN option mustn't be passed to sub-make. Call sub-make without this argument.
Sub-make must be called using $(MAKE) expression (not just plain make). Alternatively the + prefix can be added before make command. See this answer for details.
If both requirements are satisfied, then, when you run make -jN for you top-level Makefile, parent make and child make use N threads in common for building.
This is related to multi-threaded make feature. It happens when there is yet another multi-threaded make inside a make process that is already multi-threaded. As there is second level of multi-threaded make, the internal synchronization is not available.
It is possible to fix this by -j parameter.
make -j 1
The code above will avoid multiple thread and it will remove the warnings.
Again, we can use -j parameter in the makefile as well.
target:
$(MAKE) -j 1 -C ${SUBDIR} $#
The above rule will not spawn threads for submake and it will avoid the warning.
With newer versions of GNU make (4.0) and CMake (2.8.6 and 3.3) I can no longer reproduce the warning. One of these got fixed in the meantime.