cmake and parallel building with "make -jN" - makefile

I'm trying to setup a parallel CMake-based build for my source tree, but when I issue
$ cmake .
$ make -j2
I get:
jobserver unavailable: using -j1. Add '+' to parent make rule
as a warning. Does anyone have an idea if it is possible to fix it somehow?

In the generated Makefile, when calling into a sub-make it needs to either use $(MAKE) (not just 'make') or else precede the line with a +. That is, a rule should look like this:
mysubdir:
$(MAKE) -C mysubdir
or like this:
mysubdir:
+make -C mysubdir
If you don't do it one of those two ways, make will give you that warning.
I don't know anything about cmake, so maybe it's generating Makefiles that aren't correct. Or maybe you did something incorrectly on your end.

In my case (with CMake 3.5.2) the trivial cd build && cmake .. && make -j5 works just fine.
But, I do get the jobserver unavailable error when building custom targets (as dependencies of other targets) via the cmake --build . --target foo idiom.
Like this:
add_custom_target(buildroot
COMMAND ${CMAKE_COMMAND} --build . --target install
COMMENT "Populating buildroot..."
)
add_dependencies(deb buildroot)
add_dependencies(rpm buildroot) #... etc
— so that the user can make deb and it Just Works. CMake will regenerate makefiles if needed, run the compilation, install everything exactly as with make install, and then run my custom scripts to package up the populated buildroot into whatever shape or form I need.
Sure enough, I'd like to make -j15 deb — but that fails.
Now, as explained on the mailing list by CMake devs, the root cause lies, surprisingly (or not), within GNU Make; there is a workaround.
The root cause is that make will not pass its jobserver environment to child processes it thinks aren't make.
To illustrate, here's a process tree (ps -A f) branch:
…
\_ bash
\_ make -j15 deb
\_ make -f CMakeFiles/Makefile2 deb
\_ make -f CMakeFiles/buildroot.dir/build.make CMakeFiles/buildroot.dir/build
\_ /usr/bin/cmake --build . --target install ⦿
\_ /usr/bin/gmake install
…
At ⦿ point, make drops jobserver environment, ultimately causing single-threaded compilation.
The workaround which worked great for me, as given away in the linked email, is to prefix all custom commands with +env. Like this:
add_custom_target(buildroot
#-- this ↓↓↓ here -- https://stackoverflow.com/a/41268443/531179
COMMAND +env ${CMAKE_COMMAND} --build . --target install
COMMENT "Populating buildroot..."
)
add_dependencies(deb buildroot)
add_dependencies(rpm buildroot) #... etc
In the end, this appears in the rule for buildroot in the appropriate makefile (CMake generates a bunch of them), and causes GNU Make to behave properly and respect -j.
Hope this helps.

As pointed out by #Carlo Wood in his comment to this answer, trying to convince cmake to add + to the beginning of the command in the cmake-generated makefile is not possible.
A work-around I found is to shield underlying make command from the make flags coming from cmake. This can be done by setting environment variable MAKEFLAGS to empty string for the custom command:
COMMAND ${CMAKE_COMMAND} -E env
MAKEFLAGS=
make <your target and make options>
Hope this helps.

Related

Makefile internal command on linux functional but not macos

Within a Makefile on linux, we can excute shell/bash commands with it to move directories or excute another files. However when porting the same Makefile over to macOS, all the commands are not readible (therefore path and execution are broken). Is there a universal command or workflow that can work on both?
Example of Makefile
.ONESHELL:
COMMAND ?= none
GIT_HASH ?= githash
alpine:
#cd images/alpine
#make ${COMMAND} GIT_HASH=${GIT_HASH} ALPINE_VERSION=3.6.5 TAG=3.6
so in a linux box both #CD and #MAKE are executed but not for macOS Catalina. I would like to make it universal so that both system will respect the appropriate command that follows.
Chances are that your MacOS box uses its default GNU make version (3.81). .ONESHELL was introduced with 3.82. Upgrade with Homebrew or MacPort. Anyway, better avoid make in recipes, prefer $(MAKE), and instead of cd; make you can use GNU make's -C option: $(MAKE) -C images/alpine ...

What is the reason for fatal error: *** No rule to make target 'install'. Stop

The error occurs when I tried to run the command make install under Ubuntu 16.04 that
*** No rule to make target 'install'. Stop.
I have already run make command with several errors fatal: bad revision 'HEAD', which didn't lead to halting the command. I have no idea whether these errors matter.
My makefile is:
SUBDIRS := $(wildcard */.)
all: $(SUBDIRS)
$(SUBDIRS):
make -C $#
install:
for dir in $(SUBDIRS); do \
make -C $$dir install; \
done
.PHONY: all $(SUBDIRS)
Specifically, I want to know how the makefile works after install:.
The project should install an APP on the connected phone Nexus 5. But actually, there's no such APP on my phone.
I suppose your Makefile is properly formatted, with tabs where they should be, etc.
Then, when you run make install in the top level directory, your Makefile does have a rule to make the target install: it says to loop on your subdirectories, enter each one of them, and run make install there (this is what the -C option does). One of those sub-makes fails, most probably because, in its respective subdirectory, it doesn’t find a Makefile with an install recipe in it. When the sub-make fails, the loop goes on with the remaining sub-makes (unless the shell was instructed otherwise by means of the -e switch), and the final return code of the whole recipe will be the return code of the last sub-make.
There are some points worth discussing in your Makefile (for example, install should be listed as a .PHONY target), but you don’t provide enough information to clarify them: for example, is it really necessary to have the shell loop through the subdirectories in a particular order? Usually, a better policy is to have make parallelize the sub-makes whenever possible (and, as a side effect, have make stop when the first submake fails...)

custom install hook and rpmbuild

I have a package that uses autotools for build/install/etc. It is a Python module written in C++. (This isn't important except to know that the python scripts I have to install aren't part of what's being built, i.e. they're not a xxxxx_SOURCES primary.) This module is used solely for some "unit tests" for a driver that our team distributes in a HW solution. For various reasons, I have need to "install" the python unit tests with the system.
Since these python scripts aren't part of the module, I'm managing the installation separately in the Makefile. Here's what I have in my Makefile.am:
EXTRA_DIST = setupenv.sh bootstrap tests
dist-hook:
rm -rf $$(find $(distdir)/tests -name \*.swp -o -name \*.pyc)
install-exec-hook:
mkdir -p $(prefix)/unit_tests/unittest2
for f in tests/*.py; do \
cp $$f $(prefix)/unit_tests; \
done
for f in tests/unittest2/*.py; do \
cp $$f $(prefix)/unit_tests/unittest2; \
done
uninstall-hook:
rm -r $(prefix)/unit_tests
This works just fine except during rpmbuild for the module. The install-exec-hook rule isn't written correctly to make the directory where the rpm process redirects it. That is, when mkdir - $(prefix)/... is executed, the script literally tries to make the dir: /opt/oursw/.... How should this rule be rewritten so that rpmbuild puts them in the same place as the installation for the module?
You want to add $(DESTDIR) as a prefix to all those directories. That's standard for autotools installs.
As an aside, you also might want to look at the _SCRIPTS suffix, which allows "make install" to copy your scripts in for you the "official" way, removing the need for this hack in the first place.

why "make" before "make install"

I know the process of installing from source are.
./configure
make
make install
But why "make" before /etc/cups/cupsd.conf, why not just do "make install"?
My understanding so far is "make" only compile the source into executable file, and "make install" actually place them into executable PATH folder, am I right?
If we want to install executable on the machine, can we just do
./configure
make install
Instead of 3 steps shown above.
When you run make, you're instructing it to essentially follow a set of build steps for a particular target. When make is called with no parameters, it runs the first target, which usually simply compiles the project. make install maps to the install target, which usually does nothing more than copy binaries into their destinations.
Frequently, the install target depends upon the compilation target, so you can get the same results by just running make install. However, I can see at least one good reason to do them in separate steps: privilege separation.
Ordinarily, when you install your software, it goes into locations for which ordinary users do not have write access (like /usr/bin and /usr/local/bin). Often, then, you end up actually having to run make and then sudo make install, as the install step requires a privilege escalation. This is a "Good Thing™", because it allows your software to be compiled as a normal user (which actually makes a difference for some projects), limiting the scope of potential damage for a badly-behaving build procedure, and only obtains root privileges for the install step.
make without parameters takes the ./Makefile (or ./makefile) and builds the first target. By convention, this may be the all target, but not necessarily. make install builds the special target, install. By convention, this takes the results of make all, and installs them on the current computer.
Not everybody needs make install. For example, if you build some a web app to be deployed on a different server, or if you use a cross-compiler (e.g. you build an Android application on a Linux machine), it makes no sense to run make install.
In most cases, the single line ./configure && make all install will be equivalent to the three-step process you describe, but this depends on the product, on your specific needs, and again, this is only by a convention.
There are times I want to try to compile code changes but not deploy those changes. For instance, if I'm hacking the Asterisk C code base, and I want to make sure the changes I'm making still compile, I'll save and run make. However, I don't want to deploy those changes because I'm not done coding.
For me, running make is just a way to make sure I don't end up with too many compile errors in my code to where I have trouble locating them. Perhaps more experienced C programmers don't have that problem, but for me, limiting the number of changes between compiles helps reduce the number of possible changes that may have completely trashed my build, and this makes debugging easier.
Lastly, this also helps give me a stopping point. If I want to go to lunch, I know that someone can restart the application in it's currently working state without having to come find me, since only make install would copy the binaries over to the actual application folder.
There may very well be other reasons, but this is my reason for embracing the fact that the two commands are separated. As others have said, if you want them combined, you can combine them using your shell.
A lot of software these days will do the right thing with only make install.
In those that won't, the install target doesn't have a dependency on the compiled binaries.
So to play safe, most people use make && make install or a variation thereof just to be safe.
A simple Makefile example (real example), from https://github.com/jarun/googler#installation.
Most of the comments are added by me
make install PREFIX=YOUR_own_path
use zsh's autocomple to see what you can chose and you will
know many things!!
PREFIX ?= /usr/local
# These two are the same:
# FOO ?= bar
# ifeq ($(origin FOO), undefined)
# FOO = bar
# endif
# ---
BINDIR = $(DESTDIR)$(PREFIX)/bin
MANDIR = $(DESTDIR)$(PREFIX)/share/man/man1
DOCDIR = $(DESTDIR)$(PREFIX)/share/doc/googler
# the cmamnd `make YOUR_target_name`
# Call a specific target in ./Makefile (or ./makefile), which
# contains such pairs :
# targets:
# ^I shell_command_line_1
# ...
# ^I shell_command_line_n
# `make` can be regarded as using the default target: the first one in
# Makefile, which usually named `all`
# .PHONY: all install uninstall disable-self-upgrade
.PHONY: second all install uninstall disable-self-upgrade
# In terms of `Make`, whenever you ask `make <phony_target>`,
# it will run, independent from the state of what files you have,
# because a `phony target` is marked as always out-of-date
all:
echo "hi, this is the 'all' target"
my_first:
echo "hi, this is the first target"
second:
echo "hi, this is the 2nd target"
# the target `install` can usually be found in Makefile. You can change it to `buy` or others
install:
# from tldr: `install` command : Copy files and set attributes.
# -m --mode= set mode
# -d --dirctory
install --mode=755 -d $(BINDIR)
install -m755 -d $(MANDIR)
install -m755 -d $(DOCDIR)
gzip --to-stdout googler.1 > googler.1.gz
install -m755 googler $(BINDIR)
install -m644 googler.1.gz $(MANDIR)
install -m644 README.md $(DOCDIR)
rm -f googler.1.gz
# same as above
buy:
# from tldr: `install` command : Copy files and set attributes.
# -m --mode= set mode
# -d --dirctory
install --mode=755 -d $(BINDIR)
install -m755 -d $(MANDIR)
install -m755 -d $(DOCDIR)
gzip --to-stdout googler.1 > googler.1.gz
install -m755 googler $(BINDIR)
install -m644 googler.1.gz $(MANDIR)
install -m644 README.md $(DOCDIR)
rm -f googler.1.gz
uninstall:
rm -f $(BINDIR)/googler
rm -f $(MANDIR)/googler.1.gz
rm -rf $(DOCDIR)
# Ignore below if you don't use apt or others package managers to install this
# Disable the self-upgrade mechanism entirely. Intended for packagers
#
# We assume that sed(1) has the -i option, which is not POSIX but seems common
# enough in modern implementations.
disable-self-upgrade:
sed -i.bak 's/^ENABLE_SELF_UPGRADE_MECHANISM = True$$/ENABLE_SELF_UPGRADE_MECHANISM = False/' googler

Binary fortran file crashes under Make

I have a binary that runs under my default shell.
The binary runs perfectly o.k. with:
./binary input.dat
However, if I put this inside a make file:
SHELL=/bin/bash
runos:
./binary input.dat
The code crashes and leaves me quite helpless.
Here is what I tested so far, everything inside my Make file and in the shell:
ulimit -a: identical.
Set the shell to bash as seen above.
diff of the environment variables in SHELL and Make with:
env | sort > vars.1
inside make
env | sort > vars.2
Then run the binary with the extra variables in Make with the following command:
env SHLVL=2 MAKELEVEL=1 MAKEFLAGS= ./binary input.dat
strace in the shell and inside make:
strace -o debug binary input.dat
The code keeps on crashing in Make, and runs in the shell. I am already thinking to dump Make for my test cases and just write shell scripts. But I am curious to know what is the difference.
The Fortran code (a mix of F77, F90 and F95) was compiled with gfortran-4.4 and the following options:
FFLAGS= -g -fbacktrace
So, the concrete question is, what can I do to make this binary run under make in Debian!?
update:
I just tested again in a CentOS machine (v5.8), The code inside Makefile does not crash (GNU Make version 3.81).
I also tested on my Debian Wheezy and openSUSE 11.4, both with GNU Make version 3.82 - It crashes!
I tested on Debian Squeeze with GNU Make version 3.81, and it does crash. So, I think it is not dependent on the GNU Make version.
error when crashing:
enter timeloop
------------------------------------------------------------------------
timestep: 1 time: 2.500E-02 days delt: 2.500E-02 days
-------------------------------------------
terminated in routine react_snia
maximum number of iterations exceeded
bye now ...
-------------------------------------------
failure in timeloop
no further time step reduction possible
try reducing min. time step, bye now ...
trying to work around 'GNU Make' using 'waf'
It has been a while since I wanted to test waf, so here is another interesting observation:
I wrote a wscript which contains a function:
import os
def run(ctx):
os.system('./binary input.dat')
And waf run runs!
If I changed the run method to:
import subprocess as sp
def run(ctx):
sp.call('./binary input.dat', shell=True)
The binary also works as expected.
So, now I am thinking GNU Make forks a new sub-shell in a way that causes may binary to fail (although, under RHEL 5.8 Make did work).
solution: compile make from sources ...
Read to find out more.
OK, so after being pretty much desperate, I did what I simply should have done before blame make for all my troubles.
I thought the problem is Debian specific. But I am guessing the version in CentOS-5.8 is a patched version, although it says it's v.3.81.
So, for those who wonder my solution was:
wget http://ftp.gnu.org/gnu/make/make-3.82.tar.gz
tar xvzf make-3.82.tar.gz
cd make-3.82
./configure
./build.sh
# copy make to the directory with the binary and input and run the local make version
./make
# everything works as expected !!!
I thought let's narrow it down -
wget http://ftp.gnu.org/gnu/make/make-3.80.tar.gz
tar xvzf make-3.80.tar.gz
cd make-3.80
./configure
./build.sh
# copy make to the directory with the binary and input and run the local make version
./make
# everything works as expected !!!
Is it the version 3.81 ?
wget http://ftp.gnu.org/gnu/make/make-3.81.tar.gz
tar xvzf make-3.81.tar.gz
cd make-3.81
./configure
./build.sh
# copy make to the directory with the binary and input and run the local make version
./make
# FAIL! Like with the make version in Debian.
Hence, I think I bumped into some very weird bug in GNU Make v.3.81.

Resources