I'd like to add a quick check to a (GNU) makefile which can alert the user to the availability of -j/--jobs (parallel make). That is, something like
$ make
TIP: this will build faster if you use use "make -j"
Building ..
$ make -j
Building in parallel ..
How can I determine the number of parallel jobs when a Makefile is executed?
There is a trick here
http://blog.jgc.org/2015/03/gnu-make-insanity-finding-value-of-j.html
and a proposed change to GNU Make itself here
https://github.com/esantoro/make/commit/b0334e7f3009dc58dbc8e6e6fdec94711537fb3b
but perhaps there is something newer and/or easier.
The simplest/best solution is to upgrade your version of GNU make to 4.2 or above. Starting with that version, the MAKEFLAGS variable will provide the full -j option including the number. The NEWS file says:
The amount of parallelism can be determined by querying MAKEFLAGS, even when
the job server is enabled (previously MAKEFLAGS would always contain only
"-j", with no number, when job server was enabled).
So:
$ make --version
GNU Make 4.2.1
...
$ echo 'all:;#echo $(MAKEFLAGS)' | make -f-
$ echo 'all:;#echo $(MAKEFLAGS)' | make -f- -j
-j
$ echo 'all:;#echo $(MAKEFLAGS)' | make -f- -j10
-j10 --jobserver-auth=3,4
$ echo 'all:;#echo $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS)))' | make -f- -j10
10
You can determine the number of jobs easier and faster than that blog proposes by using make Jobserver protocol:
SHELL := /bin/bash
all:
#${MAKE} --no-print-directory job_count_test
job_count_test:
#+[[ "${MAKEFLAGS}" =~ --jobserver[^=]+=([0-9]+),([0-9]+) ]] && ( J=""; while read -t0 -u $${BASH_REMATCH[1]}; do read -N1 -u $${BASH_REMATCH[1]}; J="$${J}$${REPLY}"; done; echo "Building with $$(expr 1 + $${#J}) jobs."; echo -n $$J >&$${BASH_REMATCH[2]} ) || echo "TIP: this will build faster if you use use \"make -j$$(grep -c processor /proc/cpuinfo)\""
.PHONY: all job_count_test
And then:
$ make
TIP: this will build faster if you use use "make -j8"
$ make -j12
Building with 12 jobs.
Related
I can invoke (GNU) make with the -j option, enabling multiple jobs to be run in parallel:
make -j
Is it possible from within the Makefile to determine whether or not make was invoked with this option? I have tried looking for some automatic variable storing all options given, but I could not find it.
I ask because I want the build process to be slightly different depending on whether or not -j was used.
I am not sure if this is a standard practice (or a good advise), but here it is:
makefile:
something:
#echo MAKEFLAGS : $(MAKEFLAGS)
Running:
/home/user> make something
MAKEFLAGS :
/home/user> make -j something
MAKEFLAGS : -j
/home/user> make -j 10 something
MAKEFLAGS : -j10 --jobserver-auth=3,4
/home/user> make -j20 something
MAKEFLAGS : -j20 --jobserver-auth=3,4
Look in the MAKEFLAGS GNU make variable.
echo 'all:;: $(MAKEFLAGS)' | make -f-
:
echo 'all:;: $(MAKEFLAGS)' | make -f- -j
: -j
I'm new to Makefiles and I want to modify the extension of a set of files. The following command works on the shell:
for file in path/*.ext1; do j=`echo $file | cut -d . -f 1`;j=$j".ext2";echo mv $file $j; done
However, I'm not sure how to run this in a Makefile. I tried running
$(shell for file in path/*.ext1; do j=`echo $file | cut -d . -f 1`;j=$j".ext2";echo mv $file $j; done)
But this never did what I needed it to do. What do I need to do to make this work on the Makefile? How do I call it in a section?
The immediate answer to your question is that the $ character is special to make: it introduces a make variable. If you want to pass a $ to the shell, you'll have to write two of them: $$.
So, your shell function invocation would have to be written as:
$(shell for file in path/*.ext1; do j=`echo $$file | cut -d . -f 1`;j=$$j".ext2";echo mv $$file $$j; done)
However, this is almost certainly not a good way to do what you want. You don't really describe clearly what you want to do, however. If you just want to have a target in a makefile that can be invoked to make this change, you can use:
fixext:
for file in path/*.ext1; do \
j=`echo $$file | cut -d . -f 1`; \
j=$$j".ext2"; \
echo mv $$file $$j; \
done
Or, taking advantage of some useful shell shortcuts, you could just run:
fixext:
for file in path/*.ext1; do \
echo mv $$file $${file%.*}.ext2; \
done
Now if you run make fixext it will perform those steps.
But, a much more make-like way to do it would be to write a single rule that knows how to rename one file, then use prerequisites to have them all renamed:
TARGETS = $(patsubst %.ext1,%.ext2,$(wildcard path/*.ext1))
fixext: $(TARGETS)
%.ext2 : %.ext1
mv $< $#
Now you can even run make -j5 and do 5 of the move commands in parallel...
you can also add rename blocks at the top of your file eg to change a suffix
output := $(input:.mov=.mp4)
but this won't work inside a make command as far as I can see
check:
output := $(input:.mov=.mp4)
gives
$ input=walkthrough.mov make check
output := walkthrough.mp4
make: output: No such file or directory
make: *** [check] Error 1
On this line:
GCCVER:=$(shell a=`mktemp` && echo $'#include <stdio.h>\nmain() {printf("%u.%u\\n", __GNUC__, __GNUC_MINOR__);}' | gcc -o "$a" -xc -; "$a"; rm "$a")
I get:
*** unterminated call to function `shell': missing `)'. Stop.
What's wrong with my stupidly circuitous variable?
Update0
$ make --version
GNU Make 3.81
$ bash --version
GNU bash, version 4.2.8(1)-release (x86_64-pc-linux-gnu)
$ uname -a
Linux 2.6.38-10-generic #46-Ubuntu SMP x86_64 GNU/Linux
$ gcc --version
gcc (Ubuntu/Linaro 4.5.2-8ubuntu4) 4.5.2
when using $ for Bash inside a Makefile, you need to double them: $$a for example. I'm not familiar with the notation $' but I'll have to assume you know what you're doing with that. unless it's a Makefile construct, you need to double the dollar sign on that one too.
also, the hash sign # is terminating the shell expansion in Make's evaluation, which is why it never sees the right paren. escaping it helps, but I don't have it working quite right yet.
I'm debugging it by having two steps: first is setting GCCVER to be the list of commands without the enclosing $(shell), then in the 2nd step setting GCCVER := $(shell $(GCCVER)). you might want to try that too, commenting out the $(shell) step when it doesn't work, using export, and making a "set" recipe:
GCCVER := some commands here
#GCCVER := $(shell $(GCCVER)) # expand the commands, commented out now
export # all variables available to shell
set:
set # make sure this is prefixed by a tab, not spaces
Then:
make set | grep GCCVER
[update] this works:
GCCVER := a=`mktemp` && echo -e '\#include <stdio.h>\nmain() {printf("%u.%u\\n", __GNUC__, __GNUC_MINOR__);}' | gcc -o "$$a" -xc -; "$$a"; rm "$$a"
GCCVER := $(shell $(GCCVER))
export
default:
set
jcomeau#intrepid:/tmp$ make | grep GCCVER
GCCVER=4.6
And full circle, having gotten rid of the extra step:
jcomeau#intrepid:/tmp$ make | grep GCCVER; cat Makefile
GCCVER=4.6
GCCVER := $(shell a=`mktemp` && echo -e '\#include <stdio.h>\nmain() {printf("%u.%u\\n", __GNUC__, __GNUC_MINOR__);}' | gcc -o "$$a" -xc -; "$$a"; rm "$$a")
export
default:
set
Using the $' Bash construct:
jcomeau#intrepid:/tmp$ make | grep GCCVER; cat Makefile
GCCVER=4.6
GCCVER := $(shell a=`mktemp` && echo $$'\#include <stdio.h>\nmain() {printf("%u.%u\\n", __GNUC__, __GNUC_MINOR__);}' | gcc -o "$$a" -xc -; "$$a"; rm "$$a")
export
default:
set
Since your system doesn't work the same as mine, I'm going to cop out and say either use reinierpost's suggestion or, alternatively:
GCCVER := $(shell gcc -dumpversion | cut -d. -f1,2)
Criteria: Makefile is a GNU Make Makefile - I'm not interested in makepp, qmake, cmake, etc. They're all nice (especially cmake), but this is for work and at work we use GNU Make. The optimal solution is a pure Makefile solution rather than a shell script that parses make for you.
I also don't want to do a 'continue on failure' solution - if it's broken, it's broken and needs to be fixed.
The situation is this, I've got a makefile that builds several directories in parallel - if one of them fails, of course the whole build fails, but not until all the running makes run to completion (or failure). This means that the reason why make actually failed is buried somewhere arbitrarily far from the end of make's output.
Here's an example of what I've got:
all: $(SUBDIRS)
SUBDIRS = \
apple \
orange \
banana \
pineapple \
lemon \
watermelon \
grapefruit
$(SUBDIRS):
cd $# && $(MAKE) $(MFLAGS) 2>&1 | sed -e "s/^/$(notdir $(#)): /g"
If I run 'make -j 5' and 'orange' happens to fail - I'd like to see a table like this at the end
of the make process
apple - passed
orange - FAILED
banana - passed
pineapple - passed
lemon - passed
I've considered having an && echo "passed" >.result || echo "FAILED" >.result, but make still needs some sort of TRAP or __onexit() cleanup command to print at them on exit.
Any Makefile ninjas out there have a pure-makefile solution for this?
un-edit - my solution wasn't actually working the way I had hoped.. STYMIED!
When you want make to abort at the first failure, end immediately and kill all in-flight jobs instead of waiting for them to finish, you need to patch GNU Make like this
http://lists.gnu.org/archive/html/bug-make/2009-01/msg00035.html
Then you need to set a trap for every shell that make invokes (as well as set -o pipefail if you use a pipe), as described in this post http://lists.gnu.org/archive/html/help-make/2009-02/msg00011.html
In a nutshell:
target1:
trap 'kill $$(jobs -p)'; command && something || something-else
target2:
trap 'kill $$(jobs -p)'; set -o pipefail; command | sed '...'
The only way I see is self-execution with a sub-make:
all : subdirs
subdirs :
$(MAKE) -f $(lastword $(MAKEFILE_LIST)) subdirs-recursive || cat log
subdirs-recursive: $(SUBDIRS)
Not sure if this is possible in one Makefile alone, but I was hoping to write a Makefile in a way such that trying to build any target in the file auto-magically detects the number of processors on the current system and builds the target in parallel for the number of processors.
Something like the below "pseudo-code" examples, but much cleaner?
all:
#make -j$(NUM_PROCESSORS) all
Or:
all: .inparallel
... build all here ...
.inparallel:
#make -j$(NUM_PROCESSORS) $(ORIGINAL_TARGET)
In both cases, all you would have to type is:
% make all
Hopefully that makes sense.
UPDATE: Still hoping for an example Makefile for the above. Not really interested in finding the number of processes, but interested in how to write a makefile to build in parallel without the -j command line option.
The detection part is going to be OS dependent. Here's a fragment that will work on Linux and Mac OS X:
NPROCS:=1
OS:=$(shell uname -s)
ifeq($(OS),Linux)
NPROCS:=$(shell grep -c ^processor /proc/cpuinfo)
endif
ifeq($(OS),Darwin) # Assume Mac OS X
NPROCS:=$(shell system_profiler | awk '/Number Of CPUs/{print $4}{next;}')
endif
To get it working you are probably going to have to re-invoke make. Then your problem is preventing infinite recursion. You could manage that by having two makefiles (the first only resetting the -j value), but it is probably possible to finesse it.
I just added this to the top of my Makefile. It lets make create any number of jobs, but tries to keep the load average below the number of cpu cores.
MAKEFLAGS+="-j -l $(shell grep -c ^processor /proc/cpuinfo) "
Note this is Linux specific.
Here's what I went with:
ifeq ($(OS),Linux)
NUMPROC := $(shell grep -c ^processor /proc/cpuinfo)
else ifeq ($(OS),Darwin)
NUMPROC := $(shell sysctl hw.ncpu | awk '{print $$2}')
endif
# Only take half as many processors as available
NUMPROC := $(shell echo "$(NUMPROC)/2"|bc)
ifeq ($(NUMPROC),0)
NUMPROC = 1
endif
After poking around the LDD3 chapter 2 a bit and reading dmckee's answer, I came up with this not so great answer of using two makefiles (I would prefer just one).
$ cat Makefile
MAKEFLAGS += -rR --no-print-directory
NPROCS := 1
OS := $(shell uname)
export NPROCS
ifeq ($J,)
ifeq ($(OS),Linux)
NPROCS := $(shell grep -c ^processor /proc/cpuinfo)
else ifeq ($(OS),Darwin)
NPROCS := $(shell system_profiler | awk '/Number of CPUs/ {print $$4}{next;}')
endif # $(OS)
else
NPROCS := $J
endif # $J
all:
#echo "running $(NPROCS) jobs..."
#$(MAKE) -j$(NPROCS) -f Makefile.goals $#
%:
#echo "building in $(NPROCS) jobs..."
#$(MAKE) -j$(NPROCS) -f Makefile.goals $#
$ cat Makefile.goals
MAKEFLAGS += -rR --no-print-directory
NPROCS ?= 1
all: subgoal
#echo "$(MAKELEVEL) nprocs = $(NPROCS)"
subgoal:
#echo "$(MAKELEVEL) subgoal"
What do you think about this solution?
Benefits I see is that people still type make to build. So there isn't some "driver" script that does the NPROCS and make -j$(NPROCS) work which people will have to know instead of typing make.
Downside is that you'll have to explicitly use make -f Makefile.goals in order to do a serial build. And I'm not sure how to solve this problem...
UPDATED: added $J to above code segment. Seems work work quite well. Even though its two makefiles instead of one, its still quite seamless and useful.
I'll skip over the $(NPROCS) detection stuff, but here's how you could do this in a single Makefile (this is probably GNU Make specific, but that looks like the variant you're running):
ifeq ($(NPROCS),)
# Code to set NPROCS...
%:
$(MAKE) -j$(NPROCS) NPROCS=$(NPROCS)
else
# All of your targets...
endif
See Defining Last-Resort Default Rules and Overriding Part of Another Makefile in the GNU Make Manual.
If I read the question correctly, the goal is to parallelize the build process as much as possible. The make man page states the following
If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.
Isn't this basically the solution that you want? If your Makefile has enough parallel targets you will use all your CPUs and if the targets are not parallel, that -j option won't help anywas.
If you want it to be automatic, then you can override your typical make command to be an alias to itself in your .bashrc in your home directory.
Example:
alias make="make -j"
Or you could do something like:
alias jmake="make -j"
in the case that you don't want to override it, but want a quick and easy (and memorable) way to run make in parallel.