I can invoke (GNU) make with the -j option, enabling multiple jobs to be run in parallel:
make -j
Is it possible from within the Makefile to determine whether or not make was invoked with this option? I have tried looking for some automatic variable storing all options given, but I could not find it.
I ask because I want the build process to be slightly different depending on whether or not -j was used.
I am not sure if this is a standard practice (or a good advise), but here it is:
makefile:
something:
#echo MAKEFLAGS : $(MAKEFLAGS)
Running:
/home/user> make something
MAKEFLAGS :
/home/user> make -j something
MAKEFLAGS : -j
/home/user> make -j 10 something
MAKEFLAGS : -j10 --jobserver-auth=3,4
/home/user> make -j20 something
MAKEFLAGS : -j20 --jobserver-auth=3,4
Look in the MAKEFLAGS GNU make variable.
echo 'all:;: $(MAKEFLAGS)' | make -f-
:
echo 'all:;: $(MAKEFLAGS)' | make -f- -j
: -j
Related
I've tweak "a bit" Make so I can use it as a "kind of" cli for some tasks.
MAKEFLAGS += --no-builtin-rules
MAKEFLAGS += --no-builtin-variables
MAKEFLAGS += --no-print-directory
SHELL := /bin/bash
.ONESHELL:
.PHONY: project_list
project_list: all_projects_info.json
echo "Filtering project list with:" >&2
echo " PROJECT_FILTER: $(PROJECT_FILTER)" >&2
jq -r -S '.[] | select(
(.projectId | test("$(PROJECT_FILTER)"))
) | .projectId' $^ > $#
.PHONY: get_storage_info
get_storage_info: project_list
PROJECT_LIST=$$(cat $<)
$(MAKE) -f $(MKFILE) -j storage_info.json PROJECT_LIST="$$PROJECT_LIST"
all_projects_info.json:
curl -X GET https://toto/get_all_my_projects_info >$#
# here it's PHONY because we want to always rebuild it
.PHONY: storage_info.json
storage_info.json: $(STORAGE_INFO_JSON_FILES)
jq -s -S '[.[]?.items?[]?]' $(STORAGE_INFO_JSON_FILES) > $#
storage_info/:
mkdir -p $#
STORAGE_INFO_JSON_FILES=$(foreach project_name,$(PROJECT_LIST),storage_info/$(project_name).json)
$(STORAGE_INFO_JSON_FILES): storage_info/%.json: | storage_info/
curl \
-X GET \
"https://storage_api/list_s3?project=$*" \
2> /dev/null > $#
As you can see here, I've got 2 "command":
project_list witch list all project I can access too,
get_storage_info witch list all bucket in projects.
The trick here is because I've got a lot of projects and buckets, I may want to filter like this:
make get_storage_info PROJECT="foo"
And it will print ONLY bucket in project with foo in their name.
It's quit handy and fast (only the first time it may be slow, the time to get all informations).
What is bothering me, I've not found a better way than to call a sub make command (with the exact list of project to take into account).
Is it possible to express dynamic dependencies of a target ?
But something that can result from another target ?
Thanks.
I don't see anything wrong with invoking a submake. That's IMO the best way to do it, especially if you want to add -j to it.
It's not really possible to get rid of this easily. It's not the fact that you want to express a dynamic dependency: that can be done. The problem is you want the list of dependencies to be extracted from the results of running another rule. But that's not how make works: make always starts with the final target and works its way backwards. So, by the time you get around to building the prerequisite file, the target that depended on it has already been processed (not its recipe of course, but all the prerequisites).
I'd like to add a quick check to a (GNU) makefile which can alert the user to the availability of -j/--jobs (parallel make). That is, something like
$ make
TIP: this will build faster if you use use "make -j"
Building ..
$ make -j
Building in parallel ..
How can I determine the number of parallel jobs when a Makefile is executed?
There is a trick here
http://blog.jgc.org/2015/03/gnu-make-insanity-finding-value-of-j.html
and a proposed change to GNU Make itself here
https://github.com/esantoro/make/commit/b0334e7f3009dc58dbc8e6e6fdec94711537fb3b
but perhaps there is something newer and/or easier.
The simplest/best solution is to upgrade your version of GNU make to 4.2 or above. Starting with that version, the MAKEFLAGS variable will provide the full -j option including the number. The NEWS file says:
The amount of parallelism can be determined by querying MAKEFLAGS, even when
the job server is enabled (previously MAKEFLAGS would always contain only
"-j", with no number, when job server was enabled).
So:
$ make --version
GNU Make 4.2.1
...
$ echo 'all:;#echo $(MAKEFLAGS)' | make -f-
$ echo 'all:;#echo $(MAKEFLAGS)' | make -f- -j
-j
$ echo 'all:;#echo $(MAKEFLAGS)' | make -f- -j10
-j10 --jobserver-auth=3,4
$ echo 'all:;#echo $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS)))' | make -f- -j10
10
You can determine the number of jobs easier and faster than that blog proposes by using make Jobserver protocol:
SHELL := /bin/bash
all:
#${MAKE} --no-print-directory job_count_test
job_count_test:
#+[[ "${MAKEFLAGS}" =~ --jobserver[^=]+=([0-9]+),([0-9]+) ]] && ( J=""; while read -t0 -u $${BASH_REMATCH[1]}; do read -N1 -u $${BASH_REMATCH[1]}; J="$${J}$${REPLY}"; done; echo "Building with $$(expr 1 + $${#J}) jobs."; echo -n $$J >&$${BASH_REMATCH[2]} ) || echo "TIP: this will build faster if you use use \"make -j$$(grep -c processor /proc/cpuinfo)\""
.PHONY: all job_count_test
And then:
$ make
TIP: this will build faster if you use use "make -j8"
$ make -j12
Building with 12 jobs.
Criteria: Makefile is a GNU Make Makefile - I'm not interested in makepp, qmake, cmake, etc. They're all nice (especially cmake), but this is for work and at work we use GNU Make. The optimal solution is a pure Makefile solution rather than a shell script that parses make for you.
I also don't want to do a 'continue on failure' solution - if it's broken, it's broken and needs to be fixed.
The situation is this, I've got a makefile that builds several directories in parallel - if one of them fails, of course the whole build fails, but not until all the running makes run to completion (or failure). This means that the reason why make actually failed is buried somewhere arbitrarily far from the end of make's output.
Here's an example of what I've got:
all: $(SUBDIRS)
SUBDIRS = \
apple \
orange \
banana \
pineapple \
lemon \
watermelon \
grapefruit
$(SUBDIRS):
cd $# && $(MAKE) $(MFLAGS) 2>&1 | sed -e "s/^/$(notdir $(#)): /g"
If I run 'make -j 5' and 'orange' happens to fail - I'd like to see a table like this at the end
of the make process
apple - passed
orange - FAILED
banana - passed
pineapple - passed
lemon - passed
I've considered having an && echo "passed" >.result || echo "FAILED" >.result, but make still needs some sort of TRAP or __onexit() cleanup command to print at them on exit.
Any Makefile ninjas out there have a pure-makefile solution for this?
un-edit - my solution wasn't actually working the way I had hoped.. STYMIED!
When you want make to abort at the first failure, end immediately and kill all in-flight jobs instead of waiting for them to finish, you need to patch GNU Make like this
http://lists.gnu.org/archive/html/bug-make/2009-01/msg00035.html
Then you need to set a trap for every shell that make invokes (as well as set -o pipefail if you use a pipe), as described in this post http://lists.gnu.org/archive/html/help-make/2009-02/msg00011.html
In a nutshell:
target1:
trap 'kill $$(jobs -p)'; command && something || something-else
target2:
trap 'kill $$(jobs -p)'; set -o pipefail; command | sed '...'
The only way I see is self-execution with a sub-make:
all : subdirs
subdirs :
$(MAKE) -f $(lastword $(MAKEFILE_LIST)) subdirs-recursive || cat log
subdirs-recursive: $(SUBDIRS)
Not sure if this is possible in one Makefile alone, but I was hoping to write a Makefile in a way such that trying to build any target in the file auto-magically detects the number of processors on the current system and builds the target in parallel for the number of processors.
Something like the below "pseudo-code" examples, but much cleaner?
all:
#make -j$(NUM_PROCESSORS) all
Or:
all: .inparallel
... build all here ...
.inparallel:
#make -j$(NUM_PROCESSORS) $(ORIGINAL_TARGET)
In both cases, all you would have to type is:
% make all
Hopefully that makes sense.
UPDATE: Still hoping for an example Makefile for the above. Not really interested in finding the number of processes, but interested in how to write a makefile to build in parallel without the -j command line option.
The detection part is going to be OS dependent. Here's a fragment that will work on Linux and Mac OS X:
NPROCS:=1
OS:=$(shell uname -s)
ifeq($(OS),Linux)
NPROCS:=$(shell grep -c ^processor /proc/cpuinfo)
endif
ifeq($(OS),Darwin) # Assume Mac OS X
NPROCS:=$(shell system_profiler | awk '/Number Of CPUs/{print $4}{next;}')
endif
To get it working you are probably going to have to re-invoke make. Then your problem is preventing infinite recursion. You could manage that by having two makefiles (the first only resetting the -j value), but it is probably possible to finesse it.
I just added this to the top of my Makefile. It lets make create any number of jobs, but tries to keep the load average below the number of cpu cores.
MAKEFLAGS+="-j -l $(shell grep -c ^processor /proc/cpuinfo) "
Note this is Linux specific.
Here's what I went with:
ifeq ($(OS),Linux)
NUMPROC := $(shell grep -c ^processor /proc/cpuinfo)
else ifeq ($(OS),Darwin)
NUMPROC := $(shell sysctl hw.ncpu | awk '{print $$2}')
endif
# Only take half as many processors as available
NUMPROC := $(shell echo "$(NUMPROC)/2"|bc)
ifeq ($(NUMPROC),0)
NUMPROC = 1
endif
After poking around the LDD3 chapter 2 a bit and reading dmckee's answer, I came up with this not so great answer of using two makefiles (I would prefer just one).
$ cat Makefile
MAKEFLAGS += -rR --no-print-directory
NPROCS := 1
OS := $(shell uname)
export NPROCS
ifeq ($J,)
ifeq ($(OS),Linux)
NPROCS := $(shell grep -c ^processor /proc/cpuinfo)
else ifeq ($(OS),Darwin)
NPROCS := $(shell system_profiler | awk '/Number of CPUs/ {print $$4}{next;}')
endif # $(OS)
else
NPROCS := $J
endif # $J
all:
#echo "running $(NPROCS) jobs..."
#$(MAKE) -j$(NPROCS) -f Makefile.goals $#
%:
#echo "building in $(NPROCS) jobs..."
#$(MAKE) -j$(NPROCS) -f Makefile.goals $#
$ cat Makefile.goals
MAKEFLAGS += -rR --no-print-directory
NPROCS ?= 1
all: subgoal
#echo "$(MAKELEVEL) nprocs = $(NPROCS)"
subgoal:
#echo "$(MAKELEVEL) subgoal"
What do you think about this solution?
Benefits I see is that people still type make to build. So there isn't some "driver" script that does the NPROCS and make -j$(NPROCS) work which people will have to know instead of typing make.
Downside is that you'll have to explicitly use make -f Makefile.goals in order to do a serial build. And I'm not sure how to solve this problem...
UPDATED: added $J to above code segment. Seems work work quite well. Even though its two makefiles instead of one, its still quite seamless and useful.
I'll skip over the $(NPROCS) detection stuff, but here's how you could do this in a single Makefile (this is probably GNU Make specific, but that looks like the variant you're running):
ifeq ($(NPROCS),)
# Code to set NPROCS...
%:
$(MAKE) -j$(NPROCS) NPROCS=$(NPROCS)
else
# All of your targets...
endif
See Defining Last-Resort Default Rules and Overriding Part of Another Makefile in the GNU Make Manual.
If I read the question correctly, the goal is to parallelize the build process as much as possible. The make man page states the following
If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.
Isn't this basically the solution that you want? If your Makefile has enough parallel targets you will use all your CPUs and if the targets are not parallel, that -j option won't help anywas.
If you want it to be automatic, then you can override your typical make command to be an alias to itself in your .bashrc in your home directory.
Example:
alias make="make -j"
Or you could do something like:
alias jmake="make -j"
in the case that you don't want to override it, but want a quick and easy (and memorable) way to run make in parallel.
How to get the name of the makefile in the makefile?
Thanks.
Note:
I would need that because I would like my makefile to call itself, but the makefile is not called Makefile, so I'd like to write something like this:
target:
($MAKE) -f ($MAKEFILENAME) other_target
location = $(CURDIR)/$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
WHERE_ART_THOU := $(location)
$(warning $(WHERE_ART_THOU))
I also believe this is GNU make-specific, but I'm not too sure.
(Should you have any questions, refer to amazingly written GNU make manual. But remember, that, just like Makefile, this manual should be read completely before putting the concepts into practice).
I couldn't figure out how it is done easily. As far as I understand, you'll have to do some manual job.
Later I will describe how it could be done and show scripts that introduce current_makefile variable. But I would like to stress an important concept at the first place.
You should understand that if we had some kind of variable current_makefile, that expands to the current makefile name, then it will have to change during the process of reading makefiles. That means that it should be used withinin "immediate" expansion context -- i.e. within commands that are executed during reading the makefile. Most commands, however, are executed after makefiles are read. Therefore, some commands will print the correct value smoothly, while in certain places, where "deferred" expansion is used, it will always expand to the root makefile name.
If you would want to use this variable within rule text, for example, you'll have to do tricks, because rule text always has deferred expansion. So, if your have the rule
rule:
echo In makefile $(current_makefile):
echo Making target $#
it will always print the name of the root makefile. Instead, to force immediate expansion, you will have to create another variable with makefile-specific name (i.e. names of such variables should be different in each makefile):
this_makefile_unique_name := $(current_makefile)
rule:
echo In makefile $(current_makefile):
echo Making target $#
or use eval:.
define make_rule
rule:
echo In makefile $(1):
echo Making target $$#
$(eval $(call make_rule,$(current_makefile)))
If you want to use the name of current makefile for debug purpose only, consider special debugging functions, like warning or info:.
$(warning We're in makefile $(current_makefile))
These functions use "immediate" expansion and will print the correct value.
How to define such a $(current_makefile)?
You have to manually maintain stack of makefile inclusions. When you include a makefile, its name is placed to the top of the stack; when you return from included makefile to the outer one, the topmost name is popped out of stack. This is achieved by inserting special calls to the beginning and the end of makefile:
# Beginning of makefile
$(eval $(makefile_names_push))
#... makefile text
$(warning $(current_makefile))
#...
$(eval $(makefile_names_pop))
#End of file
Now define the functions at the beginning of your root makefile.
lastword=$(word $(words $(1)),$(1))
define makefile_names_push
current_makefile := $$(CURDIR)/$$(call lastword,$$(MAKEFILE_LIST))
makefile_stack :=$$(makefile_stack) $$(current_makefile)
endef
define makefile_names_pop
makefile_stack := $$(filter-out $$(current_makefile),$$(makefile_stack))
current_makefile := $$(call lastword,$$(makefile_stack))
endef
If you're sure your make is new enough (version 3.81+), replace lastword call with builtin function:.
#inctead of $$(call lastword,$$(MAKEFILE_LIST))
$$(lastword $$(MAKEFILE_LIST))
Is it useful?
Totally useless. An only use that might be useful here is to make 100 makefiles that are symlinks to one makefile, the rules in these makefiles depending on their names. But it can be achieved within one makefile and foreach-eval technique described in the manual. So my post was a complete waste of time, though I had some fun :-)
This returns the name of the first Makefile called, i.e. the one at the bottom of the call stack:
MAKEFILE_JUSTNAME := $(firstword $(MAKEFILE_LIST))
MAKEFILE_COMPLETE := $(CURDIR)/$(MAKEFILE_JUSTNAME)
When used in non-cross-recursive situations (e.g. for makedepend), it is just the name of the current makefile.
I wanted to do something similar (for echoing the contents of the Makefile) for when I use Make for managing simple repetitive tasks. I came across this page and found it was exactly what I was after and really useful for my limited understanding of make.
My result after reading this page:
# Makefile - 'make' and 'make help' now echo the makefile.
help:
cat $(lastword $(MAKEFILE_LIST))
start:
sudo -u www /path/to/webapp/myhttpd restart
stop:
sudo kill `cat /path/to/webapp/data/httpd.pid`
A quick excursion to Google suggests this site has the answer.
G'day,
If you make a copy of your original makefile, say makefile_test, and then enter the command:
make -np -f makefile_test 2>&1 | tee output
That will evaluate the makefile and your make environment but not execute any of the commands. Looking through the output file for references to makefile_test will show you what is set in make's environment and where that value is being set.
N.B. This can generate a lot of info! And don't add the -d (debug) switch which will generate tons of additional output about make's decision process but minimal additional info about make's env.
HTH
The solutions here addresses 1) POSIX make with 2) Invoked, non included, makefile in 3) A Unix alike platform.
What the OP asked for:
target:
#pid=$$$$; \
while test `ps -ocomm= $$pid` != make; do \
pid=`ps -oppid= $$pid`; \
done; \
MAKEFILENAME=`ps -oargs= $$pid|sed 's/^.* -f *\([^ ]*\).*$$/\1/'`; \
test -z "$$MAKEFILENAME" -a -f Makefile && MAKEFILENAME=Makefile; \
test -z "$$MAKEFILENAME" -a -f makefile && MAKEFILENAME=makefile; \
export MAKEFILENAME; \
$(MAKE) -e -f $$MAKEFILENAME other_target
The targets depends on the makefile, kind of bloated:
TARGET1_MAKEFILENAME = target1_preamble
all: target1 target2...
target1: $(TARGET1_MAKEFILENAME) other_dependencies...
#test $(TARGET1_MAKEFILENAME) == target1_preamble && exit 0; \
built_instructions_for_target1;
target1_preamble:
#pid=$$$$; \
while test `ps -ocomm= $$pid` != make; do \
pid=`ps -oppid= $$pid`; \
done; \
MAKEFILENAME=`ps -oargs= $$pid|sed 's/^.* -f *\([^ ]*\).*$$/\1/'`; \
test -z "$$MAKEFILENAME" -a -f Makefile && MAKEFILENAME=Makefile; \
test -z "$$MAKEFILENAME" -a -f makefile && MAKEFILENAME=makefile; \
export MAKEFILENAME; \
$(MAKE) -e -f $$MAKEFILENAME target1;
Can be a bit simplified if make is invoked only for all targets.
MAKEFILENAME = invoked_makefile_placeholder
all: target1 target2...
target1: $(MAKEFILENAME) other_dependencies...
#test $(MAKEFILENAME) == invoked_makefile_placeholder && exit 0; \
built_instructions_for_target1;
invoked_makefile_placeholder:
#pid=$$$$; \
while test `ps -ocomm= $$pid` != make; do \
pid=`ps -oppid= $$pid`; \
done; \
MAKEFILENAME=`ps -oargs= $$pid|sed 's/^.* -f *\([^ ]*\).*$$/\1/'`; \
test -z "$$MAKEFILENAME" -a -f Makefile && MAKEFILENAME=Makefile; \
test -z "$$MAKEFILENAME" -a -f makefile && MAKEFILENAME=makefile; \
export MAKEFILENAME; \
$(MAKE) -e -f $$MAKEFILENAME
With the previous approach is trivial to implement a solution for included makefiles based in grep and a unique pattern contained in the makefile.
I never answer when I feel the question got a proper solution.