I want to set some variables based on whether or not parallel builds are enabled, so I tried this:
jobs:
»·echo "executing jobs job"
ifneq (,$(findstring -j,$(MAKEFLAGS)))
»·$(warning "parallel!")
else
»·$(warning "not parallel!")
endif
And this is what happens:
$ make -j2
Makefile:2: "not parallel!"
echo "executing jobs job"
executing jobs job
I also tried testing $(JOBS), but no luck.
Is there a way for me to tell inside a Makefile that the --jobs parameter was used?
Additional info:
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for x86_64-pc-linux-gnu
Surprisingly, ${MAKEFLAGS} will only gain the -j when it is expanded at recipe expansion time.
Makefile:
$(warning [${MAKEFLAGS}])
.PHONY: all
all:
$(warning [${MAKEFLAGS}])
echo Now do something useful
Run:
$ make -j5
1:1: []
1:5: [ -j --jobserver-fds=3,4]
echo Now do something useful
Now do something useful
About the MAKEFLAGS expansion in #bobbogo's answer: If we look at the code I think I can explain the behavior:
Looking at the code, main function of make calls the define_makeflags function several times.
/* Define the MAKEFLAGS and MFLAGS variables to reflect the settings of the
command switches. Include options with args if ALL is nonzero.
Don't include options with the 'no_makefile' flag set if MAKEFILE. */
static struct variable *
define_makeflags (int all, int makefile)
{
......
Call locations in main:
1)
/* Set up the MAKEFLAGS and MFLAGS variables for makefiles to see.
Initialize it to be exported but allow the makefile to reset it. */
define_makeflags (0, 0)->export = v_export;
2)
/* Set up MAKEFLAGS and MFLAGS again, so they will be right. */
define_makeflags (1, 0);
3)
/* Set up 'MAKEFLAGS' specially while remaking makefiles. */
define_makeflags (1, 1);
There are other calls in sub-functions but this should be enough to explain.
The first call sets all parameter to false. The others set to true. With all set to false, the define_makeflags function only parses "simple flags" and j is not one of them. In order to understand the parsing one needs to look into this switch statement and the definition of command line params.
My SWAG is like the following:
I presume the parsing of ifneq statements happen after the first call to define_makeflags but before the subsequent calls. I can guess the reason of keeping the MAKEFLAGS simple at the start is to continue to support documented Makefile patterns like the following.
From doc1, doc2:
archive.a: ...
ifneq (,$(findstring t,$(MAKEFLAGS)))
+touch archive.a
+ranlib -t archive.a
else
ranlib archive.a
endif
If MAKEFLAGS contained long options or options that take parameters, then searching for single char flags in MAKEFLAGS would not be possible.
There is some guesstimate in my answer. Maybe someone who was involved in the design decision can also weigh in. Given this change Paul Smith may have an idea.
Related
I'm trying to use find_j=$(findstring j,$(filter-out --%,$(MAKEFLAGS))) to find if there is -j option, so when I echo $(find_j) the value is j
but when I compare it ifeq (j, $(find_j)) this returnes false
I cant understand where is the problem
my version of make is make-3.99.90
find_j=$(findstring -j,$(filter-out --%,$(MAKEFLAGS)))
ifneq ( , $(find_j))
PARALLEL_ENABLED=true
endif
.PHONY: PRINT
PRINT:
$(info $(PARALLEL_ENABLED))
$(info $(MAKEFLAGS))
$(info $(find_j))
---empty line---
--warn-undefined-variables -ws --jobserver-fds=5,6 -j
-j
One thing to note is that the release of GNU make you're using is a beta release of GNU make 4.0, which itself was released in 2013... so you're using a beta of a release that itself is 8.5 years old.
However, that's not related to this problem.
The issue is that the MAKEFLAGS variable's final value is not set until after all makefiles are parsed. If you try to examine it before all makefiles are parsed, it will contain only a subset of the total set of options.
When you expand that variable as part of an ifeq or ifneq statement, that happens as the makefile is being parsed and so (as per the above) only the simple options (ones that don't take an argument: -j accepts an argument so is not "simple") are available.
When you expand the variable as part of a recipe, that happens after all makefiles are parsed: at that time the final value of MAKEFLAGS is set. So your $(info ...) functions inside the recipe do the right thing.
This is easy to see:
$(info no recipe MAKEFLAGS is '$(MAKEFLAGS)')
all: ; $(info in recipe MAKEFLAGS is '$(MAKEFLAGS)')
If you run with -j10 you'll get:
no recipe MAKEFLAGS is ''
in recipe MAKEFLAGS is '-j10 --jobserver-auth=3,4'
(your "in recipe" flags might look different because you're using such an old version of GNU make).
In the next release of GNU make, the value of MAKEFLAGS is kept up-to-date constantly so you can check it at any time and it will be accurate. But that release is not available yet.
This seems to be imprecisely documented. While MAKEFLAGS has the flags like e.g. -s and -k as ks in it, the -j flag gets processed in another way: it is not stripped of the leading dash - AND it is not visible in the first pass of processing the makefile. Only when rules are executed, MAKEFLAGS receives a value, albeit a processed form of the one you gave. -j3 elicits a -j3 --jobserver-auth=3,4 response from the command line transcriber of make, while -j stays -j. So what does this mean for us? Obviously the feature to detect the requested parallelism at runtime is not stable or there are some good reasons not to access them (which is the case most of the time when you encounter exceptional behaviour in GNU tools), so maybe you can give us more information on what you are trying to achieve - maybe there is a way to circumvent accessing the command line.
I have a rule something, that works on the variable VAR. I also have another rule something-all, that needs to run something, with VAR set to each value in vars.
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach VAR,$(vars),something)
This doesn't quite work, I get
noob#work:~/Desktop$ make something-all
something something
make: something: No such file or directory
make: *** [something-all] Error 1
It should probably print hello\nworld.
I used to do this with wildcard rules by retrieving VAR from %, but got the feeling that was the wrong way to do it. This looked like this:
vars = hello world
all: $(foreach VAR,$(vars),something-$(VAR))
something-%:
echo $*
The below should fix your problem
Using foreach (Tried on GNU Make 3.80 on sparc-solaris 2.8 and windows)
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach i, $(vars), $(MAKE) something VAR=$i || exit 1;)
Using shell for-loop (Tried on GNU Make 3.80 and cc make on sparc-solaris 2.8)
vars = hello world
something:
echo $(VAR)
something-all:
for i in $(vars); do $(MAKE) something VAR=$$i || exit 1; done
TL;DR: If you want to program make, drop GNU Make in favor of BSD Make.
This is a personal recommendation. While BSD Make seems more limited than GNU Make, as it offers less programming facilities, it is much easier to program and has a few unique killer features. This is why I propose a solution with GNU Make and another solution for BSD Make:
Doing it in GNU Make
Using GNU Make, you can write a macro to define a target. The canonical way to define a sequence in a Makefile is to add the steps of the sequence as dependencies to a target, as reflected by the snippet below:
vars= hello world
define something_t =
something: something-$(1)
something-$(1):
#echo $(1)
endef
$(foreach _,$(vars),$(eval $(call something_t,$_)))
It is recommended to use this organisation (rather than defining just one target), because you can work on it to make the task easily resumable if you interrupt the sequence. A Makefile describes a job whose advancement is entirely described by the state of the file system. A task is then easily resumable, if each step is associated to a file, usually a compilation object but sometimes also an empty file which is touch'ed to indicate that important checkpoints have been passed.
Using an auxiliary macro is a flexible solution that can be adapted to more complicated tasks than just echoing a name. Note that this does work with newest versions of GNU Make (4.1). On GNU Make 3.81, you should remove the equal sign from the macro definition.
Adapting your example for BSD Make
If this is an option for you, I recommand dropping the use of GNU Make and replace it by BSD Make, which is way easier to program: it has a short and to the point documentation, while the documentation of GNU Make is very verbose and somewhat unclear, BSD Make has industrial-strength examples of complex rulesets (FreeBSD Build system or BSD Owl), and it has a simple and predictable macro language.
vars= hello world
something:
.for _var in ${vars}
echo ${_var}
.endfor
This can evolve to support more complicated tasks, just by replacing the echo by the adapted commands, or using intermediary steps.
Allow the user to override some tasks, also in BSD Make
In this slightly more advanced variation, we allow the user to override our own recipes for building targets something-hello and something-world.
For each item in our list, a target something-* is created it if it does not already exist, and added to the dependencies of something. The whole operation of defining these targets only happens if something has been left undefined. Therefore, users of these macros can:
Override the recipes for something-hello and something-world
Override the full procedure bound to something.
Implementing such customisation possibilities is mandatory if we want to write useful, reusable, macros for Make. Unluckily, customisation of this sort is nearly impossible in GNU Make.
vars = hello world
.if!target(depend)
.for _var in ${vars}
.if!target(something-${_var})
something-${_var}:
echo ${_var}
.endif
something: something-${_var}
.endfor
.endif
Here's one way to do it:
VARS := hello world
THINGS := $(addprefix something-, $(VARS))
allthings: $(THINGS)
something-%:
echo $*
It should be no surprise that
vars := hello world
something-all:
$(foreach VAR,$(vars),something)
tries to run something something. That's exactly what the foreach expands to, since you don't reference VAR in the third expression.
All you need to do is reference VAR and use a command such as echo:
vars := hello world
something-all:
$(foreach VAR,$(vars),echo $(VAR);)
$ make
echo hello; echo world;
hello
world
Note how chaining the commands with a semicolon avoids forking several shells or -- GASP! -- recursive make invocations. It doesn't get more performant than that.
Alternatively, if your command accepts several somethings as arguments,
vars := hello world
something-all:
echo $(foreach VAR,$(vars),$(VAR))
$ make
echo hello world
hello world
But that is equivalent to the super simple echo $(vars). So it might pay off to think outside the box trying to change your requirements to make this simple solution work.
I’m trying to implement include guards in Gnu Make. In this Makefile, the first inclusion is OK, while the second one fails with an error.
ifndef INCLUDED
INCLUDED = 1
$(info Including)
define macro
ifneq ($(1),)
define inner_macro
macro content...
endef
else
define inner_macro
endef
endif
endef
endif
The same effect can be simulated by explicitly giving INCLUDED = 1 before the inclusion, e.g. on command line.
Gnu Make 4.1 under Gentoo says Makefile:14: *** missing separator. Stop., while Gnu Make 3.81 under Debian Wheezy says Makefile:14: *** extraneous `endef'. Stop.. On the first inclusion, they both say:
Including
make: *** No targets. Stop.
If I try $(eval $(call macro,whatever)) after the first inclusion, it defines inner_macro as expected.
I used make INCLUDED=1 and make commands respectively to get the described behavior.
The same happens when I clear the environment and disable built-in rules and variables: env -i make -rR INCLUDE=1. When I use -p to dump the database, without INCLUDED=1, the macro is defined as it should be, but with INCLUDED=1, empty inner_macro is defined. This is consistent across both the versions of Make. This hints me that when the condition is false, Make parses the Makefile differently and thinks the else inside macro’s definition belongs to the ifndef. Other condition types behave all the same.
If I remove both the definitions of inner_macro, the problem goes away.
I read the manual pages info make conditional\ syntax and info make multi-line (formerly defining), but I found no caveat there and I still think I am doing nothing wrong.
Am I correct with my conclusions?
Is this a bug in Make, or am I invoking undefined behavior?
How should I implement include guards in Gnu Make?
That's a bug. Report it on Savannah.
There's something wrong with the tracking of nested define/endef inside a not-taken ifdef/ifndef condition. If you don't use nested define/endef then it works; for example (obviously you may not be able to do this in your environment):
ifndef INCLUDED
INCLUDED = 1
$(info Including)
define macro
ifneq ($(1),)
inner_macro = macro content...
else
inner_macro =
endif
endef
endif
How to show the makefile after it's been preprocessed? For example, if we have two makefiles:
# Makefile
include Makefile2
# Makefile2
a:a.c
gcc -o a a.c
Then <preprocessor> Makefile should give:
a:a.c
gcc -o a a.c
It's similar to what a C preprocessor does (gcc -E). Is there such a makefile preprocessor?
You didn't specify for which make tool you are writing makefile. Assuming that it is GNU make, you can try running makefile with -n (--just-print) option See Command-Line Options chapter here. That will show what make is going to execute without execution (however, the commands needed for evaluation of variables will be executed). This is probably the closest to what you want to see.
This causes make to read the makefile and print every command it would
normally execute to update the target but without executing them.
Apart from that there is $(warning ) function to debug makefiles. You can place it almost to any part in makefile and the following will show you the values of all defined variables in that place:
$(warning Variables HERE: .VARIABLES)
I have a compile job where linking is taking a lot of IO work. We have around a dozen of cores so we run make -j13, but when it comes to linking the 6 targets, I'd like those to be done in a round robin way. I thought about making one depend on the next but I think this would break the individual targets. Any ideas how to solve this small issue?
make itself doesn't provide a mechanism to request "N of these, but no more than M of those at a time".
You might try using the sem command from the GNU parallel package in the recipe of your linker rules. Its documentation has an example of ensuring only one instance of a tool runs at once. In your example, you would allow make to start up to 13 sems at a time, but only one of those at a time will run the linker, while the others block.
The downside is that you could get into a situation where 5 of your make's 13 job slots are tied up with instances of sem that are all waiting for a linker process to finish. Depending on the structure of your build, that might mean some wasted CPU time. Still beats 6 linkers thrashing the disk at once, though :-)
You should specify that your six targets cannot be built in parallel. Add a line like this to your makefile:
.NOTPARALLEL: target1 target2 target3 target4 target5 target6
For more information look here https://www.gnu.org/software/make/manual/html_node/Parallel-Disable.html.
I've stumbled upon a hacky solution:
For each recipe it runs, Make does two things: it expands variables/functions in the recipe, and then runs the shell commands.
Since the first step can read/write the global variables, it seems to be done synchronously.
So if you run all your shell commands during the first step (using $(shell )), no other recipe will be able to start while they're running.
E.g. consider this makefile:
all: a b
a:
sleep 1
b:
sleep 1
time make -j2 reports 1 second.
But if you rewrite it to this:
# A string of all single-letter Make flags, without spaces.
override single_letter_makeflags = $(filter-out -%,$(firstword $(MAKEFLAGS)))
ifneq ($(findstring n,$(single_letter_makeflags)),)
# See below.
override safe_shell = $(info Would run shell command: $1)
else ifeq ($(filter --trace,$(MAKEFLAGS)),)
# Same as `$(shell ...)`, but triggers a error on failure.
override safe_shell = $(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(.SHELLSTATUS)))
else
# Same functions but with logging.
override safe_shell = $(info Shell command: $1)$(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(>
endif
# Same as `safe_shell`, but discards the output and expands to nothing.
override safe_shell_exec = $(call,$(call safe_shell,$1))
all: a b
a:
$(call safe_shell_exec,sleep 1)
#true
b:
$(call safe_shell_exec,sleep 1)
#true
time make -j2 now reports 2 seconds.
Here, #true does nothing, and suppresses Nothing to be done for ?? output.
There are some problems with this approach though. One is that all output is discarded unless redirected to file or stderr...
It won't break individual targets.
You can create any number of (:) rules for a target, as long as only one of them has an actual recipe for building it. This appears to be a good use case for that.