Here is my Makefile:
.PHONY: test%
test1:
# jobserver is UNavailable
make -C sub
test2:
# jobserver is available, ok
+make -C sub
test3:
# jobserver is available, ok
$(MAKE) -C sub
test4:
# jobserver is available, ok
+$(MAKE) -C sub
sub is sub-directory that contains another Makefile (sub-make).
When I run test1 rule:
$ make -j8 test1
make -C sub
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
I get warning that jobserver is unavailable and sub/Makefile is really run in single thread (as if -j1).
They say I should add + and so I run a test2 target which contains + before make command. And now I don't see the warning and sub/Makefile is really run in parallel. But according to this answer, the + sign is not for running in parallel but for forcing running commands even if make is run with -n, -t, -q flags. But why does + enables jobserver?
When I run test3 target that doesn't use + but uses $(MAKE) for running sub/Makefile, it doesn't give jobserver warning as well (parallel execution works). So what is difference between make and $(MAKE)? I thought it is just for allowing to substitute default make with user-defined make. When I don't override MAKE variable, I see the same make command as I see in the test1 target. But why does the $(MAKE) enables jobserver and make does not?
Running test4 target does not give jobserver warning also (works in parallel).
Please note my question is different from this one. It is about cmake, my question is about make. There is also related question, but it doesn't answer my questions.
The GNU make manual have decent explanation for this error. The point is: 'make' won't pass information about jobserver to the called process unless it is sure that called process is 'make' too.
‘warning: jobserver unavailable: using -j1. Add `+' to parent make
rule.’
In order for make processes to communicate, the parent will pass
information to the child. Since this could result in problems if the
child process isn’t actually a make, the parent will only do this if
it thinks the child is a make. The parent uses the normal algorithms
to determine this (see How the MAKE Variable Works). If the makefile
is constructed such that the parent doesn’t know the child is a make
process, then the child will receive only part of the information
necessary. In this case, the child will generate this warning message
and proceed with its build in a sequential manner.
Section How the MAKE Variable Works referenced in error description specifies two ways of telling 'make' that the invoked process is another instance of 'make': using $(MAKE) or +.
It states that:
One should use $(MAKE) variable when calling 'make' in the recipe.
Recursive make commands should always use the variable MAKE, not the
explicit command name ‘make’, as shown here:
subsystem:
cd subdir && $(MAKE)
Using $(MAKE) and placing a + before the line that invokes 'make' in the recipe have same effect.
Using the MAKE variable has the same effect as using a ‘+’ character at the beginning of the recipe line.
Magic happens only if you explicitly type MAKE in the recipe. If this is not the case, use +.
This special feature is only enabled if the MAKE variable appears directly in the recipe: it does not apply if the MAKE variable is referenced through expansion of another variable. In the latter case you must use the ‘+’ token to get these special effects.
Related
I have such Makefile with a content for creating a script:
.PHONY cluster-run
cluster-run:
make $(TARGET) --just-print >> tmp_script.sh;
And another one nn.mk:
.PHONY nn-model
include Makefile
nn-model:
python run-nn.py
I have two separate Makefiles for readability, because their content is big and I have another '*.mk' files, like nn-lstm.mk, nn-conv.mk, etc.
I launch as follows:
make -f nn.mk cluster-run TARGET=nn-model
But make gives an error:
make nn-model --just-print >> tmp_script.sh;
make[1]: *** No rule to make target `nn-model'. Stop.
make: *** [cluster-run] Error 2
For me such behaviour is strange because target nn-model actually exists. How can I fix this problem?
First you should never use raw make in recipes. Always use the $(MAKE) variable.
Second, the problem is because when you run the sub-make you don't provide the -f option:
make nn-model --just-print >> tmp_script.sh;
Because of that, it reads Makefile but not nn.mk, and so there's no rule to build the target nn-model.
Remember if you run a sub-make like this it's starting an entirely new make process with a clean slate: none of the targets defined in the parent make process are known to the sub-make when it starts.
I don't know what you mean by target nn_model actually exists but there's definitely no file named nn_model or you wouldn't get that error.
So what's happening is that when you build cluster-run it invokes a recursive make, which reads Makefile, and asks it to build $(TARGET) (which will include nn-model).
Notice that the recursive make is a new make and does not inherit variables or rules from the parent make, so this make instance has no clue how to build nn-model If you want the child make to see this, then the child make must include the parent one...
I am unable to prevent make from communicating any variables to a submake. I've read the manual and I've followed their advice (resetting MAKEOVERRIDES and MAKEFLAGS) but it's still not working has I think it should.
Consider the following prototype Makefile:
${warning $(MAKEOVERRIDES)}
${warning $(MAKEFLAGS)}
${warning $(VAR)}
none:
$(MAKE) -f Makefile MAKEOVERRIDES= MAKEFLAGS= all
all:
echo done!
If I make VAR=10 none, I get the following:
Makefile:2: VAR=10
Makefile:3:
Makefile:4: 10
make -f Makefile MAKEOVERRIDES= MAKEFLAGS= all
make[1]: Entering directory `/home/adriano/sandbox/makes'
Makefile:2:
Makefile:3:
Makefile:4: 10
echo done!
done!
make[1]: Leaving directory `/home/adriano/sandbox/makes'
Meaning that make is communication VAR to the submake. Is this the correct behaviour?
I've tried unexport VAR and bash -c make ... without any luck.
EDIT: I've modified none's recipe to: bash -c "echo $$MAKEOVERRIDES $$MAKEFLAGS $$VAR" ; make ...
This way I found out that VAR is actually being passed through the environment that make creates for the commands to be executed and not through the other variables (the other variables are also passed this way to make).
I think my question now is: how can I create a fresh shell/environment to run my sub make?
EDIT: Someone asked why am I trying to this; I'll try to answer to that here.
I have a "module" which uses a variable named CONFIG. In order to build this module I need to build another partially unrelated "module" which also uses CONFIG, but with a different value. The problem is that when I try to build the "sub-module" CONFIG contains the value of the "super-module." I could specify CONFIG when making the "sub-module" however both modules use many variables with the same name and trying to specify them all would make the modules tightly coupled which is something I cannot afford.
How can this be so difficult...
This is wrong:
none:
$(MAKE) -f Makefile MAKEOVERRIDES= MAKEFLAGS= all
These variables (MAKEOVERRIDES and MAKEFLAGS) are set in the environment by the parent make to be passed down to the sub-makes. Setting overrides on these values inside the recipe won't help, because make has to set the environment for the recipe before it actually starts the commands in the recipe (of course).
You have to override/remove these values in the parent makefile, so that those changes are seen by the parent make before it constructs the sub-make's environment:
MAKEOVERRIDES =
none:
$(MAKE) -f Makefile all
There's no perfect way to do this. However, you can play a trick that will work most of the time:
unexport $(shell echo '$(MAKEOVERRIDES)' | sed 's/=[^ ]*//g')
MAKEOVERRIDES =
The first line tries to unexport all the variables in MAKEOVERRIDES and the second line resets MAKEOVERRIDES. There are a few issues with this. One is that if MAKEOVERRIDES is empty, it will use "unexport" by itself which unexports everything. That can be easily worked around by sticking some bogus variable before the shell function. The other is that if any variable's value contains whitespace, the expansion will consider it a variable to be unexported. That's probably OK, but it's odd.
I can't think of any better way to do it.
You don't really say why you want to do this. Have you considered doing something different, such as running the commands where you want to have a "vanilla" environment using env; for example if you want to run a command with a limited and specific set of env vars, you can run:
test:
env -i PATH='$(PATH)' LANG='$(LANG)' runMyCommand --with --my arguments
Unfortunately some versions of env use - instead of -i; check your man page.
Alternatively, you can try to start a login shell which will re-read the user's shell setup environment from scratch:
test:
/bin/sh -lc 'runMyCommand --with --my arguments'
EDIT: It's difficult because what you're asking to do (restrict the environment of the sub-make) is tricky.
Luckily based on your description, it doesn't seem necessary. Make has a hierarchy of importance for finding variable values. The command line is the highest level (well, there's override but we'll ignore that). After that comes variables set in the makefile itself. And last and lowest comes variables imported from the environment (well, default variables are even lower but we'll ignore that too).
So if your goal is to allow the variables in the sub-makes to not be affected by command line variables given to the upper-level makes, then all this rigmarole of getting the variables out of the environment is not necessary. Variables set in the sub-makefiles will take precedence over the values in the environment. So all you have to do is get rid of the variables set on the command line, which I've already shown how to do above, by setting MAKEOVERRIDES.
I have a makefile which calls multiple other makefiles.
I'd like to pass the -j param along to the other makefile calls.
Something like (make -j8):
all:
make -f libpng_linux.mk -j$(J)
Where $(J) is the value 8 from -j8. I absolutely swear I've done this before but I cannot locate my example.
$(MAKEFLAGS) seems to contain --jobserver-fds=3,4 -j regardless of what -j2 or -j8
Edit: Possible Solution:
Will post this as an answer soon.
It appears one solution to not worry about it. Include -j8 when you call the main makefile. The sub calls to make should look like this:
all:
+make -f libpng_linux.mk -j$(J)
Notice the "+" in front of make. I noticed make tossing a warning when I tried parallel builds: make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
Only certain flags go into $(MAKEFLAGS). -j isn't included because the sub-makes communicate with each other to ensure the appropriate number of jobs are occuring
Also, you should use $(MAKE) instead of make, since $(MAKE) will always evaluate to the correct executable name (which might not be make).
"Do not do that" is not always the answer, but in this case it is, at least for GNU make.
GNU make parent process has an internal jobserver. If top-level Makefile is run with -j, subprocess makes will talk to the jobserver and read a parallelism level from it, without an explicit -j.
Ongoing coordination with parent's jobserver is much better for core utilization. For example, during the same build with -j6, parent could be running 2 jobs and the child 4 more, next moment both could be running 3 jobs each, then a parent would run 1 and the child 5.
Most Linux apps are compiled with:
make
make install clean
As I understand it, the make command takes names of build targets as arguments. So for example install is usually a target that copies some files to standard locations, and clean is a target that removes temporary files.
But what target will make build if no arguments are specified (e.g. the first command in my example)?
By default, it begins by processing the first target that does not begin with a . aka the default goal; to do that, it may have to process other targets - specifically, ones the first target depends on.
The GNU Make Manual covers all this stuff, and is a surprisingly easy and informative read.
To save others a few seconds, and to save them from having to read the manual, here's the short answer. Add this to the top of your make file:
.DEFAULT_GOAL := mytarget
mytarget will now be the target that is run if "make" is executed and no target is specified.
If you have an older version of make (<= 3.80), this won't work. If this is the case, then you can do what anon mentions, simply add this to the top of your make file:
.PHONY: default
default: mytarget ;
References:
https://www.gnu.org/software/make/manual/html_node/How-Make-Works.html
GNU Make also allows you to specify the default make target using a special variable called .DEFAULT_GOAL. You can even unset this variable in the middle of the Makefile, causing the next target in the file to become the default target.
Ref: The Gnu Make manual - Special Variables
bmake's equivalent of GNU Make's .DEFAULT_GOAL is .MAIN:
$ cat Makefile
.MAIN: foo
all:
#echo all
foo:
#echo foo
$ bmake
foo
See the bmake(1) manual page.
I'm trying to glue two build systems together. Both are recursive (rules in the makefile use make to call other makefiles to build components of the project).
I'll call them 'A' and 'B' where 'A' builds the application and 'B' builds libraries used by 'A'.
The top level makefile in A calls 'make TARGET=whatever' which means that all the recursively-invoked bits of the build inherit the value of TARGET as a read-only variable, including the build system from B, which is called as part of the recursive build.
I don't want this to happen in the build system for 'B' (which come from a different project) as the makefiles there use TARGET for their own purposes and the build fails since TARGET has the wrong value and is read-only.
I can only see two solutions to this, neither of which is palettable;
1) Rename TARGET to something else in the makefile in A that sets it and in the makefiles in A that use it, to avoid the clash with the lower levels of the build system.
2) Use the 'override' directive everywhere in the makefiles in B where the TARGET variable is set, to override its read-only status.
Anyone got any better ideas? - ideally, I want nothing to be inherited by the B's build system from A's, except those options I explicitly pass to the B build system from A.
Incidentally, I'm using GNU Make v3.80.
You could set MAKEOVERRIDES to nothing in the second-level makefile in A.
callb:
cd subdir && $(MAKE) MAKEOVERRIDES=
This passes down the normal commandline parameters like -k and -s but not commandline variable definitions.
Or you use the historical MFLAGS which is the same as MAKEFLAGS except MFLAGS doesn't contain the commandline variable definitions.
callb:
cd subdir && $(MAKE) $(MFLAGS)
Details about this two options can be read here: The GNU Make Manual
Perhaps you can use the "unexport" directive to prevent TARGET from being propagated to B's makefile?
At the point where build system A invokes build system B, do not use '${MAKE}' directly; invoke a shell script that invokes build system B (possibly after sanitizing the environment).
To achieve the behaviour where the commands are executed by 'make -n', prefix the command line in the makefile with '+' (similar to prefixing the line with '#' or '-').
It sounds like you have modified the A makefile to recursively invoke the B makefile, and thus your problem. Why not instead introduce a new toplevel makefile which recursively invokes the B makefile, and then recursively invokes the A makefile? For example, combined.mk:
all:
$(MAKE) -f Makefile.B
$(MAKE) -f Makefile.A
That way the B makefile inherits nothing from the A makefile.