I've defined the following rule:
$(OUTDIR)/%_sorted.bam:
$(eval SAMPLE:=$(subst _sorted,,$(notdir $(basename $#))))
(...) do something with $(SAMPLE)
can I use it safely (= is it a "local variable" to the rule ?) with
make -j (number of jobs)
or does the variable $(SAMPLE) will be replaced for each (simultaneous) invocation of the rule ?
Yes, you can. GNU make expands the whole recipe prior to executing it. Job server mode only affects the execution of sub-processes spawned by make, everything else is single threaded.
Related
Here is my Makefile:
.PHONY: test%
test1:
# jobserver is UNavailable
make -C sub
test2:
# jobserver is available, ok
+make -C sub
test3:
# jobserver is available, ok
$(MAKE) -C sub
test4:
# jobserver is available, ok
+$(MAKE) -C sub
sub is sub-directory that contains another Makefile (sub-make).
When I run test1 rule:
$ make -j8 test1
make -C sub
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
I get warning that jobserver is unavailable and sub/Makefile is really run in single thread (as if -j1).
They say I should add + and so I run a test2 target which contains + before make command. And now I don't see the warning and sub/Makefile is really run in parallel. But according to this answer, the + sign is not for running in parallel but for forcing running commands even if make is run with -n, -t, -q flags. But why does + enables jobserver?
When I run test3 target that doesn't use + but uses $(MAKE) for running sub/Makefile, it doesn't give jobserver warning as well (parallel execution works). So what is difference between make and $(MAKE)? I thought it is just for allowing to substitute default make with user-defined make. When I don't override MAKE variable, I see the same make command as I see in the test1 target. But why does the $(MAKE) enables jobserver and make does not?
Running test4 target does not give jobserver warning also (works in parallel).
Please note my question is different from this one. It is about cmake, my question is about make. There is also related question, but it doesn't answer my questions.
The GNU make manual have decent explanation for this error. The point is: 'make' won't pass information about jobserver to the called process unless it is sure that called process is 'make' too.
‘warning: jobserver unavailable: using -j1. Add `+' to parent make
rule.’
In order for make processes to communicate, the parent will pass
information to the child. Since this could result in problems if the
child process isn’t actually a make, the parent will only do this if
it thinks the child is a make. The parent uses the normal algorithms
to determine this (see How the MAKE Variable Works). If the makefile
is constructed such that the parent doesn’t know the child is a make
process, then the child will receive only part of the information
necessary. In this case, the child will generate this warning message
and proceed with its build in a sequential manner.
Section How the MAKE Variable Works referenced in error description specifies two ways of telling 'make' that the invoked process is another instance of 'make': using $(MAKE) or +.
It states that:
One should use $(MAKE) variable when calling 'make' in the recipe.
Recursive make commands should always use the variable MAKE, not the
explicit command name ‘make’, as shown here:
subsystem:
cd subdir && $(MAKE)
Using $(MAKE) and placing a + before the line that invokes 'make' in the recipe have same effect.
Using the MAKE variable has the same effect as using a ‘+’ character at the beginning of the recipe line.
Magic happens only if you explicitly type MAKE in the recipe. If this is not the case, use +.
This special feature is only enabled if the MAKE variable appears directly in the recipe: it does not apply if the MAKE variable is referenced through expansion of another variable. In the latter case you must use the ‘+’ token to get these special effects.
I am trying to get the target_compile to work.
copy_shared_object:
cp shared_object.so ${CURRENT_DIR}
PROJECT_SIM_OPTS += -LDFLAGS -L${CURRENT_DIR},-lm -load
target_compile: copy_shared_object actual_compile_with_sim_opts
.
.
.
actual_compile_with_sim_opts:
.
.
.
I am getting the Error despite the fact that I have not added ;\ on the first line starting with cp
make: PROJECT_SIM_OPTS: Command not found
makefile:282: recipe for target 'copy_shared_object' failed
make: *** [copy_shared_object] Error 127
What you likely want is something like:
${CURRENT_DIR}/shared_object.so: shared_object.so
cp $^ $#
target_compile: PROJECT_SIM_OPTS += -LDFLAGS -L${CURRENT_DIR},-lm -load
target_compile: copy_shared_object actual_compile_with_sim_opts
#echo PROJECT_SIM_OPTS=${PROJECT_SIM_OPTS} ...
To explain a few things (and to reiterate #Beta's remarks): The variable ${CURRENT_DIR} is a makefile variable. It can come from either the environment or makefile. make will substitute the value for the variable name at its first phase (before it runs any rules). Therefore its value cannot be changed when running a rule. Makefile variables have a single $, and require braces around them if they're multi-character tokens.
${PROJECT_SIM_OPTS} is a target-specific makefile variable. It's still a makefile variable, so it cannot change its value when the make is executing the rules. That being said, its value is specific to the target_compile rule, and any rule that is being run as a result of that rule.
For shell variables, it's possible to set a value within a recipe, however, the scope of that value is that recipe line itself. In order to use shell variables you need to do $$shellvarname (with two $'s, as make expands $$ to $ before invoking the shell) That being said, each line of a recipe is run in a subshell, and any variable values will not be visible in other subshells. So, for example, if you have:
target: prereq
#shellVar="value"; echo "recipe1: shellVar is $$shellVar"
#echo "recipe2: shellVar is $$shellVar"
it will output:
recipe1: shellVar is value
recipe2: shellVar is
as recipe1's subshell does not communicate with recipe2's subshell, and therefore recipe2 is not aware of recipe1's value for the variable.
I have a few software projects which are distributed as RPMs. They are versioned using semantic versioning to which we affix a release number. Using the regular conventions, this is MAJOR.MINOR.PATCH-REL_NUM. Though beyond the scope of this article, the release numbers are stored in git. The release target in the makefile looks something like this:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
# Although the third step, this was re-ordered to step 1
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
While debugging, I eventually discovered that, although the call to eval was the third step in the recipe, it was actually being evaluated first! This is why the RPM always had a release number one less than the number I was watching get pushed to the remote.
I have done much googling on this and I haven't found any hits that explain the order of evaluation with regard to eval when used in recipes. Perhaps it isn't even with respect to eval but functions in general. Furthermore, I haven't found verbiage on this in the GNU manuals for make either (if it's there, kindly point out what chapter). I've worked around the problem so it's not a bother, I'm just wondering, is this expected and if so, why?
The missing bit, that no one above is getting, is simple: when make is going to run a recipe it expands all lines of the recipe first, before it starts the first line. So:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
# Although the third step, this was re-ordered to step 1
$(eval RELEASE_NUMBER=$(shell $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
when make decides to run the release target it first expands all the lines in the recipe, which means the eval is expanded, then it runs the resulting lines. That's why you're getting the behavior you're seeing.
I don't really see why you need to use eval here at all; why not just use:
release:
$(MAKE) clean
$(BLD_ROOT)/tools/incr_rel_num
$(MAKE) rpm RPM_RELEASE_NUM="$$(cat $(BLD_ROOT)/path/to/rel_num.txt))"
(BTW, you should never use bare make inside your makefiles; you should always use $(MAKE) (or ${MAKE}, same thing).
The $(eval ...) function
generates a fragment of make-sytax which becomes part of the parsed makefile.
The makefile is parsed entirely before any recipes are executed and when recipes
are executed all make-statements, make-expressions and make-variables have been
evaluated away.
So it does not make sense to consider an $(eval ...) call as being one
of the lines of a recipe. It might generate values that are used in the make-expansion
of the recipe, but if so then this happens when the makefile is parsed, before the recipe is run.
Thus in your example, the line:
$(eval RELEASE_NUMBER=$(shell $(BLD_ROOT)/path/to/rel_num.txt))
which I assume should really be:
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
is evaluated when the makefile is parsed, and let's say it results in the
make-variable RELEASE_NUMBER acquiring the value 1.0, because, when the
makefile is parsed, the file $(BLD_ROOT)/path/to/rel_num.txt) contains
1.0. In that case your recipe:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
will resolve to the like of:
release:
make clean
some_build_dir/tools/incr_rel_num
make rpm RPM_RELEASE_NUM=1.0
You will observe when make runs the recipe that it prints no line that
is "the expansion of" $(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt)),
because there is no such thing in the recipe. It doesn't matter that:
some_build_dir/tools/incr_rel_num
is presumably a command that writes, say, 1.1 or 2.0 in the file some_build_dir/path/to/rel_num.txt.
That action simply has no effect on the recipe. Nothing that executed in the recipe
can change the recipe.
$(eval ...) has no business in your recipe. What you want to achieve is simply:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
RELEASE_NUMBER=$$(cat $(BLD_ROOT)/path/to/rel_num.txt) && \
make rpm RPM_RELEASE_NUM=$$RELEASE_NUMBER
where $$ is what you do in a makefile to escape $ and, in this case,
leave it for the shell when the recipe is executed.
This recipe expands to 3 shell commands executed in sequence:
$ make clean
$ some_build_dir/tools/incr_rel_num
$ RELEASE_NUMBER=$(cat some_build_dir/path/to/rel_num.txt) && \
make rpm RPM_RELEASE_NUM=$RELEASE_NUMBER
and might as well be simplified further to:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
make rpm RPM_RELEASE_NUM=$$(cat $(BLD_ROOT)/path/to/rel_num.txt)
You are correct, there are multiple levels of evaluation. The content on what is inside eval is evaluated a first time before that the function is actually called. If you want the content of eval to be evaluated at the time eval is called, you have to escape the $ sign by putting it twice, like this :
$(eval RELEASE_NUMBER=$$(shell $(BLD_ROOT)/path/to/rel_num.txt))
To view what is really inside eval at the time it's called you can use the same syntax with info instead of eval :
$(info RELEASE_NUMBER=$$(shell $(BLD_ROOT)/path/to/rel_num.txt))
Now I'm not sure about the part which is evaluated too soon so the $ symbols that I doubled may not be the good one(s), but using the info function will help you to find the correct command.
I have a compile job where linking is taking a lot of IO work. We have around a dozen of cores so we run make -j13, but when it comes to linking the 6 targets, I'd like those to be done in a round robin way. I thought about making one depend on the next but I think this would break the individual targets. Any ideas how to solve this small issue?
make itself doesn't provide a mechanism to request "N of these, but no more than M of those at a time".
You might try using the sem command from the GNU parallel package in the recipe of your linker rules. Its documentation has an example of ensuring only one instance of a tool runs at once. In your example, you would allow make to start up to 13 sems at a time, but only one of those at a time will run the linker, while the others block.
The downside is that you could get into a situation where 5 of your make's 13 job slots are tied up with instances of sem that are all waiting for a linker process to finish. Depending on the structure of your build, that might mean some wasted CPU time. Still beats 6 linkers thrashing the disk at once, though :-)
You should specify that your six targets cannot be built in parallel. Add a line like this to your makefile:
.NOTPARALLEL: target1 target2 target3 target4 target5 target6
For more information look here https://www.gnu.org/software/make/manual/html_node/Parallel-Disable.html.
I've stumbled upon a hacky solution:
For each recipe it runs, Make does two things: it expands variables/functions in the recipe, and then runs the shell commands.
Since the first step can read/write the global variables, it seems to be done synchronously.
So if you run all your shell commands during the first step (using $(shell )), no other recipe will be able to start while they're running.
E.g. consider this makefile:
all: a b
a:
sleep 1
b:
sleep 1
time make -j2 reports 1 second.
But if you rewrite it to this:
# A string of all single-letter Make flags, without spaces.
override single_letter_makeflags = $(filter-out -%,$(firstword $(MAKEFLAGS)))
ifneq ($(findstring n,$(single_letter_makeflags)),)
# See below.
override safe_shell = $(info Would run shell command: $1)
else ifeq ($(filter --trace,$(MAKEFLAGS)),)
# Same as `$(shell ...)`, but triggers a error on failure.
override safe_shell = $(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(.SHELLSTATUS)))
else
# Same functions but with logging.
override safe_shell = $(info Shell command: $1)$(shell $1)$(if $(filter-out 0,$(.SHELLSTATUS)),$(error Unable to execute `$1`, exit code $(>
endif
# Same as `safe_shell`, but discards the output and expands to nothing.
override safe_shell_exec = $(call,$(call safe_shell,$1))
all: a b
a:
$(call safe_shell_exec,sleep 1)
#true
b:
$(call safe_shell_exec,sleep 1)
#true
time make -j2 now reports 2 seconds.
Here, #true does nothing, and suppresses Nothing to be done for ?? output.
There are some problems with this approach though. One is that all output is discarded unless redirected to file or stderr...
It won't break individual targets.
You can create any number of (:) rules for a target, as long as only one of them has an actual recipe for building it. This appears to be a good use case for that.
I am using Makefiles.
However, there is a command (zsh script) I want executed before any targets is executed.
How do I do this?
Thanks!
There are several techniques to have code executed before targets are built. Which one you should choose depends a little on exactly what you want to do, and why you want to do it. (What does the zsh script do? Why do you have to execute it?)
You can either do like #John suggests; placing the zsh script as the first dependency. You should then mark the zsh target as .PHONY unless it actually generates a file named zsh.
Another solution (in GNU make, at least) is to invoke the $(shell ...) function as part of a variable assignment:
ZSH_RESULT:=$(shell zsh myscript.zsh)
This will execute the script as soon as the makefile is parsed, and before any targets are executed. It will also execute the script if you invoke the makefile recursively.
Just make that a dependancy of one of the other targets
foo.obj : zsh foo.c
rule for compileing foo.c
zsh:
rule for running zsh script.
or alternatively, make your first target depend on it
goal: zsh foo.exe
Solution for both preprocessing and postprocessing in makefiles using MAKECMDGOALS and double colon rules.
MAKECMDGOALS are the targets listed on the command line.
First step is to get the first and last targets from the command line,
or if there are no targets listed, use the default target.
ifneq ($(MAKECMDGOALS),)
FIRST_GOAL := $(word 1, $(MAKECMDGOALS))
LAST_GOAL := $(word $(words $(MAKECMDGOALS)), $(MAKECMDGOALS))
else
FIRST_GOAL := all
LAST_GOAL := all
endif
Double colon rules allow multiple recipes for the same target executed in order. You'll have to change all command line targets to double colon rules.
#Dummy rule to set the default
.PHONY: all
all ::
#Preprocessing
$(FIRST_GOAL) ::
echo "Starting make..."
all :: normal_prerequistes
normal_recipe
other_stuff
#Postprocessing
$(LAST_GOAL) ::
echo "All done..."
There is a solution without modifying your existing Makefile (main difference with the accepted answer). Just create a makefile containing:
.PHONY: all
all:
pre-script
#$(MAKE) -f Makefile --no-print-directory $(MAKECMDGOALS) MAKE='$(MAKE) -f Makefile'
post-script
$(MAKECMDGOALS): all ;
The only drawback is that the pre- and post- scripts will always be run, even if there is nothing else to do. But they will not be run if you invoke make with one of the --dry-run options (other difference with the accepted answer).