I'm looking at the following toy Makefile to track a bug:
all: hey bye
hey bye: hi
cat hi
touch hey bye
In this scenario, I've created the hi file beforehand.
I then run make -j 2 all and get the following output:
cat hi
cat hi
hey
touch hey bye
hey
touch hey bye
Whereas if I run it with make -j 1 all, the commands get executed only once. The debug output is pretty verbose, but from what I understand, it's executing it twice because it tried to satisfy hey and bye with two different jobs.
How can I prevent this from happening? Is this correct behaviour? Why does it think running a command twice at once will make it finish faster?
The behavior is correct, because you're misunderstanding what you are telling make.
You can read the details in the GNU make manual.
This rule:
hey bye: hi
...
Does not tell make that one invocation of the recipe will build both targets. Instead, make interprets it exactly the same way as if you'd written:
hey: hi
...
bye: hi
...
That is, you've declared two completely different targets where each one would be created by a separate invocation of ....
When you run make without -j it sees that hey is out of date and runs the recipe. Then it looks at bye, and now (since the previous recipe just updated it) it's up to date so it seems to work.
But if you run with -j2, make sees hey is out of date and starts that job, then looks for more work. Now when it checks bye, the recipe for hey didn't update it yet so make thinks it's still out of date and starts another instance of the recipe.
The best way to fix this depends on the details of your actual usage, which you haven't given.
If your inputs and outputs are related by the names of the targets and prerequisite, you can write a pattern rule. Unlike explicit rules, multiple target patterns tell make that one invocation of the recipe builds all the targets:
%.x %.y : %.z
...
If that won't work and you're willing to restrict your builds to using GNU make 4.3 or better you can take advantage of the "grouped targets" feature:
hey bye &: hi
...
If that isn't feasible, you'll have to use a sentinel target, something like:
all: hey bye
hey bye: .sentinel ;
.sentinel: hi
...
#touch $#
Related
I have a few software projects which are distributed as RPMs. They are versioned using semantic versioning to which we affix a release number. Using the regular conventions, this is MAJOR.MINOR.PATCH-REL_NUM. Though beyond the scope of this article, the release numbers are stored in git. The release target in the makefile looks something like this:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
# Although the third step, this was re-ordered to step 1
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
While debugging, I eventually discovered that, although the call to eval was the third step in the recipe, it was actually being evaluated first! This is why the RPM always had a release number one less than the number I was watching get pushed to the remote.
I have done much googling on this and I haven't found any hits that explain the order of evaluation with regard to eval when used in recipes. Perhaps it isn't even with respect to eval but functions in general. Furthermore, I haven't found verbiage on this in the GNU manuals for make either (if it's there, kindly point out what chapter). I've worked around the problem so it's not a bother, I'm just wondering, is this expected and if so, why?
The missing bit, that no one above is getting, is simple: when make is going to run a recipe it expands all lines of the recipe first, before it starts the first line. So:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
# Although the third step, this was re-ordered to step 1
$(eval RELEASE_NUMBER=$(shell $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
when make decides to run the release target it first expands all the lines in the recipe, which means the eval is expanded, then it runs the resulting lines. That's why you're getting the behavior you're seeing.
I don't really see why you need to use eval here at all; why not just use:
release:
$(MAKE) clean
$(BLD_ROOT)/tools/incr_rel_num
$(MAKE) rpm RPM_RELEASE_NUM="$$(cat $(BLD_ROOT)/path/to/rel_num.txt))"
(BTW, you should never use bare make inside your makefiles; you should always use $(MAKE) (or ${MAKE}, same thing).
The $(eval ...) function
generates a fragment of make-sytax which becomes part of the parsed makefile.
The makefile is parsed entirely before any recipes are executed and when recipes
are executed all make-statements, make-expressions and make-variables have been
evaluated away.
So it does not make sense to consider an $(eval ...) call as being one
of the lines of a recipe. It might generate values that are used in the make-expansion
of the recipe, but if so then this happens when the makefile is parsed, before the recipe is run.
Thus in your example, the line:
$(eval RELEASE_NUMBER=$(shell $(BLD_ROOT)/path/to/rel_num.txt))
which I assume should really be:
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
is evaluated when the makefile is parsed, and let's say it results in the
make-variable RELEASE_NUMBER acquiring the value 1.0, because, when the
makefile is parsed, the file $(BLD_ROOT)/path/to/rel_num.txt) contains
1.0. In that case your recipe:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
will resolve to the like of:
release:
make clean
some_build_dir/tools/incr_rel_num
make rpm RPM_RELEASE_NUM=1.0
You will observe when make runs the recipe that it prints no line that
is "the expansion of" $(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt)),
because there is no such thing in the recipe. It doesn't matter that:
some_build_dir/tools/incr_rel_num
is presumably a command that writes, say, 1.1 or 2.0 in the file some_build_dir/path/to/rel_num.txt.
That action simply has no effect on the recipe. Nothing that executed in the recipe
can change the recipe.
$(eval ...) has no business in your recipe. What you want to achieve is simply:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
RELEASE_NUMBER=$$(cat $(BLD_ROOT)/path/to/rel_num.txt) && \
make rpm RPM_RELEASE_NUM=$$RELEASE_NUMBER
where $$ is what you do in a makefile to escape $ and, in this case,
leave it for the shell when the recipe is executed.
This recipe expands to 3 shell commands executed in sequence:
$ make clean
$ some_build_dir/tools/incr_rel_num
$ RELEASE_NUMBER=$(cat some_build_dir/path/to/rel_num.txt) && \
make rpm RPM_RELEASE_NUM=$RELEASE_NUMBER
and might as well be simplified further to:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
make rpm RPM_RELEASE_NUM=$$(cat $(BLD_ROOT)/path/to/rel_num.txt)
You are correct, there are multiple levels of evaluation. The content on what is inside eval is evaluated a first time before that the function is actually called. If you want the content of eval to be evaluated at the time eval is called, you have to escape the $ sign by putting it twice, like this :
$(eval RELEASE_NUMBER=$$(shell $(BLD_ROOT)/path/to/rel_num.txt))
To view what is really inside eval at the time it's called you can use the same syntax with info instead of eval :
$(info RELEASE_NUMBER=$$(shell $(BLD_ROOT)/path/to/rel_num.txt))
Now I'm not sure about the part which is evaluated too soon so the $ symbols that I doubled may not be the good one(s), but using the info function will help you to find the correct command.
I have a rule something, that works on the variable VAR. I also have another rule something-all, that needs to run something, with VAR set to each value in vars.
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach VAR,$(vars),something)
This doesn't quite work, I get
noob#work:~/Desktop$ make something-all
something something
make: something: No such file or directory
make: *** [something-all] Error 1
It should probably print hello\nworld.
I used to do this with wildcard rules by retrieving VAR from %, but got the feeling that was the wrong way to do it. This looked like this:
vars = hello world
all: $(foreach VAR,$(vars),something-$(VAR))
something-%:
echo $*
The below should fix your problem
Using foreach (Tried on GNU Make 3.80 on sparc-solaris 2.8 and windows)
vars = hello world
something:
echo $(VAR)
something-all:
$(foreach i, $(vars), $(MAKE) something VAR=$i || exit 1;)
Using shell for-loop (Tried on GNU Make 3.80 and cc make on sparc-solaris 2.8)
vars = hello world
something:
echo $(VAR)
something-all:
for i in $(vars); do $(MAKE) something VAR=$$i || exit 1; done
TL;DR: If you want to program make, drop GNU Make in favor of BSD Make.
This is a personal recommendation. While BSD Make seems more limited than GNU Make, as it offers less programming facilities, it is much easier to program and has a few unique killer features. This is why I propose a solution with GNU Make and another solution for BSD Make:
Doing it in GNU Make
Using GNU Make, you can write a macro to define a target. The canonical way to define a sequence in a Makefile is to add the steps of the sequence as dependencies to a target, as reflected by the snippet below:
vars= hello world
define something_t =
something: something-$(1)
something-$(1):
#echo $(1)
endef
$(foreach _,$(vars),$(eval $(call something_t,$_)))
It is recommended to use this organisation (rather than defining just one target), because you can work on it to make the task easily resumable if you interrupt the sequence. A Makefile describes a job whose advancement is entirely described by the state of the file system. A task is then easily resumable, if each step is associated to a file, usually a compilation object but sometimes also an empty file which is touch'ed to indicate that important checkpoints have been passed.
Using an auxiliary macro is a flexible solution that can be adapted to more complicated tasks than just echoing a name. Note that this does work with newest versions of GNU Make (4.1). On GNU Make 3.81, you should remove the equal sign from the macro definition.
Adapting your example for BSD Make
If this is an option for you, I recommand dropping the use of GNU Make and replace it by BSD Make, which is way easier to program: it has a short and to the point documentation, while the documentation of GNU Make is very verbose and somewhat unclear, BSD Make has industrial-strength examples of complex rulesets (FreeBSD Build system or BSD Owl), and it has a simple and predictable macro language.
vars= hello world
something:
.for _var in ${vars}
echo ${_var}
.endfor
This can evolve to support more complicated tasks, just by replacing the echo by the adapted commands, or using intermediary steps.
Allow the user to override some tasks, also in BSD Make
In this slightly more advanced variation, we allow the user to override our own recipes for building targets something-hello and something-world.
For each item in our list, a target something-* is created it if it does not already exist, and added to the dependencies of something. The whole operation of defining these targets only happens if something has been left undefined. Therefore, users of these macros can:
Override the recipes for something-hello and something-world
Override the full procedure bound to something.
Implementing such customisation possibilities is mandatory if we want to write useful, reusable, macros for Make. Unluckily, customisation of this sort is nearly impossible in GNU Make.
vars = hello world
.if!target(depend)
.for _var in ${vars}
.if!target(something-${_var})
something-${_var}:
echo ${_var}
.endif
something: something-${_var}
.endfor
.endif
Here's one way to do it:
VARS := hello world
THINGS := $(addprefix something-, $(VARS))
allthings: $(THINGS)
something-%:
echo $*
It should be no surprise that
vars := hello world
something-all:
$(foreach VAR,$(vars),something)
tries to run something something. That's exactly what the foreach expands to, since you don't reference VAR in the third expression.
All you need to do is reference VAR and use a command such as echo:
vars := hello world
something-all:
$(foreach VAR,$(vars),echo $(VAR);)
$ make
echo hello; echo world;
hello
world
Note how chaining the commands with a semicolon avoids forking several shells or -- GASP! -- recursive make invocations. It doesn't get more performant than that.
Alternatively, if your command accepts several somethings as arguments,
vars := hello world
something-all:
echo $(foreach VAR,$(vars),$(VAR))
$ make
echo hello world
hello world
But that is equivalent to the super simple echo $(vars). So it might pay off to think outside the box trying to change your requirements to make this simple solution work.
The point is that I want to have some dependencies centralized in one variable, but the dependencies themselves are contained in variables.
a=meow
b=squeek
$(a):
touch $#
$(b):
touch $#
targs=$(a) $(b)
all: $(targs)
In the real case rules for a and b differ so I need them to be in separate targets.
When I try to build such a target, only last nested dependency gets executed:
$ make
touch meow
$ ls
. .. Makefile meow
Could anyone please explain me how do I fix the situation or where I'm wrong?
I can make a phony target like targs: $(a) $(b), but if there's a way to keep the structure I mentioned, I'd like to know about it.
Thanks in advance!
UPD: solved. My mistake: instead of running make all I ran make and make executed the first target instead of all.
Make's default is to use the first target in the Makefile. Either move the all target to the beginning or use the following line somewhere in your Makefile
.DEFAULT_GOAL := all
I am not sure if i understand include statement in makefile well enough. I have recently started to use makefile for making my analysis reproducible, and thought i was in the right track until the complexity grew, and seems like include statement is the answer as I am also trying to avoid recursive makefile problem while trying to keep my makefile sane.
To test what include did i ran a dumb test basically created the following.
test_folder-|-makefile
|-test_folder2-|
|-test2.mk
In the makefile i added include test_folder2/test2.mk, which according to my understanding would add the code from the test2.mk, so when i run make, it will also run target:dependencies listed in test2.mk, but it only ran the target:dependencies from makefile.
It would be great if someone can explain or point me to another blog/stack answer that might explain the right way to avoid recursive makefiles by using include statement.
PS: My analysis mostly includes python and R scripts and some command line tools.
makefile
test.txt:
echo "test.txt" > test.txt
include test_folder2/test2.mk
test2.mk
test_folder2/test1.txt:
echo "this is a test" > test1.txt
First, there is no directory called test1, so I will modify the makefiles accordingly.
Second, notice that the test_folder2/test1.txt rule builds test1.txt locally, despite its name. there is more than one way to so what I think you want; one way is to modify test_folder2/test2.mk:
test_folder2/test1.txt:
echo "this is a test" > test_folder2/test1.txt
In order to execute this rule by running make in test_folder/, you must modify makefile.
To execute only the test_folder2/test1.txt rule, do this:
test_folder2/test1.txt:
test.txt:
echo "test.txt" > test.txt
include test_folder2/test2.mk
To execute both rules, do this:
all: test.txt test_folder2/test1.txt
test.txt:
echo "test.txt" > test.txt
include test_folder2/test2.mk
(Note that "all" is the conventional name for a phony target that comes first in the makefile -- and is therefore the default -- and requires other rules to be executed. You can use another name if you like.)
To execute test_folder2/test1.txt dirst, then test.txt, do this:
test.txt: test_folder2/test1.txt
echo "test.txt" > test.txt
include test_folder2/test2.mk
(It is possible to execute them in the opposite order, but it gets ugly.)
Further refinements are possible, but this is a start.
Initially, when I made the question I attributed the problem to wildcards, a problem, but a bigger problem apparently looming. If I understand the error right, GNU make has no make-subprocessing so have to switch to some derivative of Boor's make. I may be understanding it wrong but I try to make this problem easier to understand.
Makefile: makefile trying to execute parts of itself many times, the thing I mean by subprocessing
all:
make clean
make $$(.??*)
#I want to replace below-like-things with a wildcard above
# make .lambda
# make .lambda_t
#
# KEY: It should run NEW make-processes! how?
clean:
-rm .??*
.lambda:
#do something
.lambda_t:
You are doing it the wrong way.
You don't need to call make like that. Just declares additional targets, and use target dependancies.
Note that you can use the Make 'foreach dir' function to get files matching a pattern:
_FILES_HIDDEN = $(foreach dir,./,$(wildcard .*))