From the docs:
$?
The names of all the prerequisites that are newer than the target,
with spaces between them.
So, given a makefile:
# Force make to search for 'foo' in the VPATH directory
$(shell rm -rf foo)
# If 'D' is a "regular" file, we remove it first.
$(shell rm -rf D)
$(shell mkdir D)
# Suggest a VPATH-file, for Make to "associate" with 'foo'.
$(shell touch D/foo)
$(shell sleep 1)
# Target 'all' is newer than prerequisite 'D/foo'
$(shell touch all)
VPATH = D
all : foo phony
echo '$?'
foo ::
touch '$#'
.PHONY: phony
.PRECIOUS : D/foo
And running, I get:
$ make -r
touch 'D/foo'
echo 'D/foo phony'
D/foo phony
# Try again, but this time, with parallel-execution mode.
$ make -r -j
touch 'D/foo'
echo 'phony'
phony
Here, we have 2 serious issues:
Given the simple and explicit recipe to "touch" the prerequisite foo, which Make clearly executes - hence will guarantee that foo will be "newer" than all - Make still does not expand $? to D/foo, at-least in the 2nd case above (i.e. for the parallel-execution (-j) mode). Why?
If you come up with an explanation for the above, shouldn't it also explain, why in the 1st case (non-parallel execution), $? - does indeed - get expanded to D/foo.
I guess, I had an assumption, that parallel vs. non-parallel aside, Make will always pause before executing a target, and first check if all of its prerequisites had already finished their respective builds.
So, shouldn't the $? variable be identically expanded for both cases?
I think there are two issues going on here.
The first is that double-colon rules appear to act like phony targets in that they force make to consider the target as "newer" regardless of actual modification time. (This is why the non-parallel version behaves the way it does. Change from foo :: to foo : and you don't get foo in the $? output at all.)
The second thing is that, despite that, using parallel mode seems to force make back into considering modification times of its prerequisites (so the previous behavior is avoided).
This is conjecture and not definitive since I haven't dug through the code to see if this is actually happening but it explains the results here (it also explains the results on the other, nearly identical, question here).
Related
I have a number of makefiles that build and run tests. I would like to create a script that makes each one and notes whether the tests passed or failed. Though I can determine test status within each make file, I am having trouble finding a way to communicate that status to the caller of the make command.
My first thought is to somehow affect the return value of the make command, though this does not seem possible. Can I do this? Is there some other form of communication I can use to express the test status to the bash script that will be calling make? Perhaps by using environment variables?
Thanks
Edit: It seems that I cannot set the return code for make, so for the time being I will have to make the tests, run them in the calling script instead of the makefile, note the results, and then manually run a make clean. I appreciate everyone's assistance.
Make will only return one of the following according to the source
#define MAKE_SUCCESS 0
#define MAKE_TROUBLE 1
#define MAKE_FAILURE 2
MAKE_SUCCESS and MAKE_FAILURE should be self-explanatory; MAKE_TROUBLE is only returned when running make with the -q option.
That's pretty much all you get from make, there doesn't seem to be any way to set the return code.
The default behavior of make is to return failure and abandon any remaining targets if something failed.
for directory in */; do
if ( cd "$directory" && make ); then
echo "$0: Make in $directory succeeded" >&2
else
echo "$0: Make in $directory failed" >&2
fi
done
Simply ensure each test leaves its result in a file unique to that test. Least friction will be to create test.pass if thes test passes, otherwise create test.fail. At the end of the test run gather up all the files and generate a report.
This scheme has two advantages that I can see:
You can run the tests in parallel (You do us the -jn flag, don't you? (hint: it's the whole point of make))
You can use the result files to record whether the test needs to be re-run (standard culling of work (hint: this is nearly the whole point of make))
Assuming the tests are called test-blah where blah is any string, and that you have a list of tests in ${tests} (after all, you have just built them, so it's not an unreasonable assumption).
A sketch:
fail = ${#:%.pass=%.fail}
test-passes := $(addsuffix .pass,${tests})
${test-passes}: test-%.pass: test-%
rm -f ${fail}
touch $#
$* || mv $# ${fail}
.PHONY: all
all: ${test-passes}
all:
# Count the .pass files, and the .fail files
echo '$(words $(wildcard *.pass)) passes'
echo '$(words $(wildcard *.fail)) failures'
In more detail:
test-passes := $(addsuffix .pass,${tests})
If ${tests} contains test-1 test-2 (say), then ${test-passes} will be test-1.pass test-2.pass
${test-passes}: test-%.pass: test-%
You've just gotta love static pattern rules.
This says that the file test-1.pass depends on the file test-1. Similarly for test-2.pass.
If test-1.pass does not exist, or is older than the executable test-1, then make will run the recipe.
rm -f ${fail}
${fail} expands to the target with pass replaced by fail, or test-1.fail in this case. The -f ensures the rm returns no error in the case that the file does not exist.
touch $# — create the .pass file
$< || mv $# ${fail}
Here we run the executable
If it returns success, our work is finished
If it fails, the output file is deleted, and test-1.fail is put in its place
Either way, make sees no error
.PHONY: all — The all target is symbolic and is not a file
all: ${test-passes}
Before we run the recipe for all, we build and run all the tests
echo '$(words $(wildcard *.pass)) passes'
Before passing the text to the shell, make expands $(wildcard) into a list of pass files, and then counts the files with $(words). The shell gets the command echo 4 passes (say)
You run this with
$ make -j9 all
Make will keep 9 jobs running at once — lovely if you have 8 CPUs.
Is there a way to force a target-rule to run as part of setting a something in a variable?
For example let's say we have a target and rule:
all_mp3s:
find / -name "*.mp3" > all_mp3s
And then a variable:
MP3S := $(file < all_mp3s)
Is there a way to make sure all_mp3s file is getting created before evaluating the MP3S variable?
There is no simple straightforward way to force a rule to be evaluated before a variable gets assigned. There are more complex ways. The following is for GNU make.
Let's first assume that you want to run the (slow) find command only if the file all_mp3s does not exist, else use its content. You can use GNU make conditionals:
ifeq ($(wildcard all_mp3s),all_mp3s)
MP3S := $(shell cat all_mp3s)
else
MP3S := $(shell $(MAKE) all_mp3s ; cat all_mp3s)
endif
all_mp3s:
find / -name "*.mp3" > $#
But I if your Makefile is more complex than this, uses MP3S several times, and what you really want is:
avoid running your super-slow find several times,
run it only if needed (and only once),
get the result in a file (all_mp3s) plus a make variable (MP3S),
MadScientist has a wonderful GNU make trick that can be used here:
MP3S = $(eval MP3S := $$(shell find / -name "*.mp3"))$(MP3S)
all_mp3s:
printf '%s\n' '$(MP3S)' > all_mp3s
.PHONY: help clean
help:
printf 'MP3 finder\n'
clean:
rm -f all_mp3s
If the MP3S recursively expanded make variable is expanded because some part of your Makefile is evaluated and needs its value (e.g. if you run make all_mp3s while all_mp3s does not exist), the find command will be run, its result stored in the variable... and the variable will be turned into a simply expanded make variable, which further expansions, if any, will reuse the same, already computed, value.
Else, if your invocation of make (e.g. make cleanor make help) does not need MP3S value, the find command will not even be run.
The all_mp3s file is generated from the value of MP3S (instead of the opposite in the other solution).
However, there is another important thing to decide: do you want to declare all_mp3s as a phony target:
.PHONY: all_mp3s
or not?
If you declare it as phony, the find command will be run once and only once each time you invoke make all_mp3s (or another goal that depends on all_mp3s). But targets depending on all_mp3s will always be rebuilt too, which is not necessarily what you want.
If you don't declare it as phony, and the file exists already, the find command will not be run at all (unless the value of MP3S is needed elsewhere in your Makefile), and the content of all_mp3s will not be updated, which is not necessarily what you want.
As you do not give enough information in your question to decide, it is up to you.
I have a few software projects which are distributed as RPMs. They are versioned using semantic versioning to which we affix a release number. Using the regular conventions, this is MAJOR.MINOR.PATCH-REL_NUM. Though beyond the scope of this article, the release numbers are stored in git. The release target in the makefile looks something like this:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
# Although the third step, this was re-ordered to step 1
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
While debugging, I eventually discovered that, although the call to eval was the third step in the recipe, it was actually being evaluated first! This is why the RPM always had a release number one less than the number I was watching get pushed to the remote.
I have done much googling on this and I haven't found any hits that explain the order of evaluation with regard to eval when used in recipes. Perhaps it isn't even with respect to eval but functions in general. Furthermore, I haven't found verbiage on this in the GNU manuals for make either (if it's there, kindly point out what chapter). I've worked around the problem so it's not a bother, I'm just wondering, is this expected and if so, why?
The missing bit, that no one above is getting, is simple: when make is going to run a recipe it expands all lines of the recipe first, before it starts the first line. So:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
# Although the third step, this was re-ordered to step 1
$(eval RELEASE_NUMBER=$(shell $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
when make decides to run the release target it first expands all the lines in the recipe, which means the eval is expanded, then it runs the resulting lines. That's why you're getting the behavior you're seeing.
I don't really see why you need to use eval here at all; why not just use:
release:
$(MAKE) clean
$(BLD_ROOT)/tools/incr_rel_num
$(MAKE) rpm RPM_RELEASE_NUM="$$(cat $(BLD_ROOT)/path/to/rel_num.txt))"
(BTW, you should never use bare make inside your makefiles; you should always use $(MAKE) (or ${MAKE}, same thing).
The $(eval ...) function
generates a fragment of make-sytax which becomes part of the parsed makefile.
The makefile is parsed entirely before any recipes are executed and when recipes
are executed all make-statements, make-expressions and make-variables have been
evaluated away.
So it does not make sense to consider an $(eval ...) call as being one
of the lines of a recipe. It might generate values that are used in the make-expansion
of the recipe, but if so then this happens when the makefile is parsed, before the recipe is run.
Thus in your example, the line:
$(eval RELEASE_NUMBER=$(shell $(BLD_ROOT)/path/to/rel_num.txt))
which I assume should really be:
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
is evaluated when the makefile is parsed, and let's say it results in the
make-variable RELEASE_NUMBER acquiring the value 1.0, because, when the
makefile is parsed, the file $(BLD_ROOT)/path/to/rel_num.txt) contains
1.0. In that case your recipe:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
$(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt))
make rpm RPM_RELEASE_NUM=$(RELEASE_NUMBER)
will resolve to the like of:
release:
make clean
some_build_dir/tools/incr_rel_num
make rpm RPM_RELEASE_NUM=1.0
You will observe when make runs the recipe that it prints no line that
is "the expansion of" $(eval RELEASE_NUMBER=$(shell cat $(BLD_ROOT)/path/to/rel_num.txt)),
because there is no such thing in the recipe. It doesn't matter that:
some_build_dir/tools/incr_rel_num
is presumably a command that writes, say, 1.1 or 2.0 in the file some_build_dir/path/to/rel_num.txt.
That action simply has no effect on the recipe. Nothing that executed in the recipe
can change the recipe.
$(eval ...) has no business in your recipe. What you want to achieve is simply:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
RELEASE_NUMBER=$$(cat $(BLD_ROOT)/path/to/rel_num.txt) && \
make rpm RPM_RELEASE_NUM=$$RELEASE_NUMBER
where $$ is what you do in a makefile to escape $ and, in this case,
leave it for the shell when the recipe is executed.
This recipe expands to 3 shell commands executed in sequence:
$ make clean
$ some_build_dir/tools/incr_rel_num
$ RELEASE_NUMBER=$(cat some_build_dir/path/to/rel_num.txt) && \
make rpm RPM_RELEASE_NUM=$RELEASE_NUMBER
and might as well be simplified further to:
release:
make clean
$(BLD_ROOT)/tools/incr_rel_num
make rpm RPM_RELEASE_NUM=$$(cat $(BLD_ROOT)/path/to/rel_num.txt)
You are correct, there are multiple levels of evaluation. The content on what is inside eval is evaluated a first time before that the function is actually called. If you want the content of eval to be evaluated at the time eval is called, you have to escape the $ sign by putting it twice, like this :
$(eval RELEASE_NUMBER=$$(shell $(BLD_ROOT)/path/to/rel_num.txt))
To view what is really inside eval at the time it's called you can use the same syntax with info instead of eval :
$(info RELEASE_NUMBER=$$(shell $(BLD_ROOT)/path/to/rel_num.txt))
Now I'm not sure about the part which is evaluated too soon so the $ symbols that I doubled may not be the good one(s), but using the info function will help you to find the correct command.
It looks to me like Makefile rules can be roughly classified into "positive" and "negative" ones: "positive" rules create missing or update outdated files, while "negative" ones remove files.
Writing prerequisites for "positive" rules is quite easy: if the target and the prerequisite are file names, make by default runs the recipe if the target is missing or outdated (a missing file in this context may be viewed as an infinitely old file).
However, consider a "negative" rule, for example for target clean. A usual way to write it seems to be something like this:
clean:
rm -f *.log *.synctex.gz *.aux *.out *.toc
This is clearly not the best way to do:
rm is executed even when there is nothing to do,
its error messages and exit status need to be suppressed with -f flag, which has other (possibly undesirable) effects, and
the fact that there were nothing to do for target clean is not reported to the user, unlike what is normal for "positive" targets.
My question is: how to write a Makefile rule that shall be processed by make only if certain files are present? (Like what would be useful for make clean.)
how to write a Makefile rule that shall be processed by make only if certain files are present? (Like what would be useful for make clean.)
You can do it like so:
filenames := a b c
files := $(strip $(foreach f,$(filenames),$(wildcard $(f))))
all: $(filenames)
$(filenames):
touch $#
clean:
ifneq ($(files),)
rm -f $(files)
endif
Example session:
$ make
touch a
touch b
touch c
$ make clean
rm -f a b c
$ make clean
make: Nothing to be done for 'clean'.
Useful perhaps for some purposes, but it strikes me as a strained refinement for make clean.
This can be easily remedied:
clean:
for file in *.log *.synctex.gz *.aux *.out *.toc; do \
if [ -e "$file" ]; then \
rm "$$file" || exit 1; \
else \
printf 'No such file: %s\n' "$file" \
fi \
done
The if statement is necessary unless your shell supports and has enabled nullglob or something similar.
If your printf supports %q you should use that instead of %s to avoid possible corruptions of your terminal when printing weird filenames.
A meta-answer is: are you sure you want to do this?
The other answers suggest to me that the cure is worse than the disease, since one involves an extension to POSIX make (ifneq), and the other uses a compound command which spreads over seven lines. Both of these are sometimes necessary expedients – I'm not criticising either answer – but both are things I avoid in a Makefile if I can. If I found myself wanting to do this in a clean rule, perhaps for the reason you mention in your comment to #MikeKinghans' answer, I'd try quite hard to change the rest of the Makefile to avoid needing this.
Reflecting on your three original points in turn:
rm is executed even when there is nothing to do: so what? The alternatives still need to, for example, expand the *.log *.synctex.gz ... so there's only miniscule efficiency gain to avoiding the rm. Make is a high-level tool which generally does not concern itself with efficiency.
its error messages and exit status need to be suppressed with -f flag: the -f flag doesn't generally suppress errors and the exit status, it merely indicates to rm that a non-existing or non-permissioned file is not to be regarded as an error.
the fact that there were nothing to do for target clean is not reported to the user: should the user really care?
The last point is the most interesting. People asking about make, on Stackoverflow and elsewhere, sometimes make things hard for themselves by trying to use it as a procedural language – make is not Python, or Fortran. Instead, it's a goal programming language (if we want to get fancy about it): you write snippets of rules to achieve sub-goals, so that the user (you, later) doesn't have to care about the details or the directory's current state, but can simply indicate a goal, and the program does whatever's necessary to get there. So whether there's is or isn't anything to do, the user ‘shouldn't’ care.
I think the short version of this answer is: it's idiomatic to keep make rules as simple (and thus as readable and robust) as possible, even at the expense of a little crudity or repetition.
GPATH has a special feature.
It allows Make to recursively search for a missing target, by repeatedly applying the same VPATH on a given target.
Thus: all can turn to D/D/D/D/D/all (VPATH applied 5 times!), and so on.
This rather bizarre - and frankly buggy - relocation, that Make performs on its targets, doesn't occur for a "normal" VPATH. Thus, for a "regular" VPATH, Make will still apply the VAPTH once, and only once.
But, as soon as you add GPATH into the mix, Make goes out of its way, by doing a fully-exhaustive recursive-vpath-search, relocating the missing target to a directory - as deep as make could possible match - by repeatedly and recursively applying again and again the same VPATH.
For example, given a makefile, like:
# Force Make, to do a "directory-search" (via VPATH), for the non-existent file 'all'
$(shell rm -rf 'all')
# Make sure, that all of the (vpath) files - 'D/all', 'D/D/all' and so on - DO NOT exist!
$(shell rm -rf 'D')
VPATH = D
all :
#echo '$#'
D/all :
D/D/all :
D/D/D/all :
D/D/D/D/all :
D/D/D/D/D/all :
Executing, I get:
# 'VAPTH' alone. (i.e. no 'GPATH').
$ make
D/all
# Adding 'GPATH = D', causes Make to do an "exhaustive" recursive-directory-search.
# This recursive serach, has no limits, regarding the number of times, Make will apply the same VPATH on a given target.
$ make -f makefile -f <(echo 'GPATH = D')
D/D/D/D/D/all
Bug or feature? What do you think?
(Versions note: 3.82 and up).