I need to restart the make process in case some intermediate target gets (re)build.
This is the case when a PIP requirements file gets (re)compiled, because the checksum of the resulting file is used in the path to the virtualenv that is used in the Makefile.
If the requirements file gets updated, the same $(MAKECMDGOALS) should get rebuild from the beginning.
I have come up with the following, but this requires to make the outer/inital make process fail (exit 1), which I would like to avoid.
foo:
echo "foo"
# Need to restart make if this has been used as intermediate target.
if [ $(MAKECMDGOALS) != "$#" ]; then \
echo "Restarting make..."; \
touch $#; \
$(MAKE) $(MAKECMDGOALS); \
exit 1; \
fi
bar: foo
echo "bar"
.PHONY: bar
A makefile on the lines below might fit the bill. bar and baz are goals
that will be sabotaged if they intermediately make foo (a.k.a tricky goals).
There's no restarting involved: instead, it is arranged that the tricky
goals are only made by a fresh $(MAKE), after the top make has ensured foo is
up-to-date.
.phony: all clean
tricky = bar baz
all:
$(MAKE) $(tricky)
foo:
echo "foo"
touch $#
clean:
rm -f foo
ifndef simple
tricky_goals = $(filter $(tricky),$(MAKECMDGOALS))
ifneq ($(tricky_goals),)
$(tricky_goals): foo
$(MAKE) simple=y $#
endif
else # ifndef simple
# All tricky goals follow...
bar: foo
echo "bar"
baz: foo
echo "baz"
endif # ifndef simple
The behaviour depends on whether the variable simple is defined, which
by default it is not.
If simple is defined then the goals, whatever they are, will be made
the obvious way - the way they'd be made if it weren't for the foo pitfall.
If simple is undefined then tricky_goals becomes the tricky members $(MAKECMDGOALS).
If $(tricky_goals) is empty there is nothing tricky to do. Otherwise the target list
$(tricky_goals) becomes dependent on foo. So if foo is out-of-date, it gets made now.
Finally each tricky goal is made by calling a simple $(MAKE) for it,
in which foo will be up-todate. Any non-tricky goals will be made the simple way.
I've included all and clean goals for completeness. Note that all
needs coded to go the tricky way.
Related
The working example below always runs touch fileA with each run of make. How can I stop this?
all: analysis.txt
analysis.txt: countries
touch $#
countries: usa.txt mexico.txt
$(countries): %.txt
%.txt:
echo "bash ./cityGenerator $*" > $#
fileA depends on workflowB. So as long as workflowB is out of date, fileA will be out of date and the touch will be run.
workflowB, as shown here, is marked .PHONY so it will ALWAYS be considered out of date by make, so fileA will ALWAYS be rebuilt.
Even if it weren't .PHONY, there is no rule to create a file named workflowB so it would still always be out of date.
As for how you can stop it, that depends entirely on what these things do which you haven't told us, and why you structured workflowB that way which you also haven't told us.
One way to solve it would be to remove the .PHONY and give workflowB a recipe that touches a file, then it would be up to date.:
workflowB: workflowB_files
touch $#
But of course, that might defeat the purpose of this (which, again, you haven't told us).
Your analysis.txt depends on countries which is considered as a file, but such file is never created, hence the rule is always run. You can find out by the debug output:
$ make -dr
...
Finished prerequisites of target file 'analysis.txt'.
Prerequisite 'countries' of target 'analysis.txt' does not exist.
Must remake target 'analysis.txt'.
touch analysis.txt
...
It seems like you are treating countries as a variable, so maybe this could be reworked:
$ cat Makefile
all: analysis.txt
countries := usa.txt mexico.txt
analysis.txt: $(countries)
touch $#
%.txt:
echo "bash ./cityGenerator $*" > $#
Now there is no countries file considered and make behaves as expected:
$ make
echo "bash ./cityGenerator usa" > usa.txt
echo "bash ./cityGenerator mexico" > mexico.txt
touch analysis.txt
$ make
make: Nothing to be done for 'all'.
I have a directory with test inputs and outputs. I wanted make to automatically test my program against this directory after build, for convenience. Thus I needed to somehow force the test target of Makefile to depend on the entire testing directory (it's called good, because it contains valid inputs and outputs for the program)
I read this question and the accepted answer and the comments about deleted files under this answer: Makefile rule that depends on all files under a directory (including within subdirectories) And, incorporating advice from this answer & comments, I came out with this:
my#comp:~/wtfdir$ cat Makefile
test : test.sh $(shell find good)
./test.sh
my#comp:~/wtfdir$
For the sake of MCVE, test.sh is very rudimentary:
my#comp:~/wtfdir$ cat test.sh
echo "blah"
my#comp:~/wtfdir$
However, I noticed, this behaves in a rather unexpected way:
my#comp:~/wtfdir$ ls good
test1 test1.out
my#comp:~/wtfdir$ make
./test.sh
blah
my#comp:~/wtfdir$ touch good/test1
my#comp:~/wtfdir$ make
cp good/test1 good/test1.out
./test.sh
blah
my#comp:~/wtfdir$
Why (expletive redacted) does modifying test1 cause make to overwrite test1.out with test1??? I'm not a big fan of data losses, you know.
What's going on here?
Your Make appears to be GNU Make. Here's why this happens. Your recipe:
test : test.sh $(shell find good)
./test.sh
adds to the prerequisites of test every file and directory that is listed
by find good in the current directory, which happen to be:
good
good/test1
good/test1.out
So to make target test, Make begins by determining if any of the specified
or built-in recipes require it to rebuild any of the prerequsities:
test.sh good good/test1 good/test1.out
Among its built-in recipes it finds:
%.out: %
# recipe to execute (built-in):
#rm -f $#
cp $< $#
as you can verify by running:
$ make --print-data-base | grep -A4 '%.out'
The rule for this recipe is matched by:
good/test1.out: good/test1
and by doing:
$ touch good/test1
you have made good/test1.out out of date with respect to good/test1.
So make executes the recipe:
#rm -f good/test1.out
cp good/test1 good/test1.out
the visible output of which is what you observed:
cp good/test1 good/test1.out
Then it proceeds with the recipe for test:
./test.sh
blah
There is always a risk of such booby-traps if you write a makefile that blindly
generates at runtime some set of preqrequisites or targets you don't know beforehand.
You could avoid this one in particular by explicitly deleting the offending
implicit pattern rule in your makefile by writing:
%.out: %
with no recipe. And you can avoid all possible booby-traps of this sort by disabling all
built-in recipes, with:
$ make --no-builtin-rules ...
but that will require you to write for yourself any builtin-recipes that your
makefile relies on.
The best solution for you is probably to amend your makefile as follows:
PREREQS := $(shell find good)
test : test.sh $(PREREQS)
./test.sh
$(PREREQS): ;
Then the last line explicitly specifies an empty recipe
for each of the $(PREREQS), and Make will not consult any pattern rules for targets
that have explicit recipes.
You should additionally make test a phony target:
.PHONY: test
for the avoidance of the booby-trap where something creates a file called test in the build directory.
I'm writing my first complex Makefile for a highly-modularized project.
I have various sub-directories, each one has its own Makefile which supports at least the all and the clean targets.
These sub-Makefiles work just fine, however I have a problem with the main Makefile, that should call all the sub-Makefiles automatically from the list contained in the variable COMPONENTS.
I tried with the following Makefile:
OUTFILE = diskimage.bin
export NASM = nasm
COMPONENTS = bootloader
.PHONY = all clean FORCE $(OUTFILE) $(COMPONENTS)
all: $(OUTFILE)
$(OUTFILE): $(COMPONENTS)
./make_image
$(COMPONENTS): FORCE
for component in $(COMPONENTS); do \
make -C $component; \
done
FORCE:
clean:
for component in $(COMPONENTS); do \
make -C $component clean; \
done
This results in the following error message:
for component in bootloader; do \
make -C omponent; \
done
make: *** omponent: No such file or directory. Stop.
make: *** [bootloader] Error 2
as if the $component expression was only parsed as $c. I don't understand why that happens and how to fix it.
Just double the dollar sign:
$(COMPONENTS): FORCE
for component in $(COMPONENTS); do \
make -C $$component; \
done
The trouble is that with your makefile, Make expands $component before executing the rule. And since $c has no value (there is no such variable), it expands to nothing, leaving "omponent", which it passes to she shell, which complains that there's no such directory. (If you had written $(component), Make would have expanded it to nothing, since Make knows of no such variable, and then the shell would have complained that you were not specifying a directory at all.)
With the double dollar sign, Make expands $$component to $component, which it then passes to the shell, which interprets it as the loop variable, and everything proceeds as planned.
You really should have played around with a simple loop in a command, before attempting to do actual work with one.
Several issues.
.PHONY should be written as a dependency, not a macro definition
Don't write shell loops, use make syntax instead
When you call make recursively, you must do it via the ${MAKE} macro invocation
Leading to
OUTFILE = diskimage.bin
export NASM = nasm
COMPONENTS = bootloader
.PHONY: all
all: ${OUTFILE}
.PHONY: ${OUTFILE}
${OUTFILE}: ${COMPONENTS}
./make_image
.PHONY: ${COMPONENTS}
${COMPONENTS}:
${MAKE} -C $#
The advantage of this formulation is that it is parallel make friendly.
Always a test of a good Makefile.
Here make -j5 all will cause make to keep 5 commands running at once,
across all invocations of make.
Nice if you have 4 CPUs.
What about clean?
(Personally I hate clean targets—it's a sign of dodgy dependencies,
and of unhygienic mixing of source and target folders.)
Just add -clean (say) to each of the component names,
and repeat the pattern above.
CLEANS := $(addsuxffix -clean,${COMPONENTS})
.PHONY: clean
clean: ${CLEANS} ; #echo Clean succesful
.PHONY: ${CLEANS}
${CLEANS}: %-clean:
${MAKE} -C $* clean
These two sections can tidied up and combined into one if you feel so inclined.
Tip
Always run make with --warn (or --warn-undefined-variables to give it its full name) to catch inadvertent expansion of $c in things like $component.
So here's a Makefile to install foo.conf, based on a master copy called foo.conf.master. It installs it to the current directory rather than /etc, just for testing purposes:
all: foo.conf.copied
foo.conf.copied: foo.conf.master foo.conf
cp foo.conf.master foo.conf
touch $#
# Recipe to tell make that it is okay for foo.conf not to exist beforehand.
foo.conf:
So then create foo.conf.master:
$ touch foo.conf.master
$
and you're ready to test:
$ make
cp foo.conf.master foo.conf
touch foo.conf.copied
$
The point is that if I (with my "trusted" sysadmin hat on) modify foo.conf.master then make (possibly called by cron) will roll out the update:
$ touch foo.conf.master
$ make
cp foo.conf.master foo.conf
touch foo.conf.copied
$
But equally important: if I (with my "rogue" sysadmin hat on) modify the installed version then make will back out the update:
$ touch foo.conf
$ make
cp foo.conf.master foo.conf
touch foo.conf.copied
$
Woohoo.
Okay, so now the problem: obviously foo.conf isn't the only file I want do this for, so I need to change my static rules to pattern rules. Okay, that's easy: substitute foo.conf for % in targets and dependencies, substitute foo.conf for $* in the commands, and make a minor modification to the last recipe (which would otherwise become only '%:') so that it doesn't look like I'm trying to cancel a builtin pattern rule.
So clean up and create this Makefile:
all: foo.conf.copied
%.copied: %.master %
cp $*.master $*
touch $#
# Recipe to tell make that it is okay for foo.conf not to exist beforehand.
# Nop tells make that I'm not *cancelling* a pattern rule here
# (see http://stackoverflow.com/questions/34315150/make-implicit-rules-dont-work-without-command).
%: ;
But this doesn't work:
$ make
make: *** No rule to make target `foo.conf.copied', needed by `all'. Stop.
$
The error message is misleading; it is really foo.conf that it doesn't know how to make, which can be demonstrated by adding the following at the bottom of the Makefile:
foo.conf:
touch $#
But then that's a static rule again, which I don't want.
There are a couple more requirements I would also like to satisfy, which the above example doesn't demonstrate. These are:
foo.conf should be installable anywhere in the filesystem (e.g. /etc/foo/server/foo.conf)
foo.conf.master should be in a central directory, or subdirectly thereof, for all master versions, preferably without the '.master' extension (e.g. ~/poor-mans-puppet/master-files/etc/foo/foo.conf)
foo.conf.copied should be in a central directory, not in the same directory as foo.conf (e.g. ~/poor-mans-puppet/timestamp-files/etc/foo/foo.conf)
After much googling, hair pulling, I'm asking here! Any ideas please? (PS: if copying Makefiles from here, remember to change indentation back to tabs.)
Mad Scientist below suggested an elegant static rule, but I really need it to be a pattern rule. The reason is that I need to hook extra dependencies in using rules:
all: <new-dependency>
rather than hooking them in using variables:
STUFF_ALL_SHOULD_DEPEND_ON += <new-dependency>
The reason for this requirement is for consistency with how other (non-%.copied) targets are handled in my very large Makefile.
However, based on Mad Scientist's idea, I tried the following, which didn't work, but perhaps helps somebody to help me:
all: foo.conf.copied
%.copied: %.master %
$(eval FILES_FOR_WHICH_AN_EMPTY_RECIPE_ARE_NEEDED += $$*)
cp $*.master $*
touch $#
define GENERATE_STATIC_EMPTY_RULE
$(1):
endef
$(foreach X,$(FILES_FOR_WHICH_AN_EMPTY_RECIPE_ARE_NEEDED),$(eval $(call GENERATE_STATIC_EMPTY_RULE,$(X))))
Can you explain why you're using this extra ".copied" file? Why don't you just use:
%: %.master ; cp $< $#
?
Anyway, you're running afoul of make's special rules related to match-anything rules (pattern rules like % that can build everything). If you change your pattern so it's not match-anything, like %.conf: ; then it will work. However you probably don't want to assume that all files end in .conf.
Alternatively you can use static pattern rules, like this:
FILES_TO_COPY = foo.conf bar.conf biz.baz
all: $(FILES_TO_COPY:%=%.copied)
$(FILES_TO_COPY:%=%.copied): %.copied : %.master %
cp $*.master $*
touch $#
and you don't need the extra pattern rule.
In the end, I dynamically generated static rules. The following pseudo-code hopefully makes the actual Makefile easier to understand:
if flag not set # flag won't be set on first call
prepare static rules
set flag # prevent running this clause again
recurse! # make invokes make
else
include static rules
do the normal thing
endif
Here's the real Makefile:
ifeq ($(MAKELEVEL),0)
all:
for X in $(patsubst %.copied,%,$^); do \
echo "$$X.copied: $$X.master $$X"; \
echo " cp $$X.master $$X"; \
echo " touch \$$#"; \
echo "$$X: ;"; \
done > Makefile.include
$(MAKE)
# The real dependencies on all are defined below, but while creating
# Makefile.include, we don't want make to digress and start making
# the dependencies; this pattern rule will stop it from doing that.
%.copied: ;
else
include Makefile.include
endif
all: foo.conf.copied
The outer make can be silenced by use of .SILENT and the --no-print-directory option (not shown above for clarity).
Here's the output:
$ touch foo.conf.master
$ make
cp foo.conf.master foo.conf
touch foo.conf.copied
$ touch foo.conf
$ make
cp foo.conf.master foo.conf
touch foo.conf.copied
$
Assume a target foo.tar, that depends on a list of files foo.files, e.g.
FOO_FILES := $(shell cat foo.files)
foo.tar: foo.files $(FOO_FILES)
tar -cf foo $(FOO_FILES)
Now suppose that foo.files need to be generated e.g.:
foo.files: foo.files.template
sed -e "s/#{VERSION}/$(VERSION)/" < $< > $#
It is clear that foo.files depends on foo.files.template, but how can one make sure that FOO_FILES is evaluated after foo.files is generated?
Your original rules are correct. Because updating foo.files causes the value of FOO_FILES to become outdated you just need to make sure your Makefile is re-evaluated by gnu make when foo.files has been updated by making your Makefile depend on foo.files:
Makefile : foo.files
So, I found an answer reading about Advanced Auto-Dependency Generation over at mad-scientist.net. Basically, it is possible to re-evaluate a makefile by way of a GNU/Make feature. When there is a rule to generate an included makefile, the entire makefile will be re-read after the generation of the included file. Thus --
# -*- mode: make -*-
VERSION := 1.2.3
foo.tar: foo.files $(FOO_FILES)
tar cf $# $(FOO_FILES)
clean:
rm -f foo.files foo_files.mk foo.tar
foo.files: foo.files.template
sed -e "s/#{VERSION}/$(VERSION)/" < $< > $#
# -- voodoo start here --
# This will ensure that FOO_FILES will be evaluated only
# *after* foo.files is generated.
foo_files.mk: foo.files
echo "FOO_FILES := `xargs < $<`" > $#
include foo_files.mk
# -- voodoo ends here --
.PHONY: clean
-- seems to do the right thing.
... and just for completeness:
foo.files.template is:
a-$(VERSION)
b-$(VERSION)
and assume the presence of a-1.2.3 and b-1.2.3.
It can't be done in one pass; Make determines which targets must be rebuilt before it actually executes any rule, and in this case the full list of targets doesn't exist until one of the rules is executed.
This should do it:
FOO_FILES := $(shell cat foo.files)
foo.tar: foo.files
$(MAKE) foo-tarball
.PHONY: foo-tarball
foo-tarball: $(FOO_FILES)
tar -cf foo $^
EDIT:
As the OP points out, this will not work as written; I left out a prerequisite:
foo.tar: foo.files $(FOO_FILES)
...
Note that this will recurse even if foo.files has not changed, which is not strictly necessary; it is possible to correct this, but not elegantly. (For comparison, the selected solution, which I admit is cleaner than mine, recurses even if the target has nothing to do with foo.tar.)