I am having a problem with this Makefile structure.
make commit.install should create foo.txt and then put it into some folder. However, make doesn't call the target that creates foo.text. This means I have to call make foo.txt to create the file itself and then call make commit.install to copy this over. I think there is a problem with make not expanding variable dependencies.
./common.mk
src.install: ${INSTALLSRC}
mkdir -p result
for f in ${INSTALLSRC}; do \
cp -a $$f result/.; \
done
commit.install: src.install
echo "finished"
clean:
rm -rf result
./inner/nested/GNUmakefile
include ../common.mk
include Makefile
./inner/nested/Makefile
SRC=foo.txt
foo.txt:
echo "hello world" > foo.txt
./inner/common.mk
include ../../common.mk
INSTALLSRC=${SRC}
Repository
The problem is that you define INSTALLSRC after the rule that has it as a prerequisite. So when Make evaluates the prerequisite list, the variable is empty, so Make sees no need to build foo.txt. This would have been pretty obvious if you had simplified this collection of makefiles.
The fix is simple; swap the two lines in inner/common.mk:
INSTALLSRC=${SRC}
include ../../common.mk
and GNUMakefile:
include Makefile
include ../common.mk
Related
I have a directory with test inputs and outputs. I wanted make to automatically test my program against this directory after build, for convenience. Thus I needed to somehow force the test target of Makefile to depend on the entire testing directory (it's called good, because it contains valid inputs and outputs for the program)
I read this question and the accepted answer and the comments about deleted files under this answer: Makefile rule that depends on all files under a directory (including within subdirectories) And, incorporating advice from this answer & comments, I came out with this:
my#comp:~/wtfdir$ cat Makefile
test : test.sh $(shell find good)
./test.sh
my#comp:~/wtfdir$
For the sake of MCVE, test.sh is very rudimentary:
my#comp:~/wtfdir$ cat test.sh
echo "blah"
my#comp:~/wtfdir$
However, I noticed, this behaves in a rather unexpected way:
my#comp:~/wtfdir$ ls good
test1 test1.out
my#comp:~/wtfdir$ make
./test.sh
blah
my#comp:~/wtfdir$ touch good/test1
my#comp:~/wtfdir$ make
cp good/test1 good/test1.out
./test.sh
blah
my#comp:~/wtfdir$
Why (expletive redacted) does modifying test1 cause make to overwrite test1.out with test1??? I'm not a big fan of data losses, you know.
What's going on here?
Your Make appears to be GNU Make. Here's why this happens. Your recipe:
test : test.sh $(shell find good)
./test.sh
adds to the prerequisites of test every file and directory that is listed
by find good in the current directory, which happen to be:
good
good/test1
good/test1.out
So to make target test, Make begins by determining if any of the specified
or built-in recipes require it to rebuild any of the prerequsities:
test.sh good good/test1 good/test1.out
Among its built-in recipes it finds:
%.out: %
# recipe to execute (built-in):
#rm -f $#
cp $< $#
as you can verify by running:
$ make --print-data-base | grep -A4 '%.out'
The rule for this recipe is matched by:
good/test1.out: good/test1
and by doing:
$ touch good/test1
you have made good/test1.out out of date with respect to good/test1.
So make executes the recipe:
#rm -f good/test1.out
cp good/test1 good/test1.out
the visible output of which is what you observed:
cp good/test1 good/test1.out
Then it proceeds with the recipe for test:
./test.sh
blah
There is always a risk of such booby-traps if you write a makefile that blindly
generates at runtime some set of preqrequisites or targets you don't know beforehand.
You could avoid this one in particular by explicitly deleting the offending
implicit pattern rule in your makefile by writing:
%.out: %
with no recipe. And you can avoid all possible booby-traps of this sort by disabling all
built-in recipes, with:
$ make --no-builtin-rules ...
but that will require you to write for yourself any builtin-recipes that your
makefile relies on.
The best solution for you is probably to amend your makefile as follows:
PREREQS := $(shell find good)
test : test.sh $(PREREQS)
./test.sh
$(PREREQS): ;
Then the last line explicitly specifies an empty recipe
for each of the $(PREREQS), and Make will not consult any pattern rules for targets
that have explicit recipes.
You should additionally make test a phony target:
.PHONY: test
for the avoidance of the booby-trap where something creates a file called test in the build directory.
I have my project binary located at my repository's root, along with a Makefile used to build it.
This binary uses many of my self-made libraries, located in my lib/ folder
For the purpose of building (and cleaning) my repository's binary, I want to implement the following execution :
Instead of hardcoding the following lines,
clean_binaries:
make -C clean lib/folder1 -s
make -C clean lib/folder2 -s
make -C clean lib/another_folder -s
I created the BIN_PATH variable, containing the previous paths.
BIN_PATHS = lib/folder1 \
lib/folder2 \
lib/another_folder
And made a simple rule like this one :
clean_binaries: $(BIN_PATHS)
make -C clean $< -s
BUT it only executes the line for the first field of the variable (lib/folder1), which is not what I want to do.
I thought about using implicit rules(?), just like I compile the .c files, but I couldn't get it right.
In the end, I simply wonder how to execute a rule for every field of a given variable, and this inside a Makefile, if there is any way to do so.
Thank you for your answers :]
The way you get GNU make to generate a sequence of commands that vary by the
fields in a variable is to use the foreach function, e.g.
Makefile
BIN_PATHS := lib/folder1 lib/folder2 lib/another_folder
.PHONY: clean_binaries
clean_binaries:
$(foreach path,$(BIN_PATHS),make -C $(path) clean ;)
which runs like:
$ make
make -C lib/folder1 clean -s; make -C lib/folder2 clean -s; make -C lib/another_folder clean -s;
not requiring a shell-loop.
Note also that you need to correct:
make -C clean <directory>
to:
make -C <directory> clean
So here's a Makefile to install foo.conf, based on a master copy called foo.conf.master. It installs it to the current directory rather than /etc, just for testing purposes:
all: foo.conf.copied
foo.conf.copied: foo.conf.master foo.conf
cp foo.conf.master foo.conf
touch $#
# Recipe to tell make that it is okay for foo.conf not to exist beforehand.
foo.conf:
So then create foo.conf.master:
$ touch foo.conf.master
$
and you're ready to test:
$ make
cp foo.conf.master foo.conf
touch foo.conf.copied
$
The point is that if I (with my "trusted" sysadmin hat on) modify foo.conf.master then make (possibly called by cron) will roll out the update:
$ touch foo.conf.master
$ make
cp foo.conf.master foo.conf
touch foo.conf.copied
$
But equally important: if I (with my "rogue" sysadmin hat on) modify the installed version then make will back out the update:
$ touch foo.conf
$ make
cp foo.conf.master foo.conf
touch foo.conf.copied
$
Woohoo.
Okay, so now the problem: obviously foo.conf isn't the only file I want do this for, so I need to change my static rules to pattern rules. Okay, that's easy: substitute foo.conf for % in targets and dependencies, substitute foo.conf for $* in the commands, and make a minor modification to the last recipe (which would otherwise become only '%:') so that it doesn't look like I'm trying to cancel a builtin pattern rule.
So clean up and create this Makefile:
all: foo.conf.copied
%.copied: %.master %
cp $*.master $*
touch $#
# Recipe to tell make that it is okay for foo.conf not to exist beforehand.
# Nop tells make that I'm not *cancelling* a pattern rule here
# (see http://stackoverflow.com/questions/34315150/make-implicit-rules-dont-work-without-command).
%: ;
But this doesn't work:
$ make
make: *** No rule to make target `foo.conf.copied', needed by `all'. Stop.
$
The error message is misleading; it is really foo.conf that it doesn't know how to make, which can be demonstrated by adding the following at the bottom of the Makefile:
foo.conf:
touch $#
But then that's a static rule again, which I don't want.
There are a couple more requirements I would also like to satisfy, which the above example doesn't demonstrate. These are:
foo.conf should be installable anywhere in the filesystem (e.g. /etc/foo/server/foo.conf)
foo.conf.master should be in a central directory, or subdirectly thereof, for all master versions, preferably without the '.master' extension (e.g. ~/poor-mans-puppet/master-files/etc/foo/foo.conf)
foo.conf.copied should be in a central directory, not in the same directory as foo.conf (e.g. ~/poor-mans-puppet/timestamp-files/etc/foo/foo.conf)
After much googling, hair pulling, I'm asking here! Any ideas please? (PS: if copying Makefiles from here, remember to change indentation back to tabs.)
Mad Scientist below suggested an elegant static rule, but I really need it to be a pattern rule. The reason is that I need to hook extra dependencies in using rules:
all: <new-dependency>
rather than hooking them in using variables:
STUFF_ALL_SHOULD_DEPEND_ON += <new-dependency>
The reason for this requirement is for consistency with how other (non-%.copied) targets are handled in my very large Makefile.
However, based on Mad Scientist's idea, I tried the following, which didn't work, but perhaps helps somebody to help me:
all: foo.conf.copied
%.copied: %.master %
$(eval FILES_FOR_WHICH_AN_EMPTY_RECIPE_ARE_NEEDED += $$*)
cp $*.master $*
touch $#
define GENERATE_STATIC_EMPTY_RULE
$(1):
endef
$(foreach X,$(FILES_FOR_WHICH_AN_EMPTY_RECIPE_ARE_NEEDED),$(eval $(call GENERATE_STATIC_EMPTY_RULE,$(X))))
Can you explain why you're using this extra ".copied" file? Why don't you just use:
%: %.master ; cp $< $#
?
Anyway, you're running afoul of make's special rules related to match-anything rules (pattern rules like % that can build everything). If you change your pattern so it's not match-anything, like %.conf: ; then it will work. However you probably don't want to assume that all files end in .conf.
Alternatively you can use static pattern rules, like this:
FILES_TO_COPY = foo.conf bar.conf biz.baz
all: $(FILES_TO_COPY:%=%.copied)
$(FILES_TO_COPY:%=%.copied): %.copied : %.master %
cp $*.master $*
touch $#
and you don't need the extra pattern rule.
In the end, I dynamically generated static rules. The following pseudo-code hopefully makes the actual Makefile easier to understand:
if flag not set # flag won't be set on first call
prepare static rules
set flag # prevent running this clause again
recurse! # make invokes make
else
include static rules
do the normal thing
endif
Here's the real Makefile:
ifeq ($(MAKELEVEL),0)
all:
for X in $(patsubst %.copied,%,$^); do \
echo "$$X.copied: $$X.master $$X"; \
echo " cp $$X.master $$X"; \
echo " touch \$$#"; \
echo "$$X: ;"; \
done > Makefile.include
$(MAKE)
# The real dependencies on all are defined below, but while creating
# Makefile.include, we don't want make to digress and start making
# the dependencies; this pattern rule will stop it from doing that.
%.copied: ;
else
include Makefile.include
endif
all: foo.conf.copied
The outer make can be silenced by use of .SILENT and the --no-print-directory option (not shown above for clarity).
Here's the output:
$ touch foo.conf.master
$ make
cp foo.conf.master foo.conf
touch foo.conf.copied
$ touch foo.conf
$ make
cp foo.conf.master foo.conf
touch foo.conf.copied
$
I have defined a list of files in a Makefile.
I need to copy those files, with their directory structure, to a new location.
I can't use cp --parent because the source files live in ../ (so .. will be a part of the dest path).
I've decided, then, to do one for loop that creates the dirs, then a second for loop that copies the files. The problem, however, is that I can't seem to call $(dir $$file) in the body of the for loop. I always get "./" as the result, rather than the actual dir name. If I do something like echo $(dir foo/bar), that works fine, but echo $(dir $$file) (when $$file is 'foo/bar') always returns './'.
I've tried other options like $(shell basename $$file), but that also doesn't work.
I've tried defining a function and calling it in the body of the for, but that, too, doesn't work.
Here's a quick example Makefile:
FILES := foo/faz \
bar/baz \
gah/gaz
all:
#for f in $(FILES); do \
echo $(dir $$f); \
done
I expect the output of this to be:
foo
bar
gah
but instead, I'm getting:
./
./
./
I'm open to other solutions if my method is not the best. At the end of the day, I need to be able to copy all the $(FILES) that exist in ../ (so ../foo/bar, for example) to a new dir called $(NEWDIR) (so newdir/foo/bar, for example).
This only needs to work under Linux.
f is a shell variable here. The make function dir is called on the string $f (which the shell then interprets as a variable expansion).
If you use a shell loop, you need to use the shell's constructs to extract the directory part:
#set -e; for f in $(FILES); do \
echo $$(dirname $$f); \
done
Don't forget set -e, so that if there's an error during the copy, the make rule will fail.
You can also use a foreach function in make to generate a bunch of cp commands. It makes the makefile harder to debug though.
In my GNUmakefile, I would like to have a rule that uses a temporary directory. For example:
out.tar: TMP := $(shell mktemp -d)
echo hi $(TMP)/hi.txt
tar -C $(TMP) cf $# .
rm -rf $(TMP)
As written, the above rule creates the temporary directory at the time that the rule is parsed. This means that, even I don't make out.tar all the time, many temporary directories get created. I would like to avoid my /tmp being littered with unused temporary directories.
Is there a way to cause the variable to only be defined when the rule is fired, as opposed to whenever it is defined?
My main thought is to dump the mktemp and tar into a shell script but that seems somewhat unsightly.
In your example, the TMP variable is set (and the temporary directory created) whenever the rules for out.tar are evaluated. In order to create the directory only when out.tar is actually fired, you need to move the directory creation down into the steps:
out.tar :
$(eval TMP := $(shell mktemp -d))
#echo hi $(TMP)/hi.txt
tar -C $(TMP) cf $# .
rm -rf $(TMP)
The eval function evaluates a string as if it had been typed into the makefile manually. In this case, it sets the TMP variable to the result of the shell function call.
edit (in response to comments):
To create a unique variable, you could do the following:
out.tar :
$(eval $#_TMP := $(shell mktemp -d))
#echo hi $($#_TMP)/hi.txt
tar -C $($#_TMP) cf $# .
rm -rf $($#_TMP)
This would prepend the name of the target (out.tar, in this case) to the variable, producing a variable with the name out.tar_TMP. Hopefully, that is enough to prevent conflicts.
A relatively easy way of doing this is to write the entire sequence as a shell script.
out.tar:
set -e ;\
TMP=$$(mktemp -d) ;\
echo hi $$TMP/hi.txt ;\
tar -C $$TMP cf $# . ;\
rm -rf $$TMP ;\
I have consolidated some related tips here: Multi-line bash commands in makefile
Another possibility is to use separate lines to set up Make variables when a rule fires.
For example, here is a makefile with two rules. If a rule fires, it creates a temp dir and sets TMP to the temp dir name.
PHONY = ruleA ruleB display
all: ruleA
ruleA: TMP = $(shell mktemp -d testruleA_XXXX)
ruleA: display
ruleB: TMP = $(shell mktemp -d testruleB_XXXX)
ruleB: display
display:
echo ${TMP}
Running the code produces the expected result:
$ ls
Makefile
$ make ruleB
echo testruleB_Y4Ow
testruleB_Y4Ow
$ ls
Makefile testruleB_Y4Ow
I dislike "Don't" answers, but... don't.
make's variables are global and are supposed to be evaluated during makefile's "parsing" stage, not during execution stage.
In this case, as long as the variable local to a single target, follow #nobar's answer and make it a shell variable.
Target-specific variables, too, are considered harmful by other make implementations: kati, Mozilla pymake. Because of them, a target can be built differently depending on if it's built standalone, or as a dependency of a parent target with a target-specific variable. And you won't know which way it was, because you don't know what is already built.