I am using a Makefile to automate the installation of some tools, some of which I download
all:bin/symfony
bin/symfony: var/bin/symfonyInstaller
#echo making $#
#$< --install-dir bin > /dev/null 2>&1
var/bin/symfonyInstaller: var/bin
#echo making $#
#wget -q https://get.symfony.com/cli/installer -O $# >/dev/null 2>&1
#chmod a+x $# > /dev/null
var/bin:
#echo making $#
#mkdir -p $# > /dev/null
clean:
#echo making $#
#rm -rf bin var
For some reasons, var/bin/symfonyInstaller is always being remade :
$ make
making var/bin
making var/bin/symfonyInstaller
making bin/symfony
$ make
making var/bin/symfonyInstaller
$
Is there a way to tell make "don't rebuild var/bin/symfonyInstaller" ?
Your var/bin/symfonyInstaller target is always rebuilt because it depends on the directory in which it goes. And each time it is built the directory's last modification time is updated and becomes newer than the target. Thus the endless cycle.
What you want is called an order-only prerequisite. Every prerequisite mentioned in a make rule before the | separator is a "normal" one. After the | separator it is an order-only prerequisite.
All prerequisites, normal or order-only, are updated before the target, if they don't exist or if they are outdated. The difference is in how make decides to rebuild or not the corresponding target. Make rebuilds the target if it updated a normal prerequisite, or if the normal prerequisite is newer than the target. But not if it is an order-only prerequisite that has been updated or is newer than the target.
In your case you should use it for var/bin:
var/bin/symfonyInstaller: | var/bin
#echo making $#
#wget -q https://get.symfony.com/cli/installer -O $# >/dev/null 2>&1
#chmod a+x $# > /dev/null
If var/bin does not exist it is created before var/bin/symfonyInstaller, which is good, but var/bin/symfonyInstaller will not be rebuilt just because it is older than var/bin.
Related
I am trying to write a Makefile that builds PDF outputs with LaTeX, using Latexmk. The outputs have basically the same rule, with different prerequisites, so I tried generalising my original Makefile using GNU Make's "secondary expansion". (I also created .PHONY targets, also with secondary expansion, to make it more user-friendly.) However, this is causing the prerequisite rules to always be run, even when they don't need to be. Fortunately, Latexmk is clever enough to avoid doing unnecessary work, but I wonder if I'm doing anything wrong...
To try to abstract what I'm attempting:
,-> foo -> build/foo.pdf
all -{
`-> bar -> build/bar.pdf
That is, the all target builds foo and bar. These targets open the respective PDF file, which have a prerequisite of build/X.pdf (where X is foo or bar). These are genuine targets which build the appropriate PDF file.
This is what I've come up with:
TARGETS = foo bar
BUILD_DIR = build
OUTPUTS = $(TARGETS:%=$(BUILD_DIR)/%.pdf)
commonSRC = src/preamble.tex src/header.tex # etc...
fooSRC = src/foo.tex $(commonSRC) # etc...
barSRC = src/bar.tex $(commonSRC) # etc...
all: $(TARGETS)
.SECONDEXPANSION:
$(TARGETS): $(BUILD_DIR)/$$#.pdf
open $<
# FIXME This isn't quite right: This rule is still getting called by the
# above rule, even when it doesn't need to be. Latexmk is clever enough
# not to do any extra work, but it shouldn't run at all.
.SECONDEXPANSION:
$(OUTPUTS): $$($$(subst .pdf,SRC,$$(#F))) $(BUILD_DIR)
latexmk -outdir=$(BUILD_DIR) -auxdir=$(BUILD_DIR) -pdf $<
$(BUILD_DIR):
mkdir $#
clean:
rm -rf $(BUILD_DIR)
.PHONY: all $(TARGETS) clean
Just to be clear: The rule for build/X.pdf should run whenever the files enumerated in XSRC (again, where X is foo or bar) are newer than the PDFs, or the PDFs don't exist; but it should not run otherwise.
I believe that this got somewhat complex, more than it needs to be. Part of these second expansion statements can be just replaced with static pattern rules. The other thing is that .SECONDEXPANSION: makes all further Makefile contents to be subject to second expansion, so you don't need to explicitly state it before every target (it would be much clearer to mark .PHONY targets this way to quickly see if a target is phony or not).
Nevertheless, I believe that most important issue here is that you mention a directory as a prerequisite. Remember that make decides on whether to rebuild the target based on dependencies timestamp, and a directory gets its timestamp always updated whenever a file in this directory is updated. Therefore, whenever you write $(BUILD_DIR)/foo.pdf, $(BUILD_DIR) timestamp gets updated and the next call will build again since the directory is newer. You can avoid it by specifying a directory as an order-only prerequisite (which means: build if it doesn't exist, but do not check timestamp).
Putting it all together I would make it this way:
TARGETS = foo bar
BUILD_DIR = build
commonSRC = src/preamble.tex src/header.tex # etc...
fooSRC = src/foo.tex $(commonSRC) # etc...
barSRC = src/bar.tex $(commonSRC) # etc...
.SECONDEXPANSION:
.PHONY: all
all: $(TARGETS)
.PHONY: $(TARGETS)
$(TARGETS): %: $(BUILD_DIR)/%.pdf
echo open $<
$(BUILD_DIR)/%.pdf: $$($$*SRC) | $(BUILD_DIR)
echo latexmk -outdir=$(BUILD_DIR) -auxdir=$(BUILD_DIR) -pdf $< > $#
$(BUILD_DIR):
mkdir -p $#
.PHONY: clean
clean:
rm -rf $(BUILD_DIR)
Output:
$ make all
mkdir -p build
echo latexmk -outdir=build -auxdir=build -pdf src/foo.tex > build/foo.pdf
echo open build/foo.pdf
open build/foo.pdf
echo latexmk -outdir=build -auxdir=build -pdf src/bar.tex > build/bar.pdf
echo open build/bar.pdf
open build/bar.pdf
$ make all
echo open build/foo.pdf
open build/foo.pdf
echo open build/bar.pdf
open build/bar.pdf
Note that subsequent call did not attempt to build anything, just open the pdf. It still reacts on the file change however:
$ touch src/foo.tex
$ make all
echo latexmk -outdir=build -auxdir=build -pdf src/foo.tex > build/foo.pdf
echo open build/foo.pdf
open build/foo.pdf
echo open build/bar.pdf
open build/bar.pdf
$ touch src/header.tex
$ make all
echo latexmk -outdir=build -auxdir=build -pdf src/foo.tex > build/foo.pdf
echo open build/foo.pdf
open build/foo.pdf
echo latexmk -outdir=build -auxdir=build -pdf src/bar.tex > build/bar.pdf
echo open build/bar.pdf
open build/bar.pdf
I have a Makefile as :
BUILD_DIR= $(BASE_DIR)/build
_OBJ := a.o b.o
CLEAN_OBJECTS := $(_OBJ)
.PHONY: clean
create_code:
python ../script/my_script.py
all: create_code $(_OBJ)
$(_OBJ): %.o: %.c
mkdir -p $(BUILD_DIR)
$(CC) $(CFLAGS) $(INCLUDE_PATH) -c $< -o $#
cp *.o $(BUILD_DIR)
clean:
rm -f $(CLEAN_OBJECTS)
The target create_code executes the python script and generates a set of .c/.h files.
The target _obj compiles them.
Every time I run make all , create_code target is run even though there is no change to .c/.h generated earlier .
Any suggestions on why this is happening and how to make this create_code target run only if make clean was done earlier .
The underlying problem is that you have one or more generated files that depend on something other than the underlying file system -- namely the contents of your database.
One possibility would be to take advantage of the fact that make, having invoked a rule to rebuild a target will, nonetheless, always check the update time of that target when it is specified as a prerequisite in any other rule.
So, given the rule (untested)...
.PHONY: FORCE
%.c: FORCE
command-to-generate-source $#.tmp
diff -q $#.tmp $# || cp $#.tmp $#
Invoking make foo.c from the command line. make will run the commands...
command-to-generate-source foo.c.tmp
diff -q foo.c.tmp foo.c || cp foo.c.tmp foo.c
where command-to-generate-source foo.c.tmp is expected to leave its output in foo.c.tmp. If the newly generated output file is different than the existing file the cp operation will be run and, hence, the target timestamp will be updated and anything dependent on the target will be updated accordingly.
If, however, the newly generated output file is the same as the existing one then no cp will be run, the target file will be left untouched and make will not consider it to be changed when it appears as a prerequisite in other rules.
This is just one possibility but it's the obvious one given that you already have most (if not all) of the required logic in the command python ../script/my_script.py.
I wish to write several rules that extract the contents of tar archives to produce a number of files that are then used as input dependencies for other rules. I wish this to work even with parallel builds. I'm not using recursive make.
First up, sorry for the marathon question, but I don't think I can explain it well in a shorter form.
Think of untarring a collection of source files and then compiling them with rules stored outside of the archive to produce various build artefacts that are then, in turn, used further. I am not seeking other arrangements that lead to the omission of this problem. Just take it for granted that I have good reason to do this. :)
I'll demonstrate my issue with a contrived example. Of course, I started with something basic:
TAR := test.tar.bz2
CONTENTS := $(addprefix out/,$(filter-out %/,$(shell tar -tf $(TAR))))
out: $(TAR)
rm -rf out
mkdir out
tar -xvf $< -C out --touch || (rm -rf out; exit 1)
$(CONTENTS): out
sums: $(CONTENTS)
md5sum $^ > $#
.DELETE_ON_ERROR:
.DEFAULT_GOAL := all
.PHONY: all clean
all: sums
clean:
rm -rf out sums
The thinking here is that since $(CONTENTS) are all of the files in the archive, and they all depend on out, then to run the sums target we need to end up extracting the archive.
Unfortunately, this doesn't (always) work if you use a parallel invocation after a previous build when only test.tar.bz2 is updated, because make may decide to check the timestamp of $(CONTENTS) before running the out rule, which means it thinks that each of the sources is older than sums, so there is nothing to do:
$ make clean
rm -rf out sums
$ make -j6
rm -rf out
mkdir out
tar -xvf test.tar.bz2 -C out --touch || (rm -rf out; exit 1)
data.txt
file
weird.file.name
dir/
dir/another.c
dir/more
md5sum out/data.txt out/file out/weird.file.name out/dir/another.c out/dir/more > sums
$ touch test.tar.bz2
$ make -j6
rm -rf out
mkdir out
tar -xvf test.tar.bz2 -C out --touch || (rm -rf out; exit 1)
data.txt
file
weird.file.name
dir/
dir/another.c
dir/more
Oops! The sums rule didn't run!
So, the next attempt was to tell make that the one untar rule actually does make all the $(CONTENTS) directly. This seems better since we're telling make what's really going on, so it knows when to forget any cached timestamps for targets when they are remade through their rule.
First, let's look at what seems to work, and then I'll get to my problem:
TAR := test.tar.bz2
CONTENTS := $(addprefix out/,$(filter-out %/,$(shell tar -tf $(TAR))))
# Here's the change.
$(addprefix %/,$(patsubst out/%,%,$(CONTENTS))): $(TAR)
rm -rf out
mkdir out
tar -xvf $< -C out --touch || (rm -rf out; exit 1)
sums: $(CONTENTS)
md5sum $^ > $#
.DELETE_ON_ERROR:
.DEFAULT_GOAL := all
.PHONY: all clean
all: sums
clean:
rm -rf out sums
In this case, we've effectively got a rule that says:
%/data.txt %/file %/weird.file.name %/dir/another.c %/dir/more: test.tar.bz2
rm -rf out
mkdir out
tar -xvf $< -C out --touch || (rm -rf out; exit 1)
Now you can see one of the reasons I forced the output into an out directory: to give me a place to use the % so I could use a pattern rule. I am forced to use a pattern rule even though there isn't a strong pattern here because it is the only way make can be told that one rule creates multiple output files from a single invocation. (Isn't it?)
This works if any of the files are touched (not important for my use case) or if the test.tar.bz2 file is touched, even in parallel builds, because make has the information it needs: running this recipe makes all these files and will change all their timestamps.
For example, after a previous successful build:
$ touch test.tar.bz2
$ make -j6
rm -rf out
mkdir out
tar -xvf test.tar.bz2 -C out --touch || (rm -rf out; exit 1)
data.txt
file
weird.file.name
dir/
dir/another.c
dir/more
md5sum out/data.txt out/file out/weird.file.name out/dir/another.c out/dir/more > sums
So, if I have a working solution, what's my problem?
Well, I have many of these archives to extract, each with their own set of $(CONTENTS). I can manage that, but the trouble comes in writing a nice pattern rule. Since each archive needs its own rule defined, the patterns for each rule must not overlap even if the archives have similar (or identical) content. That means the output paths for the extracted files must be made unique for each archive, as in:
TAR := test.tar.bz2
CONTENTS := $(addprefix out.$(TAR)/,$(filter-out %/,$(shell tar -tf $(TAR))))
$(patsubst out.$(TAR)/%,out.\%/%,$(CONTENTS)): $(TAR)
rm -rf out.$(TAR)
mkdir out.$(TAR)
tar -xvf $< -C out.$(TAR) --touch || (rm -rf out.$(TAR); exit 1)
sums: $(CONTENTS)
md5sum $^ > $#
.DELETE_ON_ERROR:
.DEFAULT_GOAL := all
.PHONY: all clean
all: sums
clean:
rm -rf out.$(TAR) sums
So, this can be made to work with the right target-specific variables, but it now means that the extraction points are all "ugly" in a way that is very specifically tied to how the makefile is constructed:
$ make -j6
rm -rf out.test.tar.bz2
mkdir out.test.tar.bz2
tar -xvf test.tar.bz2 -C out.test.tar.bz2 --touch || (rm -rf out.test.tar.bz2; exit 1)
data.txt
file
weird.file.name
dir/
dir/another.c
dir/more
md5sum out.test.tar.bz2/data.txt out.test.tar.bz2/file out.test.tar.bz2/weird.file.name out.test.tar.bz2/dir/another.c out.test.tar.bz2/dir/more > sums
The next natural step I took was to try to combine static pattern rules with the multiple-targets-via-pattern-rule approach. This would let me keep the patterns very general, but limit their application to a specific set of targets:
TAR := test.tar.bz2
CONTENTS := $(addprefix out/,$(filter-out %/,$(shell tar -tf $(TAR))))
# Same as second attempt, except "$(CONTENTS):" static pattern prefix
$(CONTENTS): $(addprefix %/,$(patsubst out/%,%,$(CONTENTS))): $(TAR)
rm -rf out
mkdir out
tar -xvf $< -C out --touch || (rm -rf out; exit 1)
sums: $(CONTENTS)
md5sum $^ > $#
.DELETE_ON_ERROR:
.DEFAULT_GOAL := all
.PHONY: all clean
all: sums
clean:
rm -rf out sums
Great! Except it doesn't work:
$ make
Makefile:5: *** multiple target patterns. Stop.
$ make --version
GNU Make 4.0
So, is there a way to use multiple target patterns with a static pattern rule? If not, is there another way to achieve what I have in the last working example above, but without the constraint on the output paths to make unique patterns? I basically need to tell make "when you unpack this archive, all of the files in this directory (which I am willing to enumerate if necessary) have new timestamps". A solution where I can force make to restart if and only if it unpacks an archive would also be acceptable, but less ideal.
The problem with your original makefile is that you have a collision in names. You have a target (non-phony) named out and a directory named out. make thinks those are the same thing and gets very confused.
(Note: I added .SUFFIXES: to your first makefile to cut down on some noise but it doesn't change anything. The -r and -R flags disable make built-in rules and variables also for noise reduction.)
$ make clean
....
$ make -j6
....
$ touch test.tar.bz2
$ make -rRd -j6
....
Considering target file 'all'.
File 'all' does not exist.
Considering target file 'sums'.
Considering target file 'out/data.txt'.
Looking for an implicit rule for 'out/data.txt'.
No implicit rule found for 'out/data.txt'.
Considering target file 'out'.
Considering target file 'test.tar.bz2'.
Looking for an implicit rule for 'test.tar.bz2'.
No implicit rule found for 'test.tar.bz2'.
Finished prerequisites of target file 'test.tar.bz2'.
No need to remake target 'test.tar.bz2'.
Finished prerequisites of target file 'out'.
Prerequisite 'test.tar.bz2' is older than target 'out'.
No need to remake target 'out'.
Finished prerequisites of target file 'out/data.txt'.
Prerequisite 'out' is older than target 'out/data.txt'.
No recipe for 'out/data.txt' and no prerequisites actually changed.
No need to remake target 'out/data.txt'.
.... # This following set of lines repeats for all the other files in the tarball.
Considering target file 'out/file'.
Looking for an implicit rule for 'out/file'.
No implicit rule found for 'out/file'.
Pruning file 'out'.
Finished prerequisites of target file 'out/file'.
Prerequisite 'out' is older than target 'out/file'.
No recipe for 'out/file' and no prerequisites actually changed.
No need to remake target 'out/file'.
....
Finished prerequisites of target file 'sums'.
Prerequisite 'out/data.txt' is older than target 'sums'.
Prerequisite 'out/file' is older than target 'sums'.
Prerequisite 'out/weird.file.name' is older than target 'sums'.
Prerequisite 'out/dir/more' is older than target 'sums'.
Prerequisite 'out/dir/another.c' is older than target 'sums'.
No need to remake target 'sums'.
Finished prerequisites of target file 'all'.
Must remake target 'all'.
Successfully remade target file 'all'.
make: Nothing to be done for 'all'.
The main details here are these two lines:
Considering target file 'out'.
Prerequisite 'out' is older than target 'out/data.txt'
The out directory doesn't matter here. We don't care about it (and make doesn't deal with directory prerequisites too well anyway because modification timestamps on directories don't mean the same thing as they do on files). Even more to the point you don't want out/data.txt not being created because the build artifact directory target already existed (and seemed older).
You can "fix" this by marking the out target as .PHONY but that is just going to get make to extract the tarball every time you run make (you already run tar -tf every time you run make so it would probably be better to just combine those two steps if you were going to do this).
That said I wouldn't do that. I think the simplest solution to this problem is the "atomic rules" idea from John Graham-Cunning built-up and explained here.
sp :=
sp +=
sentinel = .sentinel.$(subst $(sp),_,$(subst /,_,$1))
atomic = $(eval $1: $(call sentinel,$1) ; #:)$(call sentinel,$1): $2 ; touch $$# $(foreach t,$1,$(if $(wildcard $t),,$(shell rm -f $(call sentinel,$1))))
.PHONY: all
all: a b
$(call atomic,a b,c d)
touch a b
You could probably also do this with an extraction stamp file (prereq on the tarball), extracting the tarball to a "shadow" directory and copy/link to the "final" location (build/$file: shadow/$file target) if you wanted to but that's going to be a bit more complicated I think.
I have this rule in my Makefile, that responds to flags I pass:
$(BUILD_DIR)/disable_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/disable_$*
rm -f $(BUILD_DIR)/enable_$*
cd $(BUILD_DIR) && rm -f Makefile
$(BUILD_DIR)/enable_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/enable_$*
rm -f $(BUILD_DIR)/disable_$*
cd $(BUILD_DIR) && rm -f Makefile
What this means is that when changing the flags by which I invoke the makefile, I can trigger some recompilations that could depend on these flags.
The code presented above is a bit redundant: you see that I remove a file, touch another and remove a Makefile in both cases. The only thing that changes is the name of the files that I touch/remove, and they are related.
For instance,
make clean
make enable_debug=yes enable_video=no # will compile from zero
make enable_debug=no enable_video=no # flag change detected -> recompile some submodules that depend on this flag
Provided that the only thing that changes between the two rules ( [en|dis]able ), what I would like is to only have 1 generic rule, something like that:
# match 2 parts in the rule
$(BUILD_DIR)/%ble_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/(???)ble_$* # should be $#
rm -f $(BUILD_DIR)/(???)able_$* # should be disable if $# is enable and inverse
cd $(BUILD_DIR) && rm -f Makefile
Is this possible ?
PS: Sorry if I didn't get the title correctly, I couldn't figure how to explain it better.
$(BUILD_DIR)/enable_% $(BUILD_DIR)/disable_%:
mkdir -p $(BUILD_DIR)
rm -f $(BUILD_DIR)/*able_$*
touch $#
cd $(BUILD_DIR) && rm -f Makefile
Not literally what you wanted (multi-wildcards are forbidden in make), but does quite the same.
How can I distinguish in makefile, which targets and how(when) they are called internally? I have a makefile with number of targets which are actually variables.
UPD: here is an example
build_dir := $(bin_dir)/build
xpi_built := $(build_dir)/$(install_rdf) \
$(build_dir)/$(chrome_manifest) \
$(chrome_jar_file) \
$(default_prefs)/*
xpi_built_no_dir := $(subst $(build_dir)/,,$(xpi_built))
.PHONY: install
install: $(build_dir) $(xpi_built)
#echo "Installing in profile folder: $(profile_location)"
#cp -Rf $(build_dir)/* $(profile_location)
#echo "Installing in profile folder. Done!"
#echo
$(xpi_file): $(build_dir) $(xpi_built)
#echo "Creating XPI file."
#cd $(build_dir); $(ZIP) -r ../$(xpi_file) $(xpi_built_no_dir)
#echo "Creating XPI file. Done!"
#cp update.rdf $(bin_dir)/
#cp -u *.xhtml $(bin_dir)/
#cp -Rf $(default_prefs) $(build_dir)/; \
$(build_dir)/%: %
cp -f $< $#
$(build_dir):
#if [ ! -x $(build_dir) ]; \
then \
mkdir $(build_dir); \
fi
If you specify a target on the command line, as in make clean Make will attempt to build that target. If you don't (that is, if you just run make), Make will attempt to build the default target; the default target is the first target in the makefile (in your case install) unless you set it to something else with the .DEFAULT_GOAL variable.
When Make tries to build a target, it first builds that target's prerequisites, if necessary. (When is it necessary? When a target is a file or directory that does not exist real (unless it's .PHONY, but that's an advanced topic), or when one of it's prerequisites is newer than the target (unless it's "order-only", but that's an advanced topic)). So if Make is trying to build your all, it will first try to build $(build_dir) and $(xpi_built), which have been defined elsewhere in the makefile.
If you're trying to figure out what Make will do and when, there are tricks you can use. For example, you can run make -n, and Make would tell you what it would do, instead of doing it. Or you can put a command like #echo now making $# in a rule, to tell you what it's doing. Or for even more information:
some_target: preq another_preq and_another
#echo Making $#. The prerequisites are $^. Of those, $? are newer than $#.
other_commands...