I wish to write several rules that extract the contents of tar archives to produce a number of files that are then used as input dependencies for other rules. I wish this to work even with parallel builds. I'm not using recursive make.
First up, sorry for the marathon question, but I don't think I can explain it well in a shorter form.
Think of untarring a collection of source files and then compiling them with rules stored outside of the archive to produce various build artefacts that are then, in turn, used further. I am not seeking other arrangements that lead to the omission of this problem. Just take it for granted that I have good reason to do this. :)
I'll demonstrate my issue with a contrived example. Of course, I started with something basic:
TAR := test.tar.bz2
CONTENTS := $(addprefix out/,$(filter-out %/,$(shell tar -tf $(TAR))))
out: $(TAR)
rm -rf out
mkdir out
tar -xvf $< -C out --touch || (rm -rf out; exit 1)
$(CONTENTS): out
sums: $(CONTENTS)
md5sum $^ > $#
.DELETE_ON_ERROR:
.DEFAULT_GOAL := all
.PHONY: all clean
all: sums
clean:
rm -rf out sums
The thinking here is that since $(CONTENTS) are all of the files in the archive, and they all depend on out, then to run the sums target we need to end up extracting the archive.
Unfortunately, this doesn't (always) work if you use a parallel invocation after a previous build when only test.tar.bz2 is updated, because make may decide to check the timestamp of $(CONTENTS) before running the out rule, which means it thinks that each of the sources is older than sums, so there is nothing to do:
$ make clean
rm -rf out sums
$ make -j6
rm -rf out
mkdir out
tar -xvf test.tar.bz2 -C out --touch || (rm -rf out; exit 1)
data.txt
file
weird.file.name
dir/
dir/another.c
dir/more
md5sum out/data.txt out/file out/weird.file.name out/dir/another.c out/dir/more > sums
$ touch test.tar.bz2
$ make -j6
rm -rf out
mkdir out
tar -xvf test.tar.bz2 -C out --touch || (rm -rf out; exit 1)
data.txt
file
weird.file.name
dir/
dir/another.c
dir/more
Oops! The sums rule didn't run!
So, the next attempt was to tell make that the one untar rule actually does make all the $(CONTENTS) directly. This seems better since we're telling make what's really going on, so it knows when to forget any cached timestamps for targets when they are remade through their rule.
First, let's look at what seems to work, and then I'll get to my problem:
TAR := test.tar.bz2
CONTENTS := $(addprefix out/,$(filter-out %/,$(shell tar -tf $(TAR))))
# Here's the change.
$(addprefix %/,$(patsubst out/%,%,$(CONTENTS))): $(TAR)
rm -rf out
mkdir out
tar -xvf $< -C out --touch || (rm -rf out; exit 1)
sums: $(CONTENTS)
md5sum $^ > $#
.DELETE_ON_ERROR:
.DEFAULT_GOAL := all
.PHONY: all clean
all: sums
clean:
rm -rf out sums
In this case, we've effectively got a rule that says:
%/data.txt %/file %/weird.file.name %/dir/another.c %/dir/more: test.tar.bz2
rm -rf out
mkdir out
tar -xvf $< -C out --touch || (rm -rf out; exit 1)
Now you can see one of the reasons I forced the output into an out directory: to give me a place to use the % so I could use a pattern rule. I am forced to use a pattern rule even though there isn't a strong pattern here because it is the only way make can be told that one rule creates multiple output files from a single invocation. (Isn't it?)
This works if any of the files are touched (not important for my use case) or if the test.tar.bz2 file is touched, even in parallel builds, because make has the information it needs: running this recipe makes all these files and will change all their timestamps.
For example, after a previous successful build:
$ touch test.tar.bz2
$ make -j6
rm -rf out
mkdir out
tar -xvf test.tar.bz2 -C out --touch || (rm -rf out; exit 1)
data.txt
file
weird.file.name
dir/
dir/another.c
dir/more
md5sum out/data.txt out/file out/weird.file.name out/dir/another.c out/dir/more > sums
So, if I have a working solution, what's my problem?
Well, I have many of these archives to extract, each with their own set of $(CONTENTS). I can manage that, but the trouble comes in writing a nice pattern rule. Since each archive needs its own rule defined, the patterns for each rule must not overlap even if the archives have similar (or identical) content. That means the output paths for the extracted files must be made unique for each archive, as in:
TAR := test.tar.bz2
CONTENTS := $(addprefix out.$(TAR)/,$(filter-out %/,$(shell tar -tf $(TAR))))
$(patsubst out.$(TAR)/%,out.\%/%,$(CONTENTS)): $(TAR)
rm -rf out.$(TAR)
mkdir out.$(TAR)
tar -xvf $< -C out.$(TAR) --touch || (rm -rf out.$(TAR); exit 1)
sums: $(CONTENTS)
md5sum $^ > $#
.DELETE_ON_ERROR:
.DEFAULT_GOAL := all
.PHONY: all clean
all: sums
clean:
rm -rf out.$(TAR) sums
So, this can be made to work with the right target-specific variables, but it now means that the extraction points are all "ugly" in a way that is very specifically tied to how the makefile is constructed:
$ make -j6
rm -rf out.test.tar.bz2
mkdir out.test.tar.bz2
tar -xvf test.tar.bz2 -C out.test.tar.bz2 --touch || (rm -rf out.test.tar.bz2; exit 1)
data.txt
file
weird.file.name
dir/
dir/another.c
dir/more
md5sum out.test.tar.bz2/data.txt out.test.tar.bz2/file out.test.tar.bz2/weird.file.name out.test.tar.bz2/dir/another.c out.test.tar.bz2/dir/more > sums
The next natural step I took was to try to combine static pattern rules with the multiple-targets-via-pattern-rule approach. This would let me keep the patterns very general, but limit their application to a specific set of targets:
TAR := test.tar.bz2
CONTENTS := $(addprefix out/,$(filter-out %/,$(shell tar -tf $(TAR))))
# Same as second attempt, except "$(CONTENTS):" static pattern prefix
$(CONTENTS): $(addprefix %/,$(patsubst out/%,%,$(CONTENTS))): $(TAR)
rm -rf out
mkdir out
tar -xvf $< -C out --touch || (rm -rf out; exit 1)
sums: $(CONTENTS)
md5sum $^ > $#
.DELETE_ON_ERROR:
.DEFAULT_GOAL := all
.PHONY: all clean
all: sums
clean:
rm -rf out sums
Great! Except it doesn't work:
$ make
Makefile:5: *** multiple target patterns. Stop.
$ make --version
GNU Make 4.0
So, is there a way to use multiple target patterns with a static pattern rule? If not, is there another way to achieve what I have in the last working example above, but without the constraint on the output paths to make unique patterns? I basically need to tell make "when you unpack this archive, all of the files in this directory (which I am willing to enumerate if necessary) have new timestamps". A solution where I can force make to restart if and only if it unpacks an archive would also be acceptable, but less ideal.
The problem with your original makefile is that you have a collision in names. You have a target (non-phony) named out and a directory named out. make thinks those are the same thing and gets very confused.
(Note: I added .SUFFIXES: to your first makefile to cut down on some noise but it doesn't change anything. The -r and -R flags disable make built-in rules and variables also for noise reduction.)
$ make clean
....
$ make -j6
....
$ touch test.tar.bz2
$ make -rRd -j6
....
Considering target file 'all'.
File 'all' does not exist.
Considering target file 'sums'.
Considering target file 'out/data.txt'.
Looking for an implicit rule for 'out/data.txt'.
No implicit rule found for 'out/data.txt'.
Considering target file 'out'.
Considering target file 'test.tar.bz2'.
Looking for an implicit rule for 'test.tar.bz2'.
No implicit rule found for 'test.tar.bz2'.
Finished prerequisites of target file 'test.tar.bz2'.
No need to remake target 'test.tar.bz2'.
Finished prerequisites of target file 'out'.
Prerequisite 'test.tar.bz2' is older than target 'out'.
No need to remake target 'out'.
Finished prerequisites of target file 'out/data.txt'.
Prerequisite 'out' is older than target 'out/data.txt'.
No recipe for 'out/data.txt' and no prerequisites actually changed.
No need to remake target 'out/data.txt'.
.... # This following set of lines repeats for all the other files in the tarball.
Considering target file 'out/file'.
Looking for an implicit rule for 'out/file'.
No implicit rule found for 'out/file'.
Pruning file 'out'.
Finished prerequisites of target file 'out/file'.
Prerequisite 'out' is older than target 'out/file'.
No recipe for 'out/file' and no prerequisites actually changed.
No need to remake target 'out/file'.
....
Finished prerequisites of target file 'sums'.
Prerequisite 'out/data.txt' is older than target 'sums'.
Prerequisite 'out/file' is older than target 'sums'.
Prerequisite 'out/weird.file.name' is older than target 'sums'.
Prerequisite 'out/dir/more' is older than target 'sums'.
Prerequisite 'out/dir/another.c' is older than target 'sums'.
No need to remake target 'sums'.
Finished prerequisites of target file 'all'.
Must remake target 'all'.
Successfully remade target file 'all'.
make: Nothing to be done for 'all'.
The main details here are these two lines:
Considering target file 'out'.
Prerequisite 'out' is older than target 'out/data.txt'
The out directory doesn't matter here. We don't care about it (and make doesn't deal with directory prerequisites too well anyway because modification timestamps on directories don't mean the same thing as they do on files). Even more to the point you don't want out/data.txt not being created because the build artifact directory target already existed (and seemed older).
You can "fix" this by marking the out target as .PHONY but that is just going to get make to extract the tarball every time you run make (you already run tar -tf every time you run make so it would probably be better to just combine those two steps if you were going to do this).
That said I wouldn't do that. I think the simplest solution to this problem is the "atomic rules" idea from John Graham-Cunning built-up and explained here.
sp :=
sp +=
sentinel = .sentinel.$(subst $(sp),_,$(subst /,_,$1))
atomic = $(eval $1: $(call sentinel,$1) ; #:)$(call sentinel,$1): $2 ; touch $$# $(foreach t,$1,$(if $(wildcard $t),,$(shell rm -f $(call sentinel,$1))))
.PHONY: all
all: a b
$(call atomic,a b,c d)
touch a b
You could probably also do this with an extraction stamp file (prereq on the tarball), extracting the tarball to a "shadow" directory and copy/link to the "final" location (build/$file: shadow/$file target) if you wanted to but that's going to be a bit more complicated I think.
Related
I'm trying to simply copy files that are modified using make. Here is the entire Makefile:
FILES = www/foo.html www/bar.html www/zap.php
all: $(FILES)
$(FILES): src/$#
cp src/$# $#
clean:
rm $(FILES)
After modifying a file src/www/bar.html, make does not copy the file:
$ make
make: Nothing to be done for 'all'.
$ make www/bar.html
make: 'www/bar.html' is up to date.
Why does make not see the prerequisite has been modified and that the file needs to be copied?
If I run make clean, make it works (copies all files).
src/$# is not well-defined. You want
$(FILES): %: src/%
which declares a pattern rule, and restricts its scope to the files in $(FILES). (You might want or even need to remove this restriction.)
I need a Makefile that allows me to enter make foo-program and, if any foo-program/**/*.hs file has changed since last build, build the target (output in foo-program/.stack-work).
Here is my directory tree:
project/
|-bar-other-program/
|-.stack-work/ # Generated output goes here
|-src/
|-BarOtherProgram.hs
|-test/
|-Tests.hs
|-foo-program/
|-.stack-work/ # Generated output goes here
|-src/
|-FooProgram.hs
|-test/
|-Tests.hs
|-notes/ # non-source, so no Make target for this
Here is what I have so far:
# evaluates to 'bar-other-program foo-program'
PROGS := $(shell find * -type f -name '*.hs' | cut -d'/' -f1 | uniq)
.SECONDEXPANSION:
$(PROGS): $$(wildcard $$#/src/*.hs) $$(wildcard $$#/test/*.hs)
# do-build $#
When I run make foo-program, whether the source has changed or not, I get:
make: Nothing to be done for 'foo-program'
UPDATE: My final (non-abstracted) Makefile can be found on GitHub. Note that my solution took a different turn than I intended when I wrote up this question. Looking at that Makefile also might also make it more clear as to my original goal.
I am not quite sure of the the purpose of cut -d'/' there.
But if you just want a list of *.hs files in the current directory (recursively found) and then build a target/executable based on whether they have changed, you can do something like this:
PROGS = $(subst ./,,$(shell find . -type f -name '*.hs'))
DEPS = $(addprefix stackwork/,$(addsuffix .dep,$(basename $(PROGS))))
DIRS = $(dir $(DEPS))
.PHONY: foo-program
foo-program: $(DEPS) $(DIRS)
stackwork/%.dep: %.hs | $(DIRS)
#echo making $#
#touch $#
$(DIRS):
#echo creating dir $#
#mkdir -p $#
clean:
#rm -rf $(DEPS) $(DIRS)
Where:
PROGS is your list of .hs files
DEPS is a list of generated dependency files (empty but date stamps will be used)
DIRS is a list of output directories that need to be created (I guess they don't exist by default since they are output folders?)
foo-program is a rule that you can call (PHONY because at the moment it does not create a real file)
%.dep: %.hs is a rule how to generate a .dep file (this could be a .o .obj or any other file type) which depends on its .hs file equivalent.
$(DIRS): is a rule to create your output directories if needed.
So if the .dep files don't exist, all of the .hs files will be "compiled". If all the .dep files exist and are up to date, then nothing will be compiled. If one or more file is out of date then just those files will be built. Here is the output of running this on my PC with a few test files:
admin#osboxes:~/sandbox$ make
creating dir stackwork/
creating dir stackwork/test/
creating dir stackwork/test/test2/
making stackwork/file.dep
making stackwork/test/file.dep
making stackwork/test/test2/file2.dep
admin#osboxes:~/sandbox$ make
make: Nothing to be done for 'foo-program'.
admin#osboxes:~/sandbox$ touch test/file.hs
admin#osboxes:~/sandbox$ make
making stackwork/test/file.dep
admin#osboxes:~/sandbox$ make
make: Nothing to be done for 'foo-program'.
Hi!
I started messing with makefiles a few days ago, so I gave myself a exercise to learn as well as making my life easier.
I basically want to read the directory the makefile is in (root) and use luamin to compress the file as much as possible before I deploy it to our server. But I would like to have it as flexible as possible, so depending on where said file is in the directory it should mirror it to the server.
So if it finds a file in a sub folder called home it should create a new folder with the same name with the compressed file within. I have gotten the compression of files in the root folder working as well as creation of the directories where the files should reside.
objs = $(wildcard *.lua)
dirs = $(wildcard */)
compress: $(objs)
mkdir -p .build
luamin -f $(objs) > .build/$(objs)
mkdir .build/$(dirs)
clean:
rm -rf ./.build
deploy: .build
cp ./.build/* ~
If you use GNU make, there are several features that really help to do what you want. Warning: this works if and only if your file names do not contain spaces:
srcfiles := $(shell find . -path .build -prune -o -type f -name '*.lua' -print)
dstfiles := $(addprefix .build/,$(srcfiles))
.PHONY: compress clean deploy
compress: $(dstfiles)
$(dstfiles): .build/%: %
mkdir -p $(dir $#)
luamin -f $< > $#
clean:
rm -rf ./.build
deploy: .build
cp ./.build/* ~
Explanation:
The shell make function is used to run the find command that searches all subdirectories, except .build, for *.lua files. The result is assigned to the srcfiles make variable.
The addprefix make function is used to add the .build/ prefix to all words of the srcfiles make variable and assign the result to the dstfiles make variable.
The compress target is the first (real) target in the Makefile. It is thus the default goal that gets run when invoking just make. It is the same as invoking make compress. The compress target is declared as phony. This tells make that it is not a real file, just like clean and deploy. The compress target depends on all destination files. If one is missing or older than its corresponding source file, it must be rebuilt.
The make static pattern rule $(dstfiles): .build/%: %... declares a generic rule where each destination file (.build/./foo/bar/baz.lua) depends on the corresponding source file (./foo/bar/baz.lua). The recipe creates the destination directory (./foo/bar/), computed thanks to the dir make function. Then, it applies the luamin command. The recipe makes use of the $# and $< automatic variables.
I have a Makefile as :
BUILD_DIR= $(BASE_DIR)/build
_OBJ := a.o b.o
CLEAN_OBJECTS := $(_OBJ)
.PHONY: clean
create_code:
python ../script/my_script.py
all: create_code $(_OBJ)
$(_OBJ): %.o: %.c
mkdir -p $(BUILD_DIR)
$(CC) $(CFLAGS) $(INCLUDE_PATH) -c $< -o $#
cp *.o $(BUILD_DIR)
clean:
rm -f $(CLEAN_OBJECTS)
The target create_code executes the python script and generates a set of .c/.h files.
The target _obj compiles them.
Every time I run make all , create_code target is run even though there is no change to .c/.h generated earlier .
Any suggestions on why this is happening and how to make this create_code target run only if make clean was done earlier .
The underlying problem is that you have one or more generated files that depend on something other than the underlying file system -- namely the contents of your database.
One possibility would be to take advantage of the fact that make, having invoked a rule to rebuild a target will, nonetheless, always check the update time of that target when it is specified as a prerequisite in any other rule.
So, given the rule (untested)...
.PHONY: FORCE
%.c: FORCE
command-to-generate-source $#.tmp
diff -q $#.tmp $# || cp $#.tmp $#
Invoking make foo.c from the command line. make will run the commands...
command-to-generate-source foo.c.tmp
diff -q foo.c.tmp foo.c || cp foo.c.tmp foo.c
where command-to-generate-source foo.c.tmp is expected to leave its output in foo.c.tmp. If the newly generated output file is different than the existing file the cp operation will be run and, hence, the target timestamp will be updated and anything dependent on the target will be updated accordingly.
If, however, the newly generated output file is the same as the existing one then no cp will be run, the target file will be left untouched and make will not consider it to be changed when it appears as a prerequisite in other rules.
This is just one possibility but it's the obvious one given that you already have most (if not all) of the required logic in the command python ../script/my_script.py.
I am trying to copy files using a my rule but my rule does not get triggered:
BUILDDIR = build
COPY_FILES = code/xml/schema/schema.xsd config.txt
all: $(BUILDDIR) $(COPY_FILES) copy
$(BUILDDIR):
mkdir -p $#
$(COPY_FILES):
cp -f $# $(BUILDDIR)
copy:
cp -f $(COPY_FILES) $(BUILDDIR)
I am trying to use $(COPY_FILES) but it is not being triggered, although $(BUILDDIR) and copy are triggered. I am not sure what is wrong with my Makefile. I would like to get the $(COPY_FILES) rule to work if possible please (and remove copy). Does anyone please know?
The problem with the $(COPY_FILES) rule is that the targets of that rule are two files that already exist, namely code/xml/schema/schema.xsd and config.txt. Make sees no reason to execute the rule. I'm not sure why Make doesn't execute the copy rule, but I suspect that there's a file called copy confusing the matter. Anyway, [copy] a bad rule.
Try this:
COPY_FILES = $(BUILD_DIR)/schema.xsd $(BUILD_DIR)/config.txt
all: $(COPY_FILES)
$(BUILD_DIR)/schema.xsd: code/xml/schema/schema.xsd
$(BUILD_DIR)/config.txt: config.txt
$(BUILD_DIR)/%:
cp -f $< $#
In my case, I use a simple "for loop" to cp all those files.
For examples, write the rule as following:
RELEASE_DIR = ../rc1
RELEASE_FILES = ai.h main.cc main_async.cc bmp_utils.h bmp_utils.cc
release: $(RELEASE_FILES)
for u in $(RELEASE_FILES); do echo $$u; cp -f $$u $(RELEASE_DIR); done
Then,
make release