In my build system, I would like to avoid rebuilding a subproject if none of its files changed. The basic idea is this:
After doing make all, I hash the contents of the directory and save it.
The next time I want to build the project, hash the contents of the directory again.
If the hashes are equal, we are done. Otherwise, go to 1.
I'm trying to implement this in GNU Make, using this helper script named update-hash:
#!/bin/bash
HASH_FILE=$1
SRC_DIR=$2
echo Hashing sources
UPDATED_HASH=$(tar c $SRC_DIR | sha256sum | cut -c-64)
SAVED_HASH=$(<$HASH_FILE)
if [[ $SAVED_HASH != $UPDATED_HASH ]]; then
echo "Updating hash"
echo $UPDATED_HASH > $HASH_FILE
else
echo "$HASH_FILE is up to date. Not touching it."
fi
Here's the tricky part: since Make relies on file modification times to decide what to build, I use two files, pre_hash and post_hash, and take advantage of the fact that my update-hash script does not update the file if the hashes match.
subproject: update_pre_hash post_hash
post_hash: pre_hash
make -C subproject clean all
update-hash pre_hash subproject
cp pre_hash post_hash
update_pre_hash:
update-hash pre_hash subproject
The way this works is like this:
On the first run of make subproject, the pre_hash file gets updated, and so the post_hash rule gets built, ending with a built subproject and with a post_hash file that is newer than pre_hash.
On subsequent runs, update_pre_hash always gets invoked, but doesn't update the pre_hash file, and so the post_hash rule does not get invoked.
If any file in the subproject changes, the pre_hash file gets updated, and so post_hash gets invoked, resulting in a full build and new hash files.
My question is: will this actually work? Some basic testing did not reveal any glaring problems, but it seems a little shaky. I'm particularly worried about relying on the dependencies of subproject getting evaluated in order, which I know is an invalid assumption with make -j, but maybe there's a way to fix that.
Do you see a problem with this implementation?
If you do, can you suggest any fixes?
Do you know of another (maybe better) way to achieve the same effect? (other than the obvious "fix the subproject to no-op if there are no changes" - I don't want to maintain the subproject's build process)
Update: #dash-o suggested a much simpler approach, with a Makefile like this:
last_update: $(shell find subproject -type f)
make -C subproject clean all
touch last_update
This way, last_update is always newer than all the files in subproject, but if any files change later, a rebulid will be triggered. Thanks!
Related
I have a Makefile target that depends on a dynamic list of source files. This list is parsed on make invocations so it could potentially change. It looks like this:
target: $(shell cat foo.txt)
#echo "build target"
foo.txt changes from time to time. What I'm seeing is that target will be rebuilt the first time foo.txt changes content to X. Say that foo.txt changes content to Y then changes back to X then target won't be built. I know I can include foo.txt as a dependency itself but for reasons difficult to explain that's not an option.
As a more concrete example, foo.txt looks like this:
google.golang.org/grpc#v1.29.0/dialoptions.go
but could change to something like:
google.golang.org/grpc#v1.28.0/dialoptions.go
What's going on? Is Makefile caching something?
EDIT
I guess Makefile is caching the timestamp of the dependencies and thus if the dependency files didn't change at all then the target won't be rebuilt? Is there a way to fix this?
Make caches nothing. It has no preserved data whatsoever about any previous build. Make uses the filesystem as its "cache".
Because you're using $(shell cat foo.txt), make knows absolutely nothing about the file foo.txt. All it knows about is the set of files inside the foo.txt files. It treats all of those as prerequisites.
Then it merely compares the time last modified of the target file (target) with the time last modified of the prerequisite file that is contained in foo.txt.
So if foo.txt contains google.golang.org/grpc#v1.29.0/dialoptions.go then it's as if your makefile was written to be:
target: google.golang.org/grpc#v1.29.0/dialoptions.go
and if the time last modified of target is older than google.golang.org/grpc#v1.29.0/dialoptions.go (use ls -l to see these values) then make decides target is out of date and must be rebuilt. If it's newer, then make decides target is up to date and doesn't have to be rebuilt.
If you change the contents of foo.txt to be google.golang.org/grpc#v1.28.0/dialoptions.go then it compares the time last modified between target and google.golang.org/grpc#v1.28.0/dialoptions.go, the same way.
That's all make does, at its root.
I don't really understand your requirements so I can't give any suggestions for solutions. If you need to have target rebuilt if either the contents of foo.txt changes or the modification time on the prerequisite listed in foo.txt changes, then the only way to do that is list foo.txt as a prerequisite so make can detect when it was changed. You say you can't do that so... I don't have any good solutions for you.
I have a situation where I need to execute a rule in make depending on a remote file. This is an example of what I'm trying to do (the actual Makefile is a lot more complicated in ways that aren't relevant here):
URL = http://cdn.sstatic.net/stackoverflow/img/favicon.ico
stackoverflow.png: favico.ico modified | check_modified
convert favicon.ico $#
check_modified: modified
#echo Icon was modified. Downloading.
rm -f favicon.ico
wget $(URL)
.PHONY: check_modified
favico.ico: check_modified
modified:
touch -d "`wget --spider -S $(URL) 2>&1 | sed -n 's/.*Modified: //p'`" $#
The idea is:
The rule to build modified should always be run. The function of that rule is to change the modification time of the file to be the same as the modification time of what the URL points to.
After that, I want modified to behave normally in terms of dependencies. If modified is more recent than favicon.ico, I want to retrieve the new file and then let the dependency on it cause the target file to be remade.
As a wrinkle, in some applications, I have to retrieve the file manually. In those cases, I want to just have a rule that fires and tells me to manually download the file, but that doesn't otherwise affect building the target. As an example, if the source image that I'm converting were behind a site login, I'd need to manually login to the site and download it to a fixed location and then rerun make.
Everything I've tried either:
Fails to check the URL if the target is up to date, or
Always checks the URL and rebuilds the target, even if modified is not more recent than the target.
Any words of wisdom?
Your problem is probably that make builds the dependency graph before running any recipe. So, when your modified tag file is created / updated, it is too late for make to consider its last modification date and use it to decide whether other targets shall be built or not.
Your first observed behaviour (fails to check the URL if the target is up to date) probably corresponds to attempts where you did not use the .PHONY special target. As soon as modified has been created, make always considers it as up-to-date because it has no prerequisites and thus cannot be outdated.
The second behaviour (always checks the URL and rebuilds the target, even if modified is not more recent than the target) corresponds to what you posted: check_modified is a prerequisite of the .PHONY special target, which forces make to always consider it as outdated. stackoverflow.png and favico.ico, directly or not, depend on check_modified and are thus also always considered as outdated.
One option to solve your problem would be to use a recursive, two-passes, make invocation. On a first run make would build the all phony target (always), update a modified tag file, and then invoke itself again to build other targets that have the tag file as prerequisite and are outdated. As, for the second invocation, make rebuilds its dependency graph, it will take the last modification date of the tag file into account. Something like:
URL = http://cdn.sstatic.net/stackoverflow/img/favicon.ico
.PHONY: all
all:
touch -d "`wget --spider -S $(URL) 2>&1 | sed -n 's/.*Modified: //p'`" modified
$(MAKE) stackoverflow.ico
stackoverflow.ico: modified
#echo Icon was modified. Downloading.; \
rm -f favicon.ico; \
wget $(URL); \
convert favicon.ico $#
Explanations:
I replaced your conversion to png by a useless conversion to ico because the favicon.ico icon of SO is composite and its conversion to png creates two files instead of one, named stackoverflow-0.png and stackoverflow-1.png, which uselessly complicates things.
all is a true phony target and the default goal so, each time you invoke make (or make all), it is built. It first updates the modified tag file and then invokes make again to build stackoverflow.ico.
The second make invocation does nothing if stackoverflow.ico is more recent than modified, else it downloads and converts.
Your second question (get a message about required manual operations) is completely different and simpler to solve. Let's first define a message and echo it in the file's recipe:
define DIY_message
Dear user, you should first:
- do this
- and that.
Unless you know it is useless, of course.
endef
export DIY_message
the_remote_file:
#echo "$$DIY_message"
The message will be printed if make is invoked with this goal (make the_remote_file) or with a goal that somehow depends on the_remote_file and either:
the_remote_file does not exist,
or the_remote_file exists but is out-of-date with respect to its prerequisites (if you declared prerequisites for it),
or the_remote_file is a prerequisite of .PHONY.
Note: using an intermediate make variable assigned by a define-endef makes things easier for formatted multi-lines messages.
I have a somewhat complicated Makefile which runs perl scripts and other tools and generates some 1000 files. I would like to edit/modify some of those generated files after all files are generated. So I thought I can simply add a new rule to do so like this:
(phony new rule): $LIST_OF_FILES_TO_EDIT
file_modifier ...
however, the point here is some of those generated files which I'd like to edit ($LIST_OF_FILES_TO_EDIT) are used in the same make process to generate a long list of files. So I have to wait to make sure those files are no longer needed in the make process before I can go ahead and edit them. But I don't know how to do that. Not to mention that it is really hard to find out what files are generated by the help of $LIST_OF_FILES_TO_EDIT.
If it was possible to mention in the Makefile that this rule should be only run as the last rule, then my problem would be solved. but as far as I know this is not possible. So anyone has an idea?
Some points:
List of files to edit ($LIST_OF_FILES_TO_EDIT) is determined dynamically (not known before make process)
I am not sure I have picked a good title for this question. :)
1) If you're going to modify the files like that, it might behoove you to give the targets different names, like foo_unmodified and foo_modified, so that the Make's dependency handling will take care of this.
2) If your phony new rule is the one you invoke on the command line ("make phonyNewRule"), then Make will build whatever else it's going to build before executing the file_modifier command. If you want to build targets not on that list, you could do it this way:
(phony new rule): $(LIST_OF_FILES_TO_EDIT) $(OTHER_TARGETS)
file_modifier ...
3) If your dependencies are set up correctly, you can find out which targets depend on $(LIST_OF_FILES_TO_EDIT), but it's not very tidy. You could just touch one of the files, run make, see which targets it built, repeat for all files. You could save a little time by using Make arguments: "make -n -W foo1 -W foo2 -W foo3 ... -W foo99 all". This will print the commands Make would run-- I don't know of any way to get it to tell you which targets it would rebuild.
I am trying to create a subdirectory in my project (let's call it $PROJECT/child) that needs to pull in a Makefile (let's call it ../Makefile.inc) from its parent, $PROJECT/Makefile.inc. Later I want to copy $PROJECT/child somewhere else so it can run independently of $PROJECT.
There is a common Makefile that needs to be included in both projects and shipped when the subdirectory is copied, and I want it to be included in both cases. So I thought I would link it in during the child build, if it isn't found. (I don't want to just include ../Makefile.inc, because this will disappear when I copy the project, and I don't want the calling build system to be responsible for putting the Makefile.inc in place.)
With those constraints, here's a horrible hack that I've come up with to do this, within $PROJECT/child/Makefile:
HACK = $(shell test -f Makefile.inc || ln -f ../Makefile.inc .)
include $(HACK)Makefile.inc
Notice the extra special duct tape on that second command. I have to actually include $(HACK) even though it's going to end up empty, so that the $(shell ...) will be evaluated. ;-)
Is there a cleaner way to make this happen?
Give a rule to build Makefile.inc. (make will complain that Makefile.inc doesn't exist when it parses the include line, but it will go on parsing the main makefile, apply any rule to build included files, and go back and re-parse the main makefile with the included files.)
include Makefile.inc
Makefile.inc:
ln ../Makefile.inc $#
Part of my Makefile:
CPUDEPS=./mydeps.cpu
(...)
deps: $(CPUDEPS)
$(CPUDEPS): $(CCFILES)
#echo [DEPS] CPU
$(CMDECHO)makedepend -Y -s'# CPU sources dependencies generated with "make deps"' \
-w4096 -f- -- $(CFLAGS) -- $^ 2> /dev/null > $(CPUDEPS)
(...)
sinclude $(CPUDEPS)
Problem 1: includes are done during the first phase of processing, targets during the second phase; so, if ./mydeps.cpu doesn't exist and I "make deps", I get first the error
Makefile:335: ./mydeps.cpu: No such file or directory
I hide the error using sinclude instead of include, but the problem is still there: the old file is included, not the just-generated-one. Have to run it twice to include the updated file. This is because make does a two-phase processing; is there any way to tell make to complete the target deps before parsing the includes?
Problem 2: even if the file ./mydeps.cpu doesn't exist and make deps actually creates it, I always get a "make: Nothing to do for deps". This doesn't happen with other targets. I don't understand why and how to avoid it.
Problem 1 is non-existant: before building a target, make automatically rebuilds makefiles (with implicit rules if no explicit rule is provided). So having a rule for the makefile ensures that will always be up to date, there is no need to run deps twice. Additionally, since CPUDEPS is a makefile, it will be updated automatically before any other rule is run, so dependencies will always be updated if necessary and make deps is not needed. You can probably notice this by yourself by observing the [DEPS] line being echoed if any of the CCFILES becomes more recent that the dependency file.
For Problem 2, adding anything to the recipe ensures that make doesn't complain about having nothing to do. If there is nothing else, you can use something like #echo OK to give feedback to the user, or a simple #true if you prefer totally silent makes.
What you are trying to achieve is useless: you can use the dependencies file that was created during the previous build. That's enough.
The main reasoning behind that rule is:
if you haven't changed any of your files, then the dependencies file is up-to-date, and there's nothing to build.
if you have changed anything, even very deep into your #include chain, on an existing file that were used by previous build, then the dependencies file have already caught it. You'll rebuild what is needed.
if you change something in a new file (you add that file!) then it was not used by previous build, and not listed in dependencies. But if you really want to use it, then you have to modify at least one of your other files that was used before, and you're back on the previous case.
The solution is to create the dependencies file during the normal process of the compilation, and to optionally include it (with sinclude) if it is present.