I have a Makefile target that depends on a dynamic list of source files. This list is parsed on make invocations so it could potentially change. It looks like this:
target: $(shell cat foo.txt)
#echo "build target"
foo.txt changes from time to time. What I'm seeing is that target will be rebuilt the first time foo.txt changes content to X. Say that foo.txt changes content to Y then changes back to X then target won't be built. I know I can include foo.txt as a dependency itself but for reasons difficult to explain that's not an option.
As a more concrete example, foo.txt looks like this:
google.golang.org/grpc#v1.29.0/dialoptions.go
but could change to something like:
google.golang.org/grpc#v1.28.0/dialoptions.go
What's going on? Is Makefile caching something?
EDIT
I guess Makefile is caching the timestamp of the dependencies and thus if the dependency files didn't change at all then the target won't be rebuilt? Is there a way to fix this?
Make caches nothing. It has no preserved data whatsoever about any previous build. Make uses the filesystem as its "cache".
Because you're using $(shell cat foo.txt), make knows absolutely nothing about the file foo.txt. All it knows about is the set of files inside the foo.txt files. It treats all of those as prerequisites.
Then it merely compares the time last modified of the target file (target) with the time last modified of the prerequisite file that is contained in foo.txt.
So if foo.txt contains google.golang.org/grpc#v1.29.0/dialoptions.go then it's as if your makefile was written to be:
target: google.golang.org/grpc#v1.29.0/dialoptions.go
and if the time last modified of target is older than google.golang.org/grpc#v1.29.0/dialoptions.go (use ls -l to see these values) then make decides target is out of date and must be rebuilt. If it's newer, then make decides target is up to date and doesn't have to be rebuilt.
If you change the contents of foo.txt to be google.golang.org/grpc#v1.28.0/dialoptions.go then it compares the time last modified between target and google.golang.org/grpc#v1.28.0/dialoptions.go, the same way.
That's all make does, at its root.
I don't really understand your requirements so I can't give any suggestions for solutions. If you need to have target rebuilt if either the contents of foo.txt changes or the modification time on the prerequisite listed in foo.txt changes, then the only way to do that is list foo.txt as a prerequisite so make can detect when it was changed. You say you can't do that so... I don't have any good solutions for you.
Related
Consider the following (MCVE of a) Makefile:
my_target: prepare test.bin
prepare:
echo >test.dat
%.bin: %.dat
cp $? $#
If you run make in a clean directory, it fails:
echo >test.dat
make: *** No rule to make target 'test.bin', needed by 'my_target'. Stop.
Run it again and it succeeds:
echo >test.dat
cp test.dat test.bin
What seems to happen is that the rule to make *.bin from *.dat only recognises that it knows how to make test.bin if test.dat exists before anything is executed, even though according to the output it has already created test.dat before it tries to create test.bin.
This is inconvenient for me as I have to prepare a few files first (import them from a different, earlier part of the build).
Is there a solution? Perhaps some way to allow the rules to be (re)evaluated in the light of the files which are now present?
There are a number of issues with your makefile. However based on your comments I'm inclined to assume that the MCVE here is just a little too "M" and it's been reduced so much that it has a number of basic problems. So I won't discuss them, unless you want me to.
The issue here is that you're creating important files without indicating to make that that's what you're doing. Make keeps internally a cache of the contents of directories that it's worked with, for performance reasons, and that cache is only updated when make invokes a rule that it understands will modify it.
Here your target is prepare but the recipe actually creates a completely different file, test.dat. So, make doesn't modify its internal cache of the directory contents and when it checks the cache to see if the file test.dat exists, it doesn't.
You need to be sure that your makefile is written such that it doesn't trick make: if a recipe creates a file foo then the target name should be foo, not bar.
This happens for wildcard targets, like %.bin. They get evaluated at the first pass. You could add an explicit target of test.bin. Or, follow the advice of tkausl and have test.dat depend on prepare (a phony target). In this case, you don't need the double dependency anymore:
my_target: test.bin
you have to write
test.dat: prepare
or (when when you want to stay with wildcards)
%.dat: prepare
#:
Usually, you might want to create and use .stamp files instead of a prepare target.
I have a situation where I need to execute a rule in make depending on a remote file. This is an example of what I'm trying to do (the actual Makefile is a lot more complicated in ways that aren't relevant here):
URL = http://cdn.sstatic.net/stackoverflow/img/favicon.ico
stackoverflow.png: favico.ico modified | check_modified
convert favicon.ico $#
check_modified: modified
#echo Icon was modified. Downloading.
rm -f favicon.ico
wget $(URL)
.PHONY: check_modified
favico.ico: check_modified
modified:
touch -d "`wget --spider -S $(URL) 2>&1 | sed -n 's/.*Modified: //p'`" $#
The idea is:
The rule to build modified should always be run. The function of that rule is to change the modification time of the file to be the same as the modification time of what the URL points to.
After that, I want modified to behave normally in terms of dependencies. If modified is more recent than favicon.ico, I want to retrieve the new file and then let the dependency on it cause the target file to be remade.
As a wrinkle, in some applications, I have to retrieve the file manually. In those cases, I want to just have a rule that fires and tells me to manually download the file, but that doesn't otherwise affect building the target. As an example, if the source image that I'm converting were behind a site login, I'd need to manually login to the site and download it to a fixed location and then rerun make.
Everything I've tried either:
Fails to check the URL if the target is up to date, or
Always checks the URL and rebuilds the target, even if modified is not more recent than the target.
Any words of wisdom?
Your problem is probably that make builds the dependency graph before running any recipe. So, when your modified tag file is created / updated, it is too late for make to consider its last modification date and use it to decide whether other targets shall be built or not.
Your first observed behaviour (fails to check the URL if the target is up to date) probably corresponds to attempts where you did not use the .PHONY special target. As soon as modified has been created, make always considers it as up-to-date because it has no prerequisites and thus cannot be outdated.
The second behaviour (always checks the URL and rebuilds the target, even if modified is not more recent than the target) corresponds to what you posted: check_modified is a prerequisite of the .PHONY special target, which forces make to always consider it as outdated. stackoverflow.png and favico.ico, directly or not, depend on check_modified and are thus also always considered as outdated.
One option to solve your problem would be to use a recursive, two-passes, make invocation. On a first run make would build the all phony target (always), update a modified tag file, and then invoke itself again to build other targets that have the tag file as prerequisite and are outdated. As, for the second invocation, make rebuilds its dependency graph, it will take the last modification date of the tag file into account. Something like:
URL = http://cdn.sstatic.net/stackoverflow/img/favicon.ico
.PHONY: all
all:
touch -d "`wget --spider -S $(URL) 2>&1 | sed -n 's/.*Modified: //p'`" modified
$(MAKE) stackoverflow.ico
stackoverflow.ico: modified
#echo Icon was modified. Downloading.; \
rm -f favicon.ico; \
wget $(URL); \
convert favicon.ico $#
Explanations:
I replaced your conversion to png by a useless conversion to ico because the favicon.ico icon of SO is composite and its conversion to png creates two files instead of one, named stackoverflow-0.png and stackoverflow-1.png, which uselessly complicates things.
all is a true phony target and the default goal so, each time you invoke make (or make all), it is built. It first updates the modified tag file and then invokes make again to build stackoverflow.ico.
The second make invocation does nothing if stackoverflow.ico is more recent than modified, else it downloads and converts.
Your second question (get a message about required manual operations) is completely different and simpler to solve. Let's first define a message and echo it in the file's recipe:
define DIY_message
Dear user, you should first:
- do this
- and that.
Unless you know it is useless, of course.
endef
export DIY_message
the_remote_file:
#echo "$$DIY_message"
The message will be printed if make is invoked with this goal (make the_remote_file) or with a goal that somehow depends on the_remote_file and either:
the_remote_file does not exist,
or the_remote_file exists but is out-of-date with respect to its prerequisites (if you declared prerequisites for it),
or the_remote_file is a prerequisite of .PHONY.
Note: using an intermediate make variable assigned by a define-endef makes things easier for formatted multi-lines messages.
Here is the context :
I am working on a makefile to create a .h at every build, including another makefile that will use this header. I can't edit the second one.
Using a target all depending on my file, it compiles the first time, creating the missing MyHeader.h. The problem is, when I recompile, the header is not regenerated...
My makefile looks like this :
all: myHeader.h
myHeader.h:
scriptToBuildMyHeader.sh
include obscureAndPrivateMakefile.make
I also tried with a .phony target at the beginning. Same result : once created, it won't be regenerated at every build.
PS : I can't call a script before make.
Do some makfile-Masters have any ideas how to deal with that ?
Thanks!
Because myHeader.h has no dependencies, it will never be rebuilt once it exists. You can work around this by creating a dependency from myHeader.h to a phony target, eg:
forcebuild:
# dummy; do nothing and don't create this file
.PHONY: forcebuild
myHeader.h: forcebuild
scriptToBuildMyHeader.sh
This will however slow down your build considerably, as the header (and any source files including it) will need to be rebuilt every time.
The trouble is that because myHeader.h does not depend on anything, it exists and is therefore up to date on the second build. To make sure it is built each time, it has to depend on a non-existent file:
myHeader.h: .FORCE
scriptToBuildMyHeader.sh
.FORCE:
The name '.FORCE' (or, sometimes, FORCE) is used classically.
If you use GNU Make, you could make the 'non-existent' file into a phony target:
.PHONY: .FORCE
The advantage of this is that (GNU) make does not create the file .FORCE even if you run make -t - which would break the automatic rebuild of the header because that rule depends on there not being a file .FORCE that actually exists.
Here's another possible approach:
all: clean foo.txt
clean:
rm foo.txt
foo.txt:
echo > foo.txt
where I'm using echo > foo.txt to simulate creation of your header.
In the event that a Makefile itself is changed, a safe bet would be to consider all targets out of date.
Is there a clever way to add this dependency? Are there any alternatives?
Make sure the object files depend on the makefile:
$(OBJFILES) : Makefile
Where Makefile is the name of the make file.
A safe bet, but a terrible idea. Example: you're using automake and update Makefile.am to add a single source file. The correct response is to compile just the new file and link it in. In your scheme everything would be rebuilt.
Moreover, adding the dependency isn't going to do anything unless you touch the file, something like:
$(SRCS): Makefile
touch $#
This will then trip up editors that use the mtime to detect concurrent modification (emacs is one example).
If you're doing something major, just run make clean all after doing the change.
Since GNU make version 4.3 it is now possible with the use of those two special variable:
.EXTRA_PREREQS
To add new prerequisite to every target
MAKEFILE_LIST
To get the path of the make file
To have every target depend on the current make file:
Put near the top of the file (before any include since it would affect the MAKEFILE_LIST) the following line:
.EXTRA_PREREQS:= $(abspath $(lastword $(MAKEFILE_LIST)))
To have every target depend on the current make file and also the make files which were included
Put the following line at the end of your file:
.EXTRA_PREREQS+=$(foreach mk, ${MAKEFILE_LIST},$(abspath ${mk}))
I'm using makefiles to convert an internal file format to an XML file which is sent to other colleagues. They would make changes to the XML file and send it back to us (Don't ask, this needs to be this way ;)). I'd like to use my makefile to update the internal files when this XML changes.
So I have these rules:
%.internal: $(DATAFILES)
# Read changes from XML if any
# Create internal representation here
%.xml: %.internal
# Convert to XML here
Now the XML could change because of the workflow described above. But since no data files have changed, make would tell me that file.internal is up-to-date. I would like to avoid making %.internal target phony and a circular dependency on %.xml obviously doesn't work.
Any other way I could force make to check for changes in the XML file and re-build %.internal?
You want to allow two different actions: making the xml file from the internal file, and making the internal file from the xml file. Since Make knows only the modification times, it knows which target is older but not whether it should be remade. So put in another file as a flag to record when either action was last taken, and make that your primary target; if either target is newer than the flag, it has been modified by something other than these actions, and make should rebuild the older target (and then touch the flag).
There are several ways to implement this. In some versions of Make (such as recent versions of GNUMake) you can write double-colon rules, so that Make will rebuild a target differently, based on which preq triggered it:
%.flag:: %.internal
# convert $*.internal to $*.xml
touch $#
%.flag:: %.xml
# rewrite $*.internal based on $*.xml
touch $#
A less elegant but more portable way is to look at $? and rebuild the other file:
%.flag: %.xml %.internal
ifeq ($?,$*.internal)
# convert $*.internal to $*.xml
else
# rewrite $*.internal based on $*.xml
endif
touch $#
I think you could do something like this:
all: .last-converted-xml .last-converted-internal
.last-converted-internal: *.internal
./internal-2-xml $?
touch $# .last-converted-xml
.last-converted-xml: *.xml
./xml-2-internal $?
touch $# .last-converted-internal
This runs "xml-convert" on any .xml files newer than an arbitrary marker file, ".last-converted". The $? should give you a list of all dependencies (*.xml) that are newer than the marker file.
Of course, the xml-convert program will have to be written to take a list of xml files and process each one.
I'm not sure from the question whether you actually need the .internal file, or if that was just an attempt to get the makefile working. So, either your "xml-convert" program can convert each .xml file in place, or it can also generate file.internal as well if you need it.
Use the -W option of make to have make think one of the data files has changed:
make -W somedatafile
This will cause make to think somedatafile has been modified without actually changing it's modification time.
Would it be possible to use different names for the XML file? The file you create from the internal format would have one name and the file your colleagues send you another? If they used different names there would be no circular dependency.