I've got a dependency between a few files, where one is generated from the other. For example bar.txt is needed to generate foo-temp.xml is needed to generate the final foo.xml file. In my Makefile this is expressed as follows:
foo-temp.xml: bar.txt
do-magic -o foo-temp.xml bar.txt
foo.xml: foo-temp.xml
do-more-magic -o foo.xml foo-temp.xml
As long as all files exist, everything works smoothly. But there are instances where I provide foo-temp.xml manually instead of generating it from bar.txt; in fact, bar.txt doesn't even exist in that case.
How can I express this "non-existence" in my Makefile without having it fail because bar.txt does not exist?
I'm not sure this is the best way but it should work (assuming bar.txt itself never needs to be created via another target).
foo-temp.xml: $(wildcard bar.txt)
The wildcard function returns the list of files that match the patterns and removes those that don't. This also happens to work for filenames with no wildcard/globbing characters in them.
Alternatively (and a bit more verbosely)
foo-temp.xml: $(and $(realpath bar.txt),bar.txt)
Same basic idea but realpath returns a "canonical absolute name" so you can't use its result directly (unless you want an absolute path there) so instead we use the fact that the and function short-circuits to only generate output when realpath returns a non-empty value.
If bar.txt needs to be generated itself than neither of those solutions will work however as it will prevent bar.txt from being a prerequisite unless it already exists (though once it does the normal make process will work).
If this is the case then you could try using make -o foo-temp.xml foo.xml to tell make that foo-temp.xml is infinitely old which should short-circuit make attempting to remake it (and process its prerequisites) and should let the foo.xml: foo-temp.xml rule work (assuming foo.xml doesn't exist or is older than the foo-temp.xml you created manually).
Related
I have a Makefile target that depends on a dynamic list of source files. This list is parsed on make invocations so it could potentially change. It looks like this:
target: $(shell cat foo.txt)
#echo "build target"
foo.txt changes from time to time. What I'm seeing is that target will be rebuilt the first time foo.txt changes content to X. Say that foo.txt changes content to Y then changes back to X then target won't be built. I know I can include foo.txt as a dependency itself but for reasons difficult to explain that's not an option.
As a more concrete example, foo.txt looks like this:
google.golang.org/grpc#v1.29.0/dialoptions.go
but could change to something like:
google.golang.org/grpc#v1.28.0/dialoptions.go
What's going on? Is Makefile caching something?
EDIT
I guess Makefile is caching the timestamp of the dependencies and thus if the dependency files didn't change at all then the target won't be rebuilt? Is there a way to fix this?
Make caches nothing. It has no preserved data whatsoever about any previous build. Make uses the filesystem as its "cache".
Because you're using $(shell cat foo.txt), make knows absolutely nothing about the file foo.txt. All it knows about is the set of files inside the foo.txt files. It treats all of those as prerequisites.
Then it merely compares the time last modified of the target file (target) with the time last modified of the prerequisite file that is contained in foo.txt.
So if foo.txt contains google.golang.org/grpc#v1.29.0/dialoptions.go then it's as if your makefile was written to be:
target: google.golang.org/grpc#v1.29.0/dialoptions.go
and if the time last modified of target is older than google.golang.org/grpc#v1.29.0/dialoptions.go (use ls -l to see these values) then make decides target is out of date and must be rebuilt. If it's newer, then make decides target is up to date and doesn't have to be rebuilt.
If you change the contents of foo.txt to be google.golang.org/grpc#v1.28.0/dialoptions.go then it compares the time last modified between target and google.golang.org/grpc#v1.28.0/dialoptions.go, the same way.
That's all make does, at its root.
I don't really understand your requirements so I can't give any suggestions for solutions. If you need to have target rebuilt if either the contents of foo.txt changes or the modification time on the prerequisite listed in foo.txt changes, then the only way to do that is list foo.txt as a prerequisite so make can detect when it was changed. You say you can't do that so... I don't have any good solutions for you.
Consider the following (MCVE of a) Makefile:
my_target: prepare test.bin
prepare:
echo >test.dat
%.bin: %.dat
cp $? $#
If you run make in a clean directory, it fails:
echo >test.dat
make: *** No rule to make target 'test.bin', needed by 'my_target'. Stop.
Run it again and it succeeds:
echo >test.dat
cp test.dat test.bin
What seems to happen is that the rule to make *.bin from *.dat only recognises that it knows how to make test.bin if test.dat exists before anything is executed, even though according to the output it has already created test.dat before it tries to create test.bin.
This is inconvenient for me as I have to prepare a few files first (import them from a different, earlier part of the build).
Is there a solution? Perhaps some way to allow the rules to be (re)evaluated in the light of the files which are now present?
There are a number of issues with your makefile. However based on your comments I'm inclined to assume that the MCVE here is just a little too "M" and it's been reduced so much that it has a number of basic problems. So I won't discuss them, unless you want me to.
The issue here is that you're creating important files without indicating to make that that's what you're doing. Make keeps internally a cache of the contents of directories that it's worked with, for performance reasons, and that cache is only updated when make invokes a rule that it understands will modify it.
Here your target is prepare but the recipe actually creates a completely different file, test.dat. So, make doesn't modify its internal cache of the directory contents and when it checks the cache to see if the file test.dat exists, it doesn't.
You need to be sure that your makefile is written such that it doesn't trick make: if a recipe creates a file foo then the target name should be foo, not bar.
This happens for wildcard targets, like %.bin. They get evaluated at the first pass. You could add an explicit target of test.bin. Or, follow the advice of tkausl and have test.dat depend on prepare (a phony target). In this case, you don't need the double dependency anymore:
my_target: test.bin
you have to write
test.dat: prepare
or (when when you want to stay with wildcards)
%.dat: prepare
#:
Usually, you might want to create and use .stamp files instead of a prepare target.
I am learning makefiles, and can't just wrap my head around this problem i am having, and would like to understand how/why this fail.
I have half a dozen erlang files in a src directory. I want to compile these into a ebin directory, without having to define a rule for each and every one of them. According to the Gnu make documentation, pattern rules should be right up my alley.
However, with the following makefile, all I get from make is make: *** No targets. Stop. Why is that?
ebin/%.beam: src/%.erl
mkdir -p ebin
erlc -o ebin $<
Edit: Based on this answer, I now understand that i would have to explicitly declare the targets, for instance by using make ebin/cmplx.beam. However, i still do not understand how i should write my makefile to get my desired behaviour - since I have half a dozen targets (and in other projects even more), this seems like an unnecessary hassle. Is there not a way to define targets based on the source file names?
The target rule tells make that whenever it needs to produce a beam file in the ebin directory, and there exists a corresponding erl file in the src directory, it can use erlc.
However, this doesn't tell make that this is what it needs to do. You could explicitly tell make what it needs to do by giving it a target on the command line:
make ebin/foo.beam
If you don't give a target on the command line, make will pick the first non-pattern rule in the makefile as its target. However, your makefile doesn't have any non-pattern rules, so there is no target.
What you probably want is that for each existing erl file in src, make should consider the corresponding beam file in ebin to be a target. You can achieve that by calling wildcard and patsubst:
erl_files=$(wildcard src/*.erl)
beam_files=$(patsubst src/%.erl,ebin/%.beam,$(erl_files))
ebin/%.beam: src/%.erl
mkdir -p ebin
erlc -o ebin $<
all: $(beam_files)
(The indented lines need to be actual physical tabs, not spaces.)
That way, running make will rebuild all beam files that are out of date. all gets chosen as the default target, and it in turn depends on all beam existing or potential, each of which in turn depends on the corresponding erl file.
This trick is described in the GNU make manual.
I have a situation where I need to execute a rule in make depending on a remote file. This is an example of what I'm trying to do (the actual Makefile is a lot more complicated in ways that aren't relevant here):
URL = http://cdn.sstatic.net/stackoverflow/img/favicon.ico
stackoverflow.png: favico.ico modified | check_modified
convert favicon.ico $#
check_modified: modified
#echo Icon was modified. Downloading.
rm -f favicon.ico
wget $(URL)
.PHONY: check_modified
favico.ico: check_modified
modified:
touch -d "`wget --spider -S $(URL) 2>&1 | sed -n 's/.*Modified: //p'`" $#
The idea is:
The rule to build modified should always be run. The function of that rule is to change the modification time of the file to be the same as the modification time of what the URL points to.
After that, I want modified to behave normally in terms of dependencies. If modified is more recent than favicon.ico, I want to retrieve the new file and then let the dependency on it cause the target file to be remade.
As a wrinkle, in some applications, I have to retrieve the file manually. In those cases, I want to just have a rule that fires and tells me to manually download the file, but that doesn't otherwise affect building the target. As an example, if the source image that I'm converting were behind a site login, I'd need to manually login to the site and download it to a fixed location and then rerun make.
Everything I've tried either:
Fails to check the URL if the target is up to date, or
Always checks the URL and rebuilds the target, even if modified is not more recent than the target.
Any words of wisdom?
Your problem is probably that make builds the dependency graph before running any recipe. So, when your modified tag file is created / updated, it is too late for make to consider its last modification date and use it to decide whether other targets shall be built or not.
Your first observed behaviour (fails to check the URL if the target is up to date) probably corresponds to attempts where you did not use the .PHONY special target. As soon as modified has been created, make always considers it as up-to-date because it has no prerequisites and thus cannot be outdated.
The second behaviour (always checks the URL and rebuilds the target, even if modified is not more recent than the target) corresponds to what you posted: check_modified is a prerequisite of the .PHONY special target, which forces make to always consider it as outdated. stackoverflow.png and favico.ico, directly or not, depend on check_modified and are thus also always considered as outdated.
One option to solve your problem would be to use a recursive, two-passes, make invocation. On a first run make would build the all phony target (always), update a modified tag file, and then invoke itself again to build other targets that have the tag file as prerequisite and are outdated. As, for the second invocation, make rebuilds its dependency graph, it will take the last modification date of the tag file into account. Something like:
URL = http://cdn.sstatic.net/stackoverflow/img/favicon.ico
.PHONY: all
all:
touch -d "`wget --spider -S $(URL) 2>&1 | sed -n 's/.*Modified: //p'`" modified
$(MAKE) stackoverflow.ico
stackoverflow.ico: modified
#echo Icon was modified. Downloading.; \
rm -f favicon.ico; \
wget $(URL); \
convert favicon.ico $#
Explanations:
I replaced your conversion to png by a useless conversion to ico because the favicon.ico icon of SO is composite and its conversion to png creates two files instead of one, named stackoverflow-0.png and stackoverflow-1.png, which uselessly complicates things.
all is a true phony target and the default goal so, each time you invoke make (or make all), it is built. It first updates the modified tag file and then invokes make again to build stackoverflow.ico.
The second make invocation does nothing if stackoverflow.ico is more recent than modified, else it downloads and converts.
Your second question (get a message about required manual operations) is completely different and simpler to solve. Let's first define a message and echo it in the file's recipe:
define DIY_message
Dear user, you should first:
- do this
- and that.
Unless you know it is useless, of course.
endef
export DIY_message
the_remote_file:
#echo "$$DIY_message"
The message will be printed if make is invoked with this goal (make the_remote_file) or with a goal that somehow depends on the_remote_file and either:
the_remote_file does not exist,
or the_remote_file exists but is out-of-date with respect to its prerequisites (if you declared prerequisites for it),
or the_remote_file is a prerequisite of .PHONY.
Note: using an intermediate make variable assigned by a define-endef makes things easier for formatted multi-lines messages.
Consider, if you will, the following situation:
$ pwd
/tmp/submake_example
$ head $(find -type f)
==> ./subdir/Makefile <==
subtarget:
echo "executing subtarget from directory $$(pwd)"
==> ./Makefile <==
include subdir/Makefile
$ make subtarget
echo "executing subtarget from directory $(pwd)"
executing subtarget from directory /tmp/submake_example
Is there a way that I can have the command for the target subtarget executed from within the directory subdir? I'd like to be able to access the targets of included makefiles from the command line in the main directory.
After some digging and experimentation, I've discovered a couple of things.
The approach recommended by this reference, which refers to the paper Recursive Make Considered Harmful, is pretty complicated, and involves manually maintaining a directory stack throughout the makefile inclusion heirarchy.
I also discovered, independently of any reference, that the last element of the variable $(MAKEFILE_LIST) will, at least for some basic test cases, contain the name of the makefile that's currently being included. This means that it's possible to store its directory in a simply expanded, target-specific variable with code like
foo: dir := $(dir $(lastword $(MAKEFILE_LIST)))
That's it. Then you can just use $(dir) within that rule like any other variable; it retains the value that it assumed during makefile inclusion. cd to it, put it in command paths, whatever.
You have to be careful to put the assignment to dir before any include directives in that makefile.
This won't work if there's a space in the directory name, but I gather that pretty much nothing in make works if there are spaces in file or directory names.
It's also notable that this approach uses features only found in GNU make. Sounds like a killer app to me, though I remain a bit perplexed as to why they don't just provide the directory name in a builtin variable or function.
Anyway this seems to work, at least at first glance. I'm still pretty much just floating this one without any real backup, so comments on the practicality of this approach, and alternative suggestions, are very welcome.