Consider the following make workflow, in which I (a) want to be sure to always work on the newest download.zip file, but (b) only run the rest of the workflow when this file changes:
my_output.file: download.zip
some_expensive_operation
download.zip:
wget -N http://some.server/download.zip
In this example, wget has got dependency information that make does not have.
If download.zip is present, the rule will never be executed and wget will never check if there is a newer file available.
I can of course make .PHONY: download.zip to have the rule executed every time, by then some_expensive_operation will be executed as well no matter if the file changed or not.
Is there any way to tell make to run the my_output.file rule above only if download.zip actually changed?
Use the force rule trick:
my_output.file: download.zip
some_expensive_operation
download.zip: FORCE
wget -N http://some.server/download.zip
FORCE: ;
(or you could declare .PHONY: FORCE if you prefer). This ensures that the recipe for download.zip is always run, but it's not marked phony itself so targets that depend on it won't be rebuilt unless it's changed.
Related
Consider the following (MCVE of a) Makefile:
my_target: prepare test.bin
prepare:
echo >test.dat
%.bin: %.dat
cp $? $#
If you run make in a clean directory, it fails:
echo >test.dat
make: *** No rule to make target 'test.bin', needed by 'my_target'. Stop.
Run it again and it succeeds:
echo >test.dat
cp test.dat test.bin
What seems to happen is that the rule to make *.bin from *.dat only recognises that it knows how to make test.bin if test.dat exists before anything is executed, even though according to the output it has already created test.dat before it tries to create test.bin.
This is inconvenient for me as I have to prepare a few files first (import them from a different, earlier part of the build).
Is there a solution? Perhaps some way to allow the rules to be (re)evaluated in the light of the files which are now present?
There are a number of issues with your makefile. However based on your comments I'm inclined to assume that the MCVE here is just a little too "M" and it's been reduced so much that it has a number of basic problems. So I won't discuss them, unless you want me to.
The issue here is that you're creating important files without indicating to make that that's what you're doing. Make keeps internally a cache of the contents of directories that it's worked with, for performance reasons, and that cache is only updated when make invokes a rule that it understands will modify it.
Here your target is prepare but the recipe actually creates a completely different file, test.dat. So, make doesn't modify its internal cache of the directory contents and when it checks the cache to see if the file test.dat exists, it doesn't.
You need to be sure that your makefile is written such that it doesn't trick make: if a recipe creates a file foo then the target name should be foo, not bar.
This happens for wildcard targets, like %.bin. They get evaluated at the first pass. You could add an explicit target of test.bin. Or, follow the advice of tkausl and have test.dat depend on prepare (a phony target). In this case, you don't need the double dependency anymore:
my_target: test.bin
you have to write
test.dat: prepare
or (when when you want to stay with wildcards)
%.dat: prepare
#:
Usually, you might want to create and use .stamp files instead of a prepare target.
I know what a .PHONY does.
If in the folder where my Makefile is, I add an empty file called clean and after I run make clean all of the clean target will not be executed since there was not any change in the file, so the target will not run and this is correct.
If I add .PHONY: clean, than the clean is seen as a command and this is also correct.
My question is why this behavior does not happen the same to all target, since I added a all file in the folder.So basically the all target still executes like if it was a .PHONY: all
I have the fallowing makefile code.
all: test1 test2
test1: test1.o
test1.o: test1.c
test2: test2.o
test2.o: test2.c
clean:
rm -rf *.o test1 test2
How do you know that the all rule is "still executing"? That rule has no recipe, so there's no way it can be "executed".
If what you mean is that even though the all file exists in the local directory, make is still building the targets test1 and test2, that's how make works (this doesn't have anything to do with phony vs. non-phony targets). When make decides whether or not build a particular target first it tries to build all the prerequisites of that target, and all the prerequisites of those targets, etc. Only after all that is complete, can make know whether or not to build the first target (all in this case).
make clean here doesn't have any dependencies, so putting a file named clean there is enough for the target to be considered built.
make all on the other hand has dependencies. Even if you put a file named all there, Make has to check whether the all file is newer than test1 and test2. This process triggers builds of test1 and test2, and it happens to have the same effect as if all was a phony target.
The basis is that all: test1 test2 is a recipe for building a file named all, that depends on the files test1 and test2.
If you ran make all, Make would do something like this:
Analyse the Makefile.
Find out that all depends on test1 and test2.
Check the timestamp of all and see if it is "up to date".
It is "up to date" if none of the dependencies are newer than itself.
In other words, Make can skip building a file if it's newer than all it's dependencies.
Build outdated or missing files.
Now, if you would like to prevent Make from considering the targets as files, you could specify them as phony targets. That is best practice for non-file targets like all.
(This answer isn't disagreeing with either of the existing answers, but suggesting another way of thinking about this).
When you have a rule like
dst: src
action
you're saying two things (as you know):
if dst doesn't exist, or is older than src, then do action; and
when action completes, the file dst will exist.
With targets such as all or clean, the second statement is of course not true. Make doesn't hold you to the promise in (2), so when you say make all, it'll compute and generate the required dependencies, and not complain that there's no file all in place afterwards. You're lying to Make, but it doesn't mind (it's cool with that...). That is, this is basically a makefile hack.
Where this goes wrong, of course, is if for some reason there happens to be a file called all or clean. Then Make will take the modification date of the file all into account when calculating the dependencies, and possibly come to a conclusion you didn't expect.
So what .PHONY: all does is legitimise the hack, and tells Make ‘even if a file all exists, pretend that it doesn't’; you're basically cancelling promise (2).
Therefore, as the other answers state, mentioning .PHONY isn't necessary. It simply forestalls an error – easy to make but easy to miss – when a file matching a phony target is accidentally created.
I have a situation where I need to execute a rule in make depending on a remote file. This is an example of what I'm trying to do (the actual Makefile is a lot more complicated in ways that aren't relevant here):
URL = http://cdn.sstatic.net/stackoverflow/img/favicon.ico
stackoverflow.png: favico.ico modified | check_modified
convert favicon.ico $#
check_modified: modified
#echo Icon was modified. Downloading.
rm -f favicon.ico
wget $(URL)
.PHONY: check_modified
favico.ico: check_modified
modified:
touch -d "`wget --spider -S $(URL) 2>&1 | sed -n 's/.*Modified: //p'`" $#
The idea is:
The rule to build modified should always be run. The function of that rule is to change the modification time of the file to be the same as the modification time of what the URL points to.
After that, I want modified to behave normally in terms of dependencies. If modified is more recent than favicon.ico, I want to retrieve the new file and then let the dependency on it cause the target file to be remade.
As a wrinkle, in some applications, I have to retrieve the file manually. In those cases, I want to just have a rule that fires and tells me to manually download the file, but that doesn't otherwise affect building the target. As an example, if the source image that I'm converting were behind a site login, I'd need to manually login to the site and download it to a fixed location and then rerun make.
Everything I've tried either:
Fails to check the URL if the target is up to date, or
Always checks the URL and rebuilds the target, even if modified is not more recent than the target.
Any words of wisdom?
Your problem is probably that make builds the dependency graph before running any recipe. So, when your modified tag file is created / updated, it is too late for make to consider its last modification date and use it to decide whether other targets shall be built or not.
Your first observed behaviour (fails to check the URL if the target is up to date) probably corresponds to attempts where you did not use the .PHONY special target. As soon as modified has been created, make always considers it as up-to-date because it has no prerequisites and thus cannot be outdated.
The second behaviour (always checks the URL and rebuilds the target, even if modified is not more recent than the target) corresponds to what you posted: check_modified is a prerequisite of the .PHONY special target, which forces make to always consider it as outdated. stackoverflow.png and favico.ico, directly or not, depend on check_modified and are thus also always considered as outdated.
One option to solve your problem would be to use a recursive, two-passes, make invocation. On a first run make would build the all phony target (always), update a modified tag file, and then invoke itself again to build other targets that have the tag file as prerequisite and are outdated. As, for the second invocation, make rebuilds its dependency graph, it will take the last modification date of the tag file into account. Something like:
URL = http://cdn.sstatic.net/stackoverflow/img/favicon.ico
.PHONY: all
all:
touch -d "`wget --spider -S $(URL) 2>&1 | sed -n 's/.*Modified: //p'`" modified
$(MAKE) stackoverflow.ico
stackoverflow.ico: modified
#echo Icon was modified. Downloading.; \
rm -f favicon.ico; \
wget $(URL); \
convert favicon.ico $#
Explanations:
I replaced your conversion to png by a useless conversion to ico because the favicon.ico icon of SO is composite and its conversion to png creates two files instead of one, named stackoverflow-0.png and stackoverflow-1.png, which uselessly complicates things.
all is a true phony target and the default goal so, each time you invoke make (or make all), it is built. It first updates the modified tag file and then invokes make again to build stackoverflow.ico.
The second make invocation does nothing if stackoverflow.ico is more recent than modified, else it downloads and converts.
Your second question (get a message about required manual operations) is completely different and simpler to solve. Let's first define a message and echo it in the file's recipe:
define DIY_message
Dear user, you should first:
- do this
- and that.
Unless you know it is useless, of course.
endef
export DIY_message
the_remote_file:
#echo "$$DIY_message"
The message will be printed if make is invoked with this goal (make the_remote_file) or with a goal that somehow depends on the_remote_file and either:
the_remote_file does not exist,
or the_remote_file exists but is out-of-date with respect to its prerequisites (if you declared prerequisites for it),
or the_remote_file is a prerequisite of .PHONY.
Note: using an intermediate make variable assigned by a define-endef makes things easier for formatted multi-lines messages.
(this is similar to GNU make: Execute target but take dependency from file but slightly different).
When I try to force to build a target, make automatically thinks that this target is out of date and forces a run of all targets which depend on it.
In my case, the target is a recursive make call which does a lot of work and might just return with "nothing to be done":
.PHONY: intermediate.dat
intermediate.dat:
$(MAKE) expensive_chain_which_finally_creates_intermediate.dat
step1.dat: intermediate.dat
sleep 10
step2.dat: step1.dat
sleep 15
step3.dat: step2.dat
sleep 10
all: step3.dat
sleep 5
In this case, "make all" runs for 40 seconds although intermediate.dat might not have changed (recursive make returned "nothing to be done"). However, if the recursive make updated intermediate.dat, the target shall be out of date.
Is there really no way to do this?
Make intermediate.dat depend on a phony target instead of being phony itself.
.PHONY : always-remake
intermediate.dat : always-remake
IIRC, the last time I solved the problem, the .PHONY didn't work as intended and I used:
always-remake :
#true
instead. However, I can't recall why, so try the .PHONY first.
The problem with making intermediate.dat itself phony is that make never checks the existence/date of a phony file, which is behaviour that you want. You only need to trigger the rebuild rule, which is done by a prerequisite that is out of date; a phony prerequisite is always out of date, so it does the job.
Another trick is to use an order-only prerequisite, plus a recursive call to make. This can be useful when you want to implement some additional logic around when the intermediate.dat file needs to be created, or when you want to decouple a target from the usual exists && newer than check that make does.
For example:
Let's say that the file contains some date-sensitive material that subsequent tasks depend on, and that it expires after 12 hours. We only want to recreate the file when any of these is true:
The file is older than 12 hours
The file does not exist
To implement the make target & recipes, we'll create a .PHONY check-intermediate.dat target, which runs the time check, conditionally removes the file if it's expired, and then re-runs make for that file.
Note that the check-intermediate.dat target will be defined via the order-only prerequisite syntax (e.g. intermediate.dat: | check-intermediate.dat) which causes it to run always before the target (e.g. intermediate.dat). The time check is inexpensive, and we always want it to run and manage the file for us, but not invalidate the original make check on intermediate.dat if it already exists like a normal prerequisite .PHONY target would.
.PHONY: check-intermediate.dat
check-intermediate.dat:
if [ -e intermediate.dat ]; then find intermediate.dat -mmin +720 -exec bash -c 'rm -f "{}"; $(MAKE) intermediate.dat' \; ; fi
intermediate.dat: | check-intermediate.dat
echo run-expensive-tasks-here-to-create-intermediate.dat
step1.dat: intermediate.dat
sleep 10
step2.dat: step1.dat
sleep 15
step3.dat: step2.dat
sleep 10
all: step3.dat
sleep 5
Now, anything that depends on the intermediate.dat target will first run the check, remove the file if it's expired, and re-run make intermediate.dat before running the rest of the dependent targets. If the intermediate.dat file already exists, it will run the check, and if the check passes, it will continue without running the time expensive recreation task.
The problem with depending on a .PHONY target is that it will always run the tasks which depend on it, because of the way GNU Make handles .PHONY targets. This can cause the make run to always perform the time expensive operations even when it does not need to:
A phony target should not be a prerequisite of a real target file; if it is, its recipe will be run every time make goes to update that file. As long as a phony target is never a prerequisite of a real target, the phony target recipe will be executed only when the phony target is a specified goal (see Arguments to Specify the Goals).
Part of my Makefile:
CPUDEPS=./mydeps.cpu
(...)
deps: $(CPUDEPS)
$(CPUDEPS): $(CCFILES)
#echo [DEPS] CPU
$(CMDECHO)makedepend -Y -s'# CPU sources dependencies generated with "make deps"' \
-w4096 -f- -- $(CFLAGS) -- $^ 2> /dev/null > $(CPUDEPS)
(...)
sinclude $(CPUDEPS)
Problem 1: includes are done during the first phase of processing, targets during the second phase; so, if ./mydeps.cpu doesn't exist and I "make deps", I get first the error
Makefile:335: ./mydeps.cpu: No such file or directory
I hide the error using sinclude instead of include, but the problem is still there: the old file is included, not the just-generated-one. Have to run it twice to include the updated file. This is because make does a two-phase processing; is there any way to tell make to complete the target deps before parsing the includes?
Problem 2: even if the file ./mydeps.cpu doesn't exist and make deps actually creates it, I always get a "make: Nothing to do for deps". This doesn't happen with other targets. I don't understand why and how to avoid it.
Problem 1 is non-existant: before building a target, make automatically rebuilds makefiles (with implicit rules if no explicit rule is provided). So having a rule for the makefile ensures that will always be up to date, there is no need to run deps twice. Additionally, since CPUDEPS is a makefile, it will be updated automatically before any other rule is run, so dependencies will always be updated if necessary and make deps is not needed. You can probably notice this by yourself by observing the [DEPS] line being echoed if any of the CCFILES becomes more recent that the dependency file.
For Problem 2, adding anything to the recipe ensures that make doesn't complain about having nothing to do. If there is nothing else, you can use something like #echo OK to give feedback to the user, or a simple #true if you prefer totally silent makes.
What you are trying to achieve is useless: you can use the dependencies file that was created during the previous build. That's enough.
The main reasoning behind that rule is:
if you haven't changed any of your files, then the dependencies file is up-to-date, and there's nothing to build.
if you have changed anything, even very deep into your #include chain, on an existing file that were used by previous build, then the dependencies file have already caught it. You'll rebuild what is needed.
if you change something in a new file (you add that file!) then it was not used by previous build, and not listed in dependencies. But if you really want to use it, then you have to modify at least one of your other files that was used before, and you're back on the previous case.
The solution is to create the dependencies file during the normal process of the compilation, and to optionally include it (with sinclude) if it is present.