TL; DR
A dependency shared by multiple targets is only invoked once.
Why?
Summary
In this tiny Makefile example, I expect deploy-everywhere to deploy staging-repo code to staging, and prod-repo code to prod.
Instead, both staging and prod get staging-repo code, which is, admittedly, suboptimal.
This seems to be happening because clone-repo is skipped / pruned when it is examined 2nd time around, as part of deploy-to-prod (see -d log output below).
Why is this happening, and can I change this behavior?
$ cat Makefile
.PHONY: deploy-everywhere deploy-to-prod deploy-to-staging clone-repo
deploy-everywhere: deploy-to-staging deploy-to-prod
#echo "deployed everywhere"
deploy-to-prod: repo="prod-repo"
deploy-to-prod: clone-repo
#echo "deployed to prod (from $(repo))"
deploy-to-staging: repo="staging-repo"
deploy-to-staging: clone-repo
#echo "deployed to staging (from $(repo))"
clone-repo:
#echo "clone-repo $(repo)"
More Details (logs, version)
Here is the output from running make:
$ make deploy-everywhere
clone-repo staging-repo
deployed to staging (from staging-repo)
deployed to prod (from prod-repo) <<<<< THIS IS A LIE, WE NEVER CLONED THE PROD REPO!!
deployed everywhere <<<< RIIIIGHT.
Extra logging:
$ make -d deploy-everywhere
...
Successfully remade target file `deploy-to-staging'.
Considering target file `deploy-to-prod'.
File `deploy-to-prod' does not exist.
Pruning file `clone-repo'. <<<< WHY? WHY?!?!?!?! WHY.
Finished prerequisites of target file `deploy-to-prod'.
Must remake target `deploy-to-prod'.
...
I am running GNU Make 3.81:
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin11.3.0
This is by design. Targets in a makefile are more like nouns than like verbs.
You can achieve what you want several ways. Here is one:
deploy-to-prod: clone-prod-repo
#echo "deployed to prod (from prod-repo)"
deploy-to-staging: clone-staging-repo
#echo "deployed to staging (from staging-repo)"
clone-%-repo:
#echo "clone $*-repo"
EDIT: All right, the clone operation takes multiple parameters. Here is one way to do it:
deploy-to-prod: PARAM1=ProdOne
deploy-to-prod: PARAM2=ProdTwo
deploy-to-prod: clone-prod-repo
#echo "deployed to prod (from prod-repo)"
deploy-to-staging: PARAM1=StagingFirst
deploy-to-staging: PARAM2=StagingSecond
deploy-to-staging: clone-staging-repo
#echo "deployed to staging (from staging-repo)"
clone-%-repo:
#echo clone $*-repo with $(PARAM1) and $(PARAM2)
And here is another:
define clone-repo
#echo clone using $(1)
#echo and $(2) in some way
endef
deploy-to-prod:
#echo $#
$(call clone-repo, ProdOne, ProdTwo)
deploy-to-staging:
#echo $#
$(call clone-repo, StagingFirst, StagingSecond)
To add some explanation to Beta's solution: The programmatic contract of a make rule is that it is an opaque update of a permanent entity in the filesystem, thereby satisfying a "less-than" (younger-than) relation laid down in the dependency tree. Your construction OTOH hides the dependency from make by passing it as a parameter. The problem that you tried to circumvent is that there is no notion for "less-than" in make for non-filesystem-present entities (in your case, a revision control system).
The only question that I have is why you didn't simply add the clone call to each deploy target - if you really want to solve it as a "less-than" dependency (ie. cloning only if local repository non-existent or older) there is quite some make programming necessary.
EDIT after your first comment:
If I got you right, your idea of PHONY is a bit off (not totally, just a bit). PHONY is not about facilitating something across different make runs, it is a way to have a short-circtuit in your dependency tree. Given a chain foo -> bar -> phony_baz (foo depending on bar depending on phony_baz) make will, in the case that foo was the main target, first look at bar to see if it is older than phony_baz which always will be the case. This is because phony_baz itself has a make-implicit property that will always yield a "younger_than" when it is compared as a prerequisite against a target (and now comes the important part:) at the first encounter in the dependency tree. All subsequent encounters of phony_baz in other branches of the tree will do the exact opposite, terminating the evaluation right there at the phony target. This is no different from the behaviour at any other node in the dependency graph - make assumes that the contract which I noted above is fulfilled and no repetitive action is justified. This is the explanation for your clone-repo not firing a second time although it is .PHONY - your setup is effectively a directed acyclic graph where two edges join at the same node (which make is completely fine with, just worth noting), not a tree . Beta's first solution is to dynamically create two entirely unrelated nodes by parametrizing the rule, thereby returning the shape to a tree.
Related
I know what a .PHONY does.
If in the folder where my Makefile is, I add an empty file called clean and after I run make clean all of the clean target will not be executed since there was not any change in the file, so the target will not run and this is correct.
If I add .PHONY: clean, than the clean is seen as a command and this is also correct.
My question is why this behavior does not happen the same to all target, since I added a all file in the folder.So basically the all target still executes like if it was a .PHONY: all
I have the fallowing makefile code.
all: test1 test2
test1: test1.o
test1.o: test1.c
test2: test2.o
test2.o: test2.c
clean:
rm -rf *.o test1 test2
How do you know that the all rule is "still executing"? That rule has no recipe, so there's no way it can be "executed".
If what you mean is that even though the all file exists in the local directory, make is still building the targets test1 and test2, that's how make works (this doesn't have anything to do with phony vs. non-phony targets). When make decides whether or not build a particular target first it tries to build all the prerequisites of that target, and all the prerequisites of those targets, etc. Only after all that is complete, can make know whether or not to build the first target (all in this case).
make clean here doesn't have any dependencies, so putting a file named clean there is enough for the target to be considered built.
make all on the other hand has dependencies. Even if you put a file named all there, Make has to check whether the all file is newer than test1 and test2. This process triggers builds of test1 and test2, and it happens to have the same effect as if all was a phony target.
The basis is that all: test1 test2 is a recipe for building a file named all, that depends on the files test1 and test2.
If you ran make all, Make would do something like this:
Analyse the Makefile.
Find out that all depends on test1 and test2.
Check the timestamp of all and see if it is "up to date".
It is "up to date" if none of the dependencies are newer than itself.
In other words, Make can skip building a file if it's newer than all it's dependencies.
Build outdated or missing files.
Now, if you would like to prevent Make from considering the targets as files, you could specify them as phony targets. That is best practice for non-file targets like all.
(This answer isn't disagreeing with either of the existing answers, but suggesting another way of thinking about this).
When you have a rule like
dst: src
action
you're saying two things (as you know):
if dst doesn't exist, or is older than src, then do action; and
when action completes, the file dst will exist.
With targets such as all or clean, the second statement is of course not true. Make doesn't hold you to the promise in (2), so when you say make all, it'll compute and generate the required dependencies, and not complain that there's no file all in place afterwards. You're lying to Make, but it doesn't mind (it's cool with that...). That is, this is basically a makefile hack.
Where this goes wrong, of course, is if for some reason there happens to be a file called all or clean. Then Make will take the modification date of the file all into account when calculating the dependencies, and possibly come to a conclusion you didn't expect.
So what .PHONY: all does is legitimise the hack, and tells Make ‘even if a file all exists, pretend that it doesn't’; you're basically cancelling promise (2).
Therefore, as the other answers state, mentioning .PHONY isn't necessary. It simply forestalls an error – easy to make but easy to miss – when a file matching a phony target is accidentally created.
I know this has been asked before, but none of the solutions I've found work for me because they're anti-DRY.
I have a number of targets that depend on things that can't readily be timestamped -- such as files copied from another system. What I'd like to be able to do is list dependencies in a variable, like nobuild=this,that, and have those targets be assumed to be up-to-date. Since I have a lot of these, I don't want to ifdef around each one; what would be pseudocodibly preferable would be something like
ignorable-target: dependencies
$(call ifnobuild,$#)
.. rest of normal build steps ..
where the ifnobuild macro expanded to some sort of exit-from-this-recipe-with-success gmake instruction if ignorable-target was mentioned in the nobuild variable.
I also don't want to get into multi-line continued shell commands in order to defer the conditional to the recipe itself; I want to be able to tell make "Assume these targets are up-to-date and don't try to build them," so I can test other aspects with the local copies already obtained from the problematic recipes.
There isn't any sort of exit-recipe-with-success mechanism in gmake, is there?
[Edited to hopefully make the situation more clear.]
Here's an example. Targets remote1 and remote2 each involve using ssh to do something time-consuming on a remote system, and then copying the results locally. Target local1 is built locally, and isn't a time sink. target-under-work depends on all three of the above.
local1: local1.c Makefile
remote1: local1
scp local1 remote-host:/tmp/
ssh remote-host /tmp/local1 some-args # takes a long time
scp remote-host:/tmp/local1.out remote1
remote2: local1
scp local1 other-host:/tmp/
ssh other-host /tmp/local1 other-args # takes a long time
scp other-host:/tmp/local1.out remote2
target-under-work: local1 remote1 remote2
do-something-with remote1,remote2
Now, when I just run make target-under-work, it's going to run the recipes for remote1 and remote2. However, the local copies of those files are 'good enough' for my testing, so I don't want them run every time. Once things go into production, they will be run every time, but while I'm developing target-under-work, I just want to use the copies already built, and I can rebuild them daily (or whatever) for the necessary testing granularity.
The above is over-simplified; there are multiple steps and targets that depend on remote1 and/or remote2. I see how I can get the effect I want by making them order-only prerequisites -- but that would mean changing the dependency list of every target that has them as prerequisites, rather than making a single change to remote1 and remote2 so I can use some variable from the command line to tell their recipes 'pretend this has been built, don't actually build it if there's already a copy.'
I hope this makes my question more clear.
No, this early exit make feature does not exist.
Note that your problem is probably under-specified because you don't explain what behaviour you want when a slow target does not exist yet.
Let's assume that the slow targets listed in nobuild shall be rebuilt if and only if they don't exist. Instead of using make functions to early exit their recipe you could use make functions to "hide" their list of prerequisites. This way, if they already exist, they will not be rebuilt, even if they are outdated. The only subtlety is that you will need the second expansion to use the $# automatic variable in the lists of prerequisites. In the following example slow (your remoteX) depends on fast1 (your local1). fast2 (your target-under-work) depends on fast1 and slow:
host> cat Makefile
# Expands as empty string if $(1) exists and
# listed in $(nobuild). Else expands as $(2).
# $(1): target
# $(2): prerequisites
define HIDE_IF_NOBUILD
$(if $(wildcard $(1)),$(if $(filter $(1),$(nobuild)),,$(2)),$(2))
endef
nobuild :=
fast1:
#echo 'build $#'
#touch $#
fast2: fast1 slow
#echo 'build $#'
#touch $#
.SECONDEXPANSION:
slow: $$(call HIDE_IF_NOBUILD,$$#,fast1)
#echo 'build $#'
#touch $#
# Case 1: slow target not listed in nobuild and not existing
host> rm -f slow; touch fast1; make fast2
build slow
build fast2
# Case 2: slow target not listed in nobuild and existing and outdated
host> touch slow; sleep 2; touch fast1; make fast2
build slow
build fast2
# Case 3: slow target listed in nobuild and not existing
host> rm -f slow; touch fast1; make nobuild="slow" fast2
build slow
build fast2
# Case 4: slow target listed in nobuild and existing and outdated
host> touch slow; sleep 2; touch fast1; make nobuild="slow" fast2
build fast2
The Problem:
Is it possible to give a target a different name or alias, such that it can be invoked using either the original target name or the alias.
For example something like
/very/long/path/my_binary: dep_a dep_b dep_c
# Compile
# Desired command
ALIAS my_binary = /very/long/path/my_binary
# NOTE: Notice the use of 'my_binary' in the dependencies
data1: my_binary datafile
# Build data file using compiled my_binary
Attempt 1: .PHONY
I have tried using a .PHONY target:
.PHONY: my_binary
my_binary: /very/long/path/my_binary
This works great when invoked from the command-line:
# Runs rule 'my_binary' and then *only* runs rule '/very/long/path/my_binary'
# if the rule '/very/long/path/my_binary' needs updating.
make my_binary
However, this does not work well when the alias my_binary is listed as a dependency:
# *Always* thinks that rule 'data1' needs updating, because it always thinks that
# the .PHONY target 'my_binary' "needs updating". As a result, 'data1' is
# rebuilt every time.
make /very/long/path/my_binary
Possible hack?
A possible hack is to use an empty target as suggested in an answer to this question, but that would require introducing fake files with names corresponding to the alias:
my_binary: /very/long/path/my_binary
touch my_binary
This will clutter the main directory with files! Placing the fake files in a sub-directory would defeat the purpose, as the alias would have to be referred to as 'directory/my_binary'
Okay, I needed something similar. The path to my output artifacts were quite long, but I wanted short target names and also benefit easily from bash-completion.
Here is what I'm came up with:
os := [arbitrary long path to an artifact]
platform := [arbitrary long path to a differ artifact]
packer := [common parts of my packer build command]
.PHONY: all
all: $(platform)
.PHONY: platform
platform: $(platform)
$(platform): platform.json $(os)
#$(packer) $<
.PHONY: os
os: $(os)
$(os): os.json
#$(packer) $<
.PHONY: clean
clean:
rm -fr build/
With the Makefile above you can say:
$ make os
$ make platform
Which will be aliases for the long artifact names. I've made the snippet above quite long, because it's important to see the relationships between the .PHONY aliases and the real targets. I hope that works for you.
Note: I did not delete the clean target from the above example, because many people does not make that a .PHONY target. However, semantically it should be.
I don't think there's any way to do it so that you can use the alias from within your makefile as well as the command line, except by creating those temporary files.
Why can't you just set a variable in the makefile, like:
my_binary = /very/long/path/my_binary
then use $(my_binary) everywhere in the makefile? I don't see any point in creating a real alias target for use inside the makefile.
I had a somewhat similar need. I wanted users of my makefile to be able to enter any of the following to accomplish the same result, such that the following were effectively synonyms of each other:
make hit list
make hitlist
make hit_list
What I did in my makefile was the following:
hit_list:
#echo Got here
<the real recipe goes here>
hit: hit_list
hitlist: hit_list
.PHONY: list
list:
#echo > /dev/null
Then, when I tested it using any of the commands "make hit list", "make hitlist", or "make hit_list", I got identical results, as intended.
By extension, if one of your targets was the one with the long name but you used this approach whereby a simple short name identified the target with the long name as a prerequisite, I think that you should be able to say "make short_name" and accomplish what you're asking about.
This differs from your Approach 1 in that none of the synonyms is defined as a phony target (considering that "make hit list" is a command to make two targets, the second being effectively a noop), so the complication that you described would not arise.
I have a Makefile with a pair of rules like the following:
file.c: filesvn
filesvn: .svn/entries
action1
action2
Within the svn repository, this of course works fine. The file is dependent upon living in a subversion repository. When exported from the repository, this does not work (No rule to make target...), and I would like to fix that. I have tried to simply bring a previously-generated version of filesvn out into the exported directory, but Make still insists on verifying filesvn's dependency.
Simply deleting the dependency of .svn/entries does work, of course, but then the spirit of the rule is broken since watching for a revision update is the goal.
Is there a way to get Make to not care that the .svn/entries file is not there?
The merits of such a technique are not really part of the question. I can't fundamentally change this, but if there is a change that maintains the spirit that might work. An answer of "You can't" is perfectly valid of course. :)
You can use the wildcard function to ignore the file unless it exists:
filesvn: $(wildcard .svn/entries)
Aside: Subversion 1.7 changes the format of the local working copy, so this rule would stop working even in working copies.
Pattern % rules
For pattern rules, you need .SECONDEXPANSION:
.SECONDEXPANSION:
%.a: %.b $$(wildcard $$*.c)
cc $< -o $#
See also: How can I make a pattern rule dependency optional in a Makefile?
Tested in Make 4.1.
You can add a target that generates copy of actual makefile with changes that drop the svn dependency, run make with it and then deletes it.
I have a project that I am working to release that actually contains 3 subprojects, all of which need to be compiled in one go. My makefile looks roughly like this:
all: a b c
a:
#cd a && make
b:
#cd b && make
c:
#cd c && make
Projects A and B both compile fine but for the 3rd project, it tells me there is nothing to be done although switching to the C directory and running make does in fact compile code.
To be a little more specific: Project C in the example above is actually Mozilla's SpiderMonkey. Whereas A and B are code/makefiles that I have written, C is just a raw copy of SpiderMonkey from the Mozilla website. The actually compile command for it is:
make JS_DIST=/usr JS_THREADSAFE=1 JS_HAS_FILE_OBJECT=1
In my master Makefile, I have:
spidermonkey:
#cd spidermonkey/src && $(MAKE) JS_DIST=/usr JS_THREADSAFE=1 JS_HAS_FILE_OBJECT=1
Running "make spidermonkey" outputs "make: Nothing to be done for `spidermonkey'." How do I get make to run the command?
EDIT:
I've tried adding the following lines to my makefile:
.PHONY: spidermonkey
As well as renaming the spidermonkey rule to sm, but still no change.
EDIT:
My bad! I had spaces when I should have had a tab. doh!
You probably have a file or directory at the toplevel called "spidermonkey". Make thinks this is what its supposed to create, and since it is already there, make stops.
One of the most important rules to follow when writing makefiles is each target should create one file with the same name as the target. In other words, if you have
a:
<some command>
That command should produce a single file called "a".
Rules which do not produce files but are only there as placeholders are called phony targets, and they should be declared like this:
.PHONY: a
Make will then always assume that a has to be remade.
Also, as a general rule do not use "make" to invoke make recursively, use $(MAKE) instead.
EDIT: changed "pseudo" to "phony"
Make only checks for the existance of a file (or directory) named the same as the rule target, and if there is (and it is newer than the dependencies) then from make's point of view there is nothing more to do.
So your problem is that you have a spidermonkey rule (with no dependencies) as well as a directory called spidermonkey, and then make thinks "the target is already made, nothing for me to do". To get make to do what you want, rename the spidermonkey rule (or the directory).
Speaking of recursive make by the way, this is not neccessarily a good idea,
see Recursive Make Considered Harmful.