How to integrate an external, black-box build system like go build into Make?
I don't think this question is specific to Go, but I will use it as an example. The go build system tracks the relation between inputs and outputs (= the dependencies) internally, and avoids rebuilding the output if no inputs have changed.
I have a Makefile which contains targets based on shell scripts and targets based on invoking go build, for example:
my-exe:
go build <some-url>
intermediate: my-exe
<expensive shell script>
test: intermediate test-data
<some test>
Requirements:
Let the make goal be test.
When I touch the test-data, I don't want to run the intermediate target.
When I touch the go source code, I want to run all steps.
Options considered:
It is possible to list the go source files as dependencies of my-exe. However, my go source folder contains files for multiple targets, and I would have to somehow list the right files/folders in my Makefile. I could also overshoot and list all go source files as dependencies in the Makefile.
If I turn my-exe into a phony target, then requirement 2. is fulfilled but 1. is broken.
Renaud's solution will work. But as in my comment above, I think all you need to do is take advantage of the FORCE target trick described here: https://www.gnu.org/software/make/manual/html_node/Force-Targets.html
Change your makefile to this:
FORCE:
my-exe: FORCE
go build <some-url>
intermediate: my-exe
<expensive shell script>
test: intermediate test-data
<some test>
By adding a prerequisite that can never be satisfied to my-exe you will force it to always be built. Assuming that the go build ... command doesn't actually update the my-exe target if nothing changed and does update it when something changed, this makefile will work the way you want.
Let's abstract all this a bit with 2 targets: always and expensive. We want to always build always because it is cheap and it relies on a external build system. But we want to build expensive only if building always changed something. The solution consists in declaring always as phony, such that it is always remade, but not declaring it as a prerequisite of expensive, else it will always be remade too. We thus need a third target, relay that is not phony, that is also cheap and that will really change only if always did something.
In the following the GO make variable is used to emulate the external build system. If it is set to a non-empty string always changes, else it stays the same.
expensive: relay
#echo "$#"
#touch "$#"
relay: always
#echo "$#"
#if ! [ -f "$#" ] || [ "$<" -nt "$#" ]; then touch "$#"; fi
.PHONY: always
always:
#echo "$#"
#if ! [ -f "$#" ] || [ -n "$(GO)" ]; then touch "$#"; fi
And that's it:
$ make
always
relay
expensive
$ make
always
relay
$ make GO=1
always
relay
expensive
If we turn the external build step into a phony target, it will always be built. However, any target which depends on a phony target is also always built. GNU make documentation:
A phony target should not be a prerequisite of a real target file; if it is, its recipe will be run every time make goes to update that file.
There are two ways to use this:
Make the external build step depend on a phony target:
.PHONY: always-rebuild
my-exe: always-rebuild
go build -o my-exe <some-url> # creates my-exe
intermediate: my-exe
<expensive shell script>
test: intermediate test-data
<some test>
(GNU) make evaluates the timestamp of my-exe after the target my-exe has run. If the target did not change the timestamp, then the succeeding steps (intermediate, test) are not run.
Introduce a dummy target to remove the "phony" property:
.PHONY: external_my-exe
external_my-exe:
go build -o my-exe <some-url> # creates my-exe
my-exe: external_my-exe
# do nothing!
true
intermediate: my-exe
<expensive shell script>
test: intermediate test-data
<some test>
external_my-exe is always built (if it occurs in the dependency tree of the make goal).
my-exe is always built because it depends on a phony target.
intermediate depends on an actual file (my-exe), so it is only run if the file's timestamp changed.
Related
I have a Makefile with tons of targets and would like for a certain script to get executed first, irrespective of what target is being called. I like to call it a global prerequisite.
I do not want to create a target for the script and set it as a prerequisite for all existing targets (which, as I said aren't few). Besides, someone else could add a target in future and not add my script as a prerequisite for their target, so the global prerequisite would take care of that.
Does GNU-make provide for a means to achieve this?
Another approach:
-include dummy
.PHONY: dummy
dummy:
run-the-script
Make will always attempt to rebuild any file which the makefile attempts to include (if it is out of date or does not exist). In this case there is no such file, and the rule to build it runs the script and does nothing else.
There is a solution without modifying your existing Makefile (main difference with the answers pointed to by tripleee). Just create a makefile containing:
.PHONY: all
all:
pre-script
#$(MAKE) -f Makefile --no-print-directory $(MAKECMDGOALS) MAKE='$(MAKE) -f Makefile'
post-script
$(MAKECMDGOALS): all ;
The only drawback is that the pre- and post- scripts will always be run, even if there is nothing else to do. But they will not be run if you invoke make with one of the --dry-run options (other difference with the answers pointed to by tripleee).
I know this has been asked before, but none of the solutions I've found work for me because they're anti-DRY.
I have a number of targets that depend on things that can't readily be timestamped -- such as files copied from another system. What I'd like to be able to do is list dependencies in a variable, like nobuild=this,that, and have those targets be assumed to be up-to-date. Since I have a lot of these, I don't want to ifdef around each one; what would be pseudocodibly preferable would be something like
ignorable-target: dependencies
$(call ifnobuild,$#)
.. rest of normal build steps ..
where the ifnobuild macro expanded to some sort of exit-from-this-recipe-with-success gmake instruction if ignorable-target was mentioned in the nobuild variable.
I also don't want to get into multi-line continued shell commands in order to defer the conditional to the recipe itself; I want to be able to tell make "Assume these targets are up-to-date and don't try to build them," so I can test other aspects with the local copies already obtained from the problematic recipes.
There isn't any sort of exit-recipe-with-success mechanism in gmake, is there?
[Edited to hopefully make the situation more clear.]
Here's an example. Targets remote1 and remote2 each involve using ssh to do something time-consuming on a remote system, and then copying the results locally. Target local1 is built locally, and isn't a time sink. target-under-work depends on all three of the above.
local1: local1.c Makefile
remote1: local1
scp local1 remote-host:/tmp/
ssh remote-host /tmp/local1 some-args # takes a long time
scp remote-host:/tmp/local1.out remote1
remote2: local1
scp local1 other-host:/tmp/
ssh other-host /tmp/local1 other-args # takes a long time
scp other-host:/tmp/local1.out remote2
target-under-work: local1 remote1 remote2
do-something-with remote1,remote2
Now, when I just run make target-under-work, it's going to run the recipes for remote1 and remote2. However, the local copies of those files are 'good enough' for my testing, so I don't want them run every time. Once things go into production, they will be run every time, but while I'm developing target-under-work, I just want to use the copies already built, and I can rebuild them daily (or whatever) for the necessary testing granularity.
The above is over-simplified; there are multiple steps and targets that depend on remote1 and/or remote2. I see how I can get the effect I want by making them order-only prerequisites -- but that would mean changing the dependency list of every target that has them as prerequisites, rather than making a single change to remote1 and remote2 so I can use some variable from the command line to tell their recipes 'pretend this has been built, don't actually build it if there's already a copy.'
I hope this makes my question more clear.
No, this early exit make feature does not exist.
Note that your problem is probably under-specified because you don't explain what behaviour you want when a slow target does not exist yet.
Let's assume that the slow targets listed in nobuild shall be rebuilt if and only if they don't exist. Instead of using make functions to early exit their recipe you could use make functions to "hide" their list of prerequisites. This way, if they already exist, they will not be rebuilt, even if they are outdated. The only subtlety is that you will need the second expansion to use the $# automatic variable in the lists of prerequisites. In the following example slow (your remoteX) depends on fast1 (your local1). fast2 (your target-under-work) depends on fast1 and slow:
host> cat Makefile
# Expands as empty string if $(1) exists and
# listed in $(nobuild). Else expands as $(2).
# $(1): target
# $(2): prerequisites
define HIDE_IF_NOBUILD
$(if $(wildcard $(1)),$(if $(filter $(1),$(nobuild)),,$(2)),$(2))
endef
nobuild :=
fast1:
#echo 'build $#'
#touch $#
fast2: fast1 slow
#echo 'build $#'
#touch $#
.SECONDEXPANSION:
slow: $$(call HIDE_IF_NOBUILD,$$#,fast1)
#echo 'build $#'
#touch $#
# Case 1: slow target not listed in nobuild and not existing
host> rm -f slow; touch fast1; make fast2
build slow
build fast2
# Case 2: slow target not listed in nobuild and existing and outdated
host> touch slow; sleep 2; touch fast1; make fast2
build slow
build fast2
# Case 3: slow target listed in nobuild and not existing
host> rm -f slow; touch fast1; make nobuild="slow" fast2
build slow
build fast2
# Case 4: slow target listed in nobuild and existing and outdated
host> touch slow; sleep 2; touch fast1; make nobuild="slow" fast2
build fast2
Basically I have the usual Makefile construct:
target: dependency1 dependency2 dependency3
runtargetscript.sh
However in this case, the target only needs one of the dependencies and some dependencies may not be buildable. (so I cannot just build all dependencies)
Is it possible to tell make to trigger "target" when one of the dependencies changed/was created (i.e. normal behaviour) but NOT to try to rebuild any missing dependencies?
With GNU make you can use shell escapes to build the dependencies dynamically, adding them only if they already exist:
if_exist = $(shell if [ -e $(1) ]; then echo $(1); fi)
target: $(call if_exist,dependency1) $(call if_exist,dependency2) $(call if_exist,dependency3)
runtargetscript.sh
This will run the script if target does not exist, or if it is older than any of the dependencies that do exist at the time the makefile was read, but will not attempt to build them if they do not exist at that time.
Note the important caveat there -- if the file(s) do not exist, but some other unrelated rule runs an action that creates them, it won't rebuild target, unless you rerun make target again.
Assuming your rule body does not do anything special depending on which dependencies are newer than the target you can use the -W flag to make to instruct it to consider certain targets as always new (and thus not in need of building).
So for the given example assuming you can (and want) to build dependency2 but not dependency1 or dependency3 you would run:
make -W dependency1 -W dependency3 target
Edit: As pointed out in the comments this does not work correctly when dependency2 is not newer than target as target will still be built.
In that case I believe the only solution (given the comments below) is to use something like:
DEPENDENCY_BIN := $(or $(wildcard /path/to/mysql),$(wildcard /path/to/sqlite3),/path/that/does/not/exist)
target: $(DEPENDENCY_BIN) dependency2
The Problem:
Is it possible to give a target a different name or alias, such that it can be invoked using either the original target name or the alias.
For example something like
/very/long/path/my_binary: dep_a dep_b dep_c
# Compile
# Desired command
ALIAS my_binary = /very/long/path/my_binary
# NOTE: Notice the use of 'my_binary' in the dependencies
data1: my_binary datafile
# Build data file using compiled my_binary
Attempt 1: .PHONY
I have tried using a .PHONY target:
.PHONY: my_binary
my_binary: /very/long/path/my_binary
This works great when invoked from the command-line:
# Runs rule 'my_binary' and then *only* runs rule '/very/long/path/my_binary'
# if the rule '/very/long/path/my_binary' needs updating.
make my_binary
However, this does not work well when the alias my_binary is listed as a dependency:
# *Always* thinks that rule 'data1' needs updating, because it always thinks that
# the .PHONY target 'my_binary' "needs updating". As a result, 'data1' is
# rebuilt every time.
make /very/long/path/my_binary
Possible hack?
A possible hack is to use an empty target as suggested in an answer to this question, but that would require introducing fake files with names corresponding to the alias:
my_binary: /very/long/path/my_binary
touch my_binary
This will clutter the main directory with files! Placing the fake files in a sub-directory would defeat the purpose, as the alias would have to be referred to as 'directory/my_binary'
Okay, I needed something similar. The path to my output artifacts were quite long, but I wanted short target names and also benefit easily from bash-completion.
Here is what I'm came up with:
os := [arbitrary long path to an artifact]
platform := [arbitrary long path to a differ artifact]
packer := [common parts of my packer build command]
.PHONY: all
all: $(platform)
.PHONY: platform
platform: $(platform)
$(platform): platform.json $(os)
#$(packer) $<
.PHONY: os
os: $(os)
$(os): os.json
#$(packer) $<
.PHONY: clean
clean:
rm -fr build/
With the Makefile above you can say:
$ make os
$ make platform
Which will be aliases for the long artifact names. I've made the snippet above quite long, because it's important to see the relationships between the .PHONY aliases and the real targets. I hope that works for you.
Note: I did not delete the clean target from the above example, because many people does not make that a .PHONY target. However, semantically it should be.
I don't think there's any way to do it so that you can use the alias from within your makefile as well as the command line, except by creating those temporary files.
Why can't you just set a variable in the makefile, like:
my_binary = /very/long/path/my_binary
then use $(my_binary) everywhere in the makefile? I don't see any point in creating a real alias target for use inside the makefile.
I had a somewhat similar need. I wanted users of my makefile to be able to enter any of the following to accomplish the same result, such that the following were effectively synonyms of each other:
make hit list
make hitlist
make hit_list
What I did in my makefile was the following:
hit_list:
#echo Got here
<the real recipe goes here>
hit: hit_list
hitlist: hit_list
.PHONY: list
list:
#echo > /dev/null
Then, when I tested it using any of the commands "make hit list", "make hitlist", or "make hit_list", I got identical results, as intended.
By extension, if one of your targets was the one with the long name but you used this approach whereby a simple short name identified the target with the long name as a prerequisite, I think that you should be able to say "make short_name" and accomplish what you're asking about.
This differs from your Approach 1 in that none of the synonyms is defined as a phony target (considering that "make hit list" is a command to make two targets, the second being effectively a noop), so the complication that you described would not arise.
What does .PHONY mean in a Makefile? I have gone through this, but it is too complicated.
Can somebody explain it to me in simple terms?
By default, Makefile targets are "file targets" - they are used to build files from other files. Make assumes its target is a file, and this makes writing Makefiles relatively easy:
foo: bar
create_one_from_the_other foo bar
However, sometimes you want your Makefile to run commands that do not represent physical files in the file system. Good examples for this are the common targets "clean" and "all". Chances are this isn't the case, but you may potentially have a file named clean in your main directory. In such a case Make will be confused because by default the clean target would be associated with this file and Make will only run it when the file doesn't appear to be up-to-date with regards to its dependencies.
These special targets are called phony and you can explicitly tell Make they're not associated with files, e.g.:
.PHONY: clean
clean:
rm -rf *.o
Now make clean will run as expected even if you do have a file named clean.
In terms of Make, a phony target is simply a target that is always out-of-date, so whenever you ask make <phony_target>, it will run, independent from the state of the file system. Some common make targets that are often phony are: all, install, clean, distclean, TAGS, info, check.
Let's assume you have install target, which is a very common in makefiles. If you do not use .PHONY, and a file named install exists in the same directory as the Makefile, then make install will do nothing. This is because Make interprets the rule to mean "execute such-and-such recipe to create the file named install". Since the file is already there, and its dependencies didn't change, nothing will be done.
However if you make the install target PHONY, it will tell the make tool that the target is fictional, and that make should not expect it to create the actual file. Hence it will not check whether the install file exists, meaning: a) its behavior will not be altered if the file does exist and b) extra stat() will not be called.
Generally all targets in your Makefile which do not produce an output file with the same name as the target name should be PHONY. This typically includes all, install, clean, distclean, and so on.
NOTE: The make tool reads the makefile and checks the modification time-stamps of the files at both the side of ':' symbol in a rule.
Example
In a directory 'test' following files are present:
prerit#vvdn105:~/test$ ls
hello hello.c makefile
In makefile a rule is defined as follows:
hello:hello.c
cc hello.c -o hello
Now assume that file 'hello' is a text file containing some data, which was created after 'hello.c' file. So the modification (or creation) time-stamp of 'hello' will be newer than that of the 'hello.c'. So when we will invoke 'make hello' from command line, it will print as:
make: `hello' is up to date.
Now access the 'hello.c' file and put some white spaces in it, which doesn't affect the code syntax or logic then save and quit. Now the modification time-stamp of hello.c is newer than that of the 'hello'. Now if you invoke 'make hello', it will execute the commands as:
cc hello.c -o hello
And the file 'hello' (text file) will be overwritten with a new binary file 'hello' (result of above compilation command).
If we use .PHONY in makefile as follow:
.PHONY:hello
hello:hello.c
cc hello.c -o hello
and then invoke 'make hello', it will ignore any file present in the pwd 'test' and execute the command every time.
Now suppose, that 'hello' target has no dependencies declared:
hello:
cc hello.c -o hello
and 'hello' file is already present in the pwd 'test', then 'make hello' will always show as:
make: `hello' is up to date.
.PHONY: install
means the word "install" doesn't represent a file name in this
Makefile;
means the Makefile has nothing to do with a file called "install"
in the same directory.
It is a build target that is not a filename.
The special target .PHONY: allows to declare phony targets, so that make will not check them as actual file names: it will work all the time even if such files still exist.
You can put several .PHONY: in your Makefile :
.PHONY: all
all : prog1 prog2
...
.PHONY: clean distclean
clean :
...
distclean :
...
There is another way to declare phony targets : simply put :: without prerequisites :
all :: prog1 prog2
...
clean ::
...
distclean ::
...
The :: has other special meanings, see here, but without prerequisites it always execute the recipes, even if the target already exists, thus acting as a phony target.
The best explanation is the GNU make manual itself: 4.6 Phony Targets section.
.PHONY is one of make's Special Built-in Target Names. There are other targets that you may be interested in, so it's worth skimming through these references.
When it is time to consider a .PHONY target, make will run its recipe
unconditionally, regardless of whether a file with that name exists or
what its last-modification time is.
You may also be interested in make's Standard Targets such as all and clean.
There's also one important tricky treat of ".PHONY" - when a physical target depends on phony target that depends on another physical target:
TARGET1 -> PHONY_FORWARDER1 -> PHONY_FORWARDER2 -> TARGET2
You'd simply expect that if you updated TARGET2, then TARGET1 should be considered stale against TARGET1, so TARGET1 should be rebuild. And it really works this way.
The tricky part is when TARGET2 isn't stale against TARGET1 - in which case you should expect that TARGET1 shouldn't be rebuild.
This surprisingly doesn't work because: the phony target was run anyway (as phony targets normally do), which means that the phony target was considered updated. And because of that TARGET1 is considered stale against the phony target.
Consider:
all: fileall
fileall: file2 filefwd
echo file2 file1 >fileall
file2: file2.src
echo file2.src >file2
file1: file1.src
echo file1.src >file1
echo file1.src >>file1
.PHONY: filefwd
.PHONY: filefwd2
filefwd: filefwd2
filefwd2: file1
#echo "Produced target file1"
prepare:
echo "Some text 1" >> file1.src
echo "Some text 2" >> file2.src
You can play around with this:
first do 'make prepare' to prepare the "source files"
play around with that by touching particular files to see them updated
You can see that fileall depends on file1 indirectly through a phony target - but it always gets rebuilt due to this dependency. If you change the dependency in fileall from filefwd to file, now fileall does not get rebuilt every time, but only when any of dependent targets is stale against it as a file.
I often use them to tell the default target not to fire.
superclean: clean andsomethingelse
blah: superclean
clean:
#echo clean
%:
#echo catcher $#
.PHONY: superclean
Without PHONY, make superclean would fire clean, andsomethingelse, and catcher superclean; but with PHONY, make superclean won't fire the catcher superclean.
We don't have to worry about telling make the clean target is PHONY, because it isn't completely phony. Though it never produces the clean file, it has commands to fire so make will think it's a final target.
However, the superclean target really is phony, so make will try to stack it up with anything else that provides deps for the superclean target — this includes other superclean targets and the % target.
Note that we don't say anything at all about andsomethingelse or blah, so they clearly go to the catcher.
The output looks something like this:
$ make clean
clean
$ make superclean
clean
catcher andsomethingelse
$ make blah
clean
catcher andsomethingelse
catcher blah