Let's pretend I have two servers A and B that I want to have the ability to deploy to separately and together in Makefile targets. Let's also say there's a preparation step that I want to run exactly once before any deployment, whether it's to one or both. How could I do that?
Here's what I have so far, but it's not DRY.
clean:
# do cleanup stuff
deploy:
make clean
ssh $(SERVER) do-remote-deployment-stuff
deploy-to-a:
deploy SERVER=myservera
deploy-to-b:
deploy SERVER=myserverb
deploy-all:
deploy-to-a
deploy-to-b
See how it'd run 'clean' twice?
Instead of using recursion, just put the target you only want to run once as a prerequisite:
deploy-to-a: clean
...
deploy-to-b: clean
...
However, your attempt to set SERVER= etc. won't work, and neither will your deploy-all rule. You can't put make targets into make recipes. You can use pattern rules:
SERVERS = A B
A_SERVER = myservera
B_SERVER = myserverb
deploy-all: $(SERVERS:%=deploy-to-%)
deploy-to-%: clean
ssh $($*_SERVER) do-remote-deployment-stuff
clean:
# do cleanup stuff
You can specify the clean target as a dependency for your other targets. In order to keep it DRY, you could have something like this in your Makefile:
SERVERLIST = myservera myserverb
.PHONY: clean deploy-all $(SERVERLIST)
clean:
# do cleanup stuff
$(SERVERLIST): clean
ssh $# do-remote-deployment-stuff
deploy-all: $(SERVERLIST)
You could define as many servers as you want with the SERVERLIST variable.
Running make -n deploy-all shows you the commands that will be run:
# do cleanup stuff
ssh myservera do-remote-deployment-stuff
ssh myserverb do-remote-deployment-stuff
To deploy to just one server, you can run make myservera, etc. You can of course create an "alias" for your server as well:
SERVERLIST = my.longservername.com someother.server.somewhere.else.com
myservera: someother.server.somewhere.else.com
Related
I have some software built using parallel multi-level makefiles and I see that when my main Makefile runs two separate targets from a submakefile that have the same dependency, this dependency is run twice simultaneously and an error is created.
Consider the following main Makefile in the project root folder:
TARGETS = t1 t2 t3 t4 t5 t6 t7 t8
.PHONY: all $(TARGETS) clean
all: $(TARGETS)
$(TARGETS):
#echo Making $#
#sleep 1
$(MAKE) -C folder s$#
clean:
#echo Making $#
$(MAKE) -C folder clean
and the sub-makefile folder/Makefile:
SUBTARGETS = st1 st2 st3 st4 st5 st6 st7 st8
$(SUBTARGETS): dep
#echo Making $#
#sleep 1
#touch $#
dep:
#echo Making $#
#sleep 1
#echo bla >> dep
clean:
rm -f $(SUBTARGETS)
rm -f dep
rm -f dep2dump
Then running make -j8 in the root folder will run targets t1...t8 in parallel, which will then run subtargets st1...st8, which all depend on dependency dep. From the shell output and the contents of the dep file (8 lines) it is obvious that the dep rule is run 8 times, as if the 8 implications of folder/Makefile are completely independent.
I thought submakes coordinated when running in parallel and that they would avoid running the same target twice, but it seems this is not the case.
Can anyone suggest a correct way to solve such a case?
If eventually this is an unavoidable weakness of make, what alternative build tools should I look into?
Thanks
EDIT: The answers by MadScientist and Renaud Pacalet are useful but don't exactly solve my problem because they both require that the author of the top-level makefile has knowledge about the internals of the sub-makefile. I have not explained this requirement explicitly in my original post though.
So to give more details, the use case I am trying to solve is that where the source code in path folder/ is a separate project, eg. a collection of utilities st1...st8 where all (or some) of them have a dependency on library dep, internal to the utilities project in folder. Then I want to be able to use this sub-project (as seamlessly as possible) in various master projects, each of them using just a (possible different) subset of the utilities st1...st8. Additionally, the master project may contain many targets t1...t8, each depending on a different subset of st1...st8, as shown in my example above. Targets t1...t8 need to be able to run separately, building only the required dependencies from subproject (so make t1 only builds st1, etc), thus having to build all st1...st8 for each one of t1...t8 is not desired. On the other hand they also need to be able to run in parallel, eg. by running make all.
Ideally I would not want the author of each master makefile to have to know about internals of sub-project, nor have to include in the sub-makefile all the possible combinations of st1...st8 so that each master project can call just ONE of these to avoid the parallel build issue.
So far I have in mind but not tested the following imperfect solutions:
As Renaud suggested, use something like flock to at least ensure that the multiple runs of dep (by separate sub-make instances) won't happen simultaneously. Cons: requires extra tool (flock or similar) to be installed + dep runs multiple times, so extra work is needed to avoid doing the actual compilation over and over again, otherwise just eat the performance cost.
Include the sub-makefile in the master makefile so that everything runs in one make instance. This requires makes the sub-makefile able to work regardless of the path of the master makefile that includes it. No big issue. Cons: merging / including two makefile from different authors can open a can of worms, i.e. variables with same name, etc.
Modify sub-makefile as described in (2) + In the main project create another makefile, eg. utils.make, that contains a rule for the targets of sub-makefile needed and includes the sub-makefile. So utils.make will be (assuming this master project only needs st1, st5 and st7:
utils: st1 st5 st7
include foldes/Makefile
Then the master makefile will have a utils-ext rule as dependency of each of t1...t8 that will be:
utils-ext:
$(MAKE) -f rules.make utils
to build all the utils needed. This keeps the two main makefiles separate but has all utils / subtargets built when building any single one of t1...t8, which is suboptimal.
You could try to move the dep dependency to your top Makefile:
.PHONY: all $(TARGETS) clean dep
all: $(TARGETS)
$(TARGETS): dep
#echo Making $#
#sleep 1
$(MAKE) -C folder s$#
dep:
$(MAKE) -C folder s#
The only decent solution to your problem is to have ONE instance of make build all the sub-directory targets you want. Having the parent make invoke multiple sub-makes in parallel in the same directory, unless every invocation uses a completely disjoint set of targets, is a guaranteed fail situation. So if you have multiple things you want to do in the submake you should collect them all in one invocation of the sub-make and let the sub-make's parallelism handle it for you.
You could do something like this:
TARGETS = t1 t2 t3 t4 t5 t6 t7 t8
.PHONY: all $(TARGETS) clean
all: $(TARGETS)
$(TARGETS): .submake ;
.submake:
$(MAKE) -C folder $(addprefix s,$(MAKECMDGOALS))
Then in the sub-make add this so that when invoked with no arguments it builds everything:
all: $(SUBTARGETS)
Here, if you run make then the sub-make is invoked with no arguments and builds all the things in parallel. If you invoke make t1 t2 then the submake is invoked with the arguments st1 st2.
Alternatively, you can re-architect your makefiles so that you don't use recursive make at all, and one instance of make knows all the different rules and dependency relationships.
I know this has been asked before, but none of the solutions I've found work for me because they're anti-DRY.
I have a number of targets that depend on things that can't readily be timestamped -- such as files copied from another system. What I'd like to be able to do is list dependencies in a variable, like nobuild=this,that, and have those targets be assumed to be up-to-date. Since I have a lot of these, I don't want to ifdef around each one; what would be pseudocodibly preferable would be something like
ignorable-target: dependencies
$(call ifnobuild,$#)
.. rest of normal build steps ..
where the ifnobuild macro expanded to some sort of exit-from-this-recipe-with-success gmake instruction if ignorable-target was mentioned in the nobuild variable.
I also don't want to get into multi-line continued shell commands in order to defer the conditional to the recipe itself; I want to be able to tell make "Assume these targets are up-to-date and don't try to build them," so I can test other aspects with the local copies already obtained from the problematic recipes.
There isn't any sort of exit-recipe-with-success mechanism in gmake, is there?
[Edited to hopefully make the situation more clear.]
Here's an example. Targets remote1 and remote2 each involve using ssh to do something time-consuming on a remote system, and then copying the results locally. Target local1 is built locally, and isn't a time sink. target-under-work depends on all three of the above.
local1: local1.c Makefile
remote1: local1
scp local1 remote-host:/tmp/
ssh remote-host /tmp/local1 some-args # takes a long time
scp remote-host:/tmp/local1.out remote1
remote2: local1
scp local1 other-host:/tmp/
ssh other-host /tmp/local1 other-args # takes a long time
scp other-host:/tmp/local1.out remote2
target-under-work: local1 remote1 remote2
do-something-with remote1,remote2
Now, when I just run make target-under-work, it's going to run the recipes for remote1 and remote2. However, the local copies of those files are 'good enough' for my testing, so I don't want them run every time. Once things go into production, they will be run every time, but while I'm developing target-under-work, I just want to use the copies already built, and I can rebuild them daily (or whatever) for the necessary testing granularity.
The above is over-simplified; there are multiple steps and targets that depend on remote1 and/or remote2. I see how I can get the effect I want by making them order-only prerequisites -- but that would mean changing the dependency list of every target that has them as prerequisites, rather than making a single change to remote1 and remote2 so I can use some variable from the command line to tell their recipes 'pretend this has been built, don't actually build it if there's already a copy.'
I hope this makes my question more clear.
No, this early exit make feature does not exist.
Note that your problem is probably under-specified because you don't explain what behaviour you want when a slow target does not exist yet.
Let's assume that the slow targets listed in nobuild shall be rebuilt if and only if they don't exist. Instead of using make functions to early exit their recipe you could use make functions to "hide" their list of prerequisites. This way, if they already exist, they will not be rebuilt, even if they are outdated. The only subtlety is that you will need the second expansion to use the $# automatic variable in the lists of prerequisites. In the following example slow (your remoteX) depends on fast1 (your local1). fast2 (your target-under-work) depends on fast1 and slow:
host> cat Makefile
# Expands as empty string if $(1) exists and
# listed in $(nobuild). Else expands as $(2).
# $(1): target
# $(2): prerequisites
define HIDE_IF_NOBUILD
$(if $(wildcard $(1)),$(if $(filter $(1),$(nobuild)),,$(2)),$(2))
endef
nobuild :=
fast1:
#echo 'build $#'
#touch $#
fast2: fast1 slow
#echo 'build $#'
#touch $#
.SECONDEXPANSION:
slow: $$(call HIDE_IF_NOBUILD,$$#,fast1)
#echo 'build $#'
#touch $#
# Case 1: slow target not listed in nobuild and not existing
host> rm -f slow; touch fast1; make fast2
build slow
build fast2
# Case 2: slow target not listed in nobuild and existing and outdated
host> touch slow; sleep 2; touch fast1; make fast2
build slow
build fast2
# Case 3: slow target listed in nobuild and not existing
host> rm -f slow; touch fast1; make nobuild="slow" fast2
build slow
build fast2
# Case 4: slow target listed in nobuild and existing and outdated
host> touch slow; sleep 2; touch fast1; make nobuild="slow" fast2
build fast2
I am looking at a pre-existing, working, complex makefile for a project which will both build and deploy the code on multiple OS's.
I'm looking at some separate IDE support (Visual Studio) for the build process (i.e. half the make will already be done), so need to insert a phony target for the deploy action such that the old flow (make all) still works, including the deloy step, but that make deploy will just do the final deployment step for those using the IDE.
Not being familiar with make, I'm having difficulty seeing how/if make allows such an entry point and if so how to implement it.
The current code has:
$(BUILT_INS): git$X
$(QUIET_BUILT_IN)$(RM) $# && \
ln $< $# 2>/dev/null || \
ln -s $< $# 2>/dev/null || \
cp $< $#
whose actions are the deployment step.
So conceptually I think I need
.PHONY: deploy
$(BUILT_INS): git$X
deploy
deploy:
$(QUIET_BUILT_IN)$(RM) $# && \
ln $< $# 2>/dev/null || \
ln -s $< $# 2>/dev/null || \
cp $< $#
which is clearly not right, because the phony target can't be an action.
In summary; How to create an entry point into a makefile to do the rule's actions? (a critical desire is to avoid duplicating the action code)
Your "conceptually" solution has many more problems than using a target as a recipe (which you're right, won't work, but you could fix by using $(MAKE) deploy as the recipe to invoke a recursive make); the other thing is that this:
$(BUILT_INS):
... using $# ...
means run that recipe one time for each word in the BUILT_INS variable, and each time the automatic variable $# will be assigned to that word (the target).
Your replacement:
deploy:
... using $# ...
does an entirely different thing: it runs the recipe one time, with the value of $# set to deploy. Not going to work.
The simple answer to your question is that you just declare a new target deploy that lists the targets you want to run as prerequisites:
.PHONY: deploy
deploy: $(BUILT_INS)
Now when you run make deploy it will try to build the BUILT_INS targets, and run the install rule for each one.
However, my suspicion is that this will be a problem for you, depending on what the git$X prerequisite is... it might cause a big part of the rest of your makefile to run as well. But, you don't give any information about that so I can't say.
ETA Sure enough, git$X is a problem. So, it appears you want VS to generate your git$X file (which will be git.exe presumably). Then you want to run make deploy to copy it. The trick here is to keep make from rebuilding git$X when you use the deploy target. You can do something like this; replace the rule that builds git$X with:
.PHONY: deploy
deploy: $(BUILT_INS)
ifeq (,$(filter deploy,$(MAKECMDGOALS)))
git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS)
$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $# $(ALL_LDFLAGS) git.o \
$(BUILTIN_OBJS) $(LIBS)
endif
This creates a new target deploy which depends on the BUILT_INS targets, as I show above. Then, I enclose the rule to create git$X inside a test so that if you run make deploy that rule is not defined. Now make doesn't know how to build git$X at all, so if it doesn't exist then make deploy will fail, but if it does exist then make will copy it, without trying to rebuild it.
I inherited a project that has multiple top level Makefiles, one for each very-similar platform that the firmware image runs on. They have names like:
Makefile.apple.mak
Makefile.banana.mak
...
So every time I want to build a specific image, I run the command
Make -f Makefile.apple.mak // default is release
Much of the code is shared between the different images that get built. A problem with doing this is that sometimes when a change is made that compiles and works for the apple platform, it breaks the banana platform. There is a whole mess of #ifdefs that conditionally include or exclude code based on the platform specified by the Makefile. I know, it's a mess and badly in need of a refactor (did I mention I inherited this?)
In order to easily build all of the firmware images after making a change and declaring it good on one platform, I want to have a wrapping Makefile that invokes a build of one or all of the available firmware images.
I am currently looking at this brute-force approach. I know there's got to be a more efficient way to do this. I also don't know offhand how to pass additional arguments to this outer Makefile (i.e. make apple debug) if that is possible. Any help is appreciated.
.PHONY: all
all: cleanall releaseall
.PHONY: cleanall
cleanall:
make -f Makefile.apple.mak clean # Can I instead call apple target with an arg?
make -f Makefile.banana.mak clean # Can I instead call banana target with an arg?
.PHONY: releaseall
releaseall:
make -f Makefile.apple.mak
make -f Makefile.banana.mak
.PHONY: apple
apple:
make -f Makefile.apple.mak # how do I pass in the <clean | debug | release> args?
.PHONY: banana
banana:
make -f Makefile.banana.mak # how do I pass in ...
(BTW, you never want to use make to invoke a recursive make. Always use $(MAKE))
The way to do this is construct target names encoding the image and the target for that image you want to build. Then you can use make prerequisites.
So, for example, you can do something like this:
IMAGES := apple banana ...
all: cleanall releaseall
cleanall: $(IMAGES:%=%.clean)
releaseall: $(IMAGES)
$(IMAGES):
$(MAKE) -f Makefile.$#.mak
$(IMAGES:%=%.clean):
$(MAKE) -f Makefile.$(basename $#).mak clean
.PHONY: all cleanall releaseall $(IMAGES) $(IMAGES:%=%.clean)
This also lets you run make apple.clean to clean just the apple directory, or whatever.
You can add more targets like debug, etc. as well.
At work we use a common makefile that other makefiles include (via the include statement) and it has a generic "clean" target that kills some common files. I want to add on to that target in my new makefile so I can delete some specific files, but if I add a clean target in my makefile, it just overrides the old one.
I know I can just make a new target with a new name and have it call clean, and then do other stuff, but for sake of consistency I'd like to be able to just call make clean and have it do everything.
Is that possible?
I've seen this done at several shops. The most common approach is to use double-colon rules, assuming you're using something like GNU make. In your common makefile you would have something like this:
clean::
# standard cleanup, like remove all .o's:
rm -f *.o
Note that there are two colons following clean, not just one!
In your other makefile you just declare clean again, as a double-colon rule:
clean::
# custom cleanup, like remove my special generated files:
rm -f *.h.gen
When you invoke make clean, GNU make will automagically run both of these "branches" of the clean rule:
% make clean
rm -f *.o
rm -f *.h.gen
It's simple to set up and it composes quite neatly I think. Note that specifically because it is a double-colon rule, you don't get the "overriding commands" errors you normally get when you define two rules for the same target. That's sort of the point of double-colon rules.
You can write your own clean and make it a preq of the common clean.
clean: myclean
myclean:
rm whatever
Yours will run first. If for some reason you want the common clean to run first then the solution will be more complicated.
EDIT:
Here is the best solution I can see which runs the common rule before the local one:
include Makefile.common
clean:
$(MAKE) -f Makefile.common $#
rm whatever additional things
The include directive is necessary because the local makefile relies on the common one for things other than clean. The local clean rule overrides the common clean rule, but invokes the common clean rule before doing the additional work. (This overriding will cause some warnings, which is a nuisance; I don't know a good way to silence them.)
Use implicit rules:
existing-target: my-extention
my-extention:
echo running command 1
echo running command 2
Very simple make tutorial to ramp up.
When using :: you can run into issues since make complains when you mix single colon : and double colon :: rules:
a:
echo a
a::
echo aa
will result in:
. . .
*** target file `a' has both : and :: entries. Stop.
It seems like the common makefile's rule should be called something like common-clean. Then each main makefile would declare their clean rule as
clean: common-clean
and you're set.
If that isn't an option, you could take a look at double colon rules, but those introduce a whole other set of issues to consider.
Adding another possible solution I've seen for posterity... I know the OP was wary about changing the common makefile, but something like this works and involves minimal changes.
local makefile 1:
CLEAN=MyExe1 MyExe2
....
include /my/common/makefile
local makefile 2:
CLEAN=MyExe3 MyExe4
....
include /my/common/makefile
common makefile:
clean:
rm -f *.dep *.o *.a $(CLEAN)
Basically the idea is to define some variable (in this case CLEAN) in each local makefile with all the specific items you want to delete. Then the common makefile runs rm -f on all the common file types to delete, plus whatever was specifically flagged for deletion in each local makefile via the CLEAN variable. If there's nothing specific to delete, simply omit the variable declaration or leave it empty (CLEAN=)
So now if we run make clean for local makefile 1, it executes
rm -f *.dep *.o *.a MyExe1 MyExe2
And if we run make clean for local makefile 2, it executes
rm -f *.dep *.o *.a MyExe3 MyExe4
I've found a better solution:
.PHONY: my-extra-clean
clean: my-extra-clean
my-extra-clean:
rm <whatever-you-want>
include Makefile.common
The key line is clean: my-extra-clean. Ie, you can add dependencies in separate stanzas in different makefiles to add behaviour. my-extra-clean is run as a dependency of the root clean target.
For ours, we define a variable, EXTRAFILESTOCLEAN, then when the clean rule runs, it has a step to remove anything specified in the EXTRAFILESTOCLEAN variable
clean:
rm -f *.o
ifdef $(EXTRAFILESTOCLEAN)
rm -f $(EXTRAFILESTOCLEAN)
endif
That can cause unexpected problems if you set that variable to weird values, but you could guard against those by adding prefixes or other tests.
It's in the docs: https://www.gnu.org/software/make/manual/html_node/Overriding-Makefiles.html
So instead of include Makefile you use a wildcard target and forward it to the base Makefile:
# -include base.Makefile <--- not this
%:
#$(MAKE) -f base.Makefile $#