I have some "phony" targets that I occasionally want to run in a specific order, but are not strictly dependent on each other. For example:
.PHONY: image-build
image-build: Dockerfile
docker build --tag foobar --file $< .
.PHONY: image-tag
image-tag: image-build
docker tag foobar foobar:latest
.PHONY: image-push
image-push:
docker push foobar:latest
I don't always want to run image-tag before image-push. However, if I do happen to run them together, as in make image-tag image-push, I want to ensure that image-tag runs before image-push.
That is, whenever image-tag and image-push both appear in the dependency graph, I want image-tag to be executed first, but when image-push is in the dependency graph and image-tag is not, I do not want image-tag to be added to the dependency graph.
Is it possible to enforce this rule in GNU Make?
For now, I have created a workaround command:
.PHONY: image-deploy
image-deploy: image-tag
$(MAKE) image-push
You can do this:
image-push: $(filter image-tag,$(MAKECMDGOALS))
docker push foobar:latest
The filter will expand to nothing if image-tag is not specified as a goal target (on the command line), else it will expand to image-tag.
Related
I have a Makefile as below, and I am concerned with the dependency order for up-clean:
.PHONY: up
up: down
docker-compose up -d
.PHONY: up-clean
up-clean: down-clean up
.PHONY: down
down:
docker-compose down
.PHONY: down-clean
down-clean:
docker-compose down -v
Obviously it is important that in up-clean: down-clean up, down-clean must be executed before up. For ordinary make targets, the solution would be to add an entry, up: down-clean, but as these are PHONY targets, that would make up functionally equivalent to up-clean, removing volumes every time. Obviously, this is unacceptable.
In practice, GNU make respects the order of dependencies, but it does not guarantee it, and so is not entirely trustworthy, or with -j not trustworthy at all. What methods, if any, exist to ensure the order of execution of dependencies in this situation without changing the result of building other targets?
EDIT: Attempting to use order-only prerequisites does not appear to work, possibly because of an interaction with .PHONY. Adding
up: | down-clean
Causes the execution log to be:
$ make up
docker-compose down
<...>
docker-compose down -v
Removing volume <...>
Which is what is supposed to happen for normal prerequisites, not order-only ones.
The simplest answer is to use recursive invocations of make:
up-clean:
$(MAKE) down-clean
$(MAKE) up
Another alternative would be to model up-clean on up instead of making the latter a prerequisite for the former:
.PHONY: up
up: down
docker-compose up -d
.PHONY: up-clean
up-clean: down-clean
docker-compose up -d
If you want to make that a little DRYer, you could factor out the docker-compose command to a variable:
UP_COMMAND = docker-compose up -d
.PHONY: up
up: down
$(UP_COMMAND)
.PHONY: up-clean
up-clean: down-clean
$(UP_COMMAND)
Starting with GNU make 4.4 you can explicitly serialize your prerequisites with the .WAIT pseudo-target, e.g.,
.PHONY: up-clean
up-clean: down-clean .WAIT up
Before 4.4, the GNU make documentation was avoiding making any commitments about the order or execution to enable parallel execution. However, any POSIX-compliant implementation of make (emphasis mine)
... shall treat all prerequisites as targets themselves and recursively ensure that they are up-to-date, processing them in the order in which they appear in the rule. The make utility shall use the modification times of files to determine whether the corresponding targets are out-of-date.
Of course, in the parallel mode, GNU make can't be fully compliant with this requirement.
But, the 4.4 release also adds the --shuffle option, and from the research made by the implementor of this option, it is evident that the only source of non-determinism is the -j option and the parallel mode of execution, which is also witnessed by the contents of the patch and the tests, which were reviewed by other members of the GNU make project.
Therefore, in version prior to 4.4, we can safely assume that the prerequisites are executed strictly in the order in which they are specified, from left to right, as long as make is executed in non-parallel mode. In versions before 4.4, we can disable parallelism with the .NOTPARALLEL pseudo target, just add it to your file, e.g.,
.NOTPARALLEL: # ensures that all deps are executed strictly in order
.PHONY: up
up: down
docker-compose up -d
.PHONY: up-clean
up-clean: down-clean up
.PHONY: down
down:
docker-compose down
.PHONY: down-clean
down-clean:
docker-compose down -v
Notice also, that --shuffle respects the presence of the .NOTPARALLEL target, which corroborates our hypothesis the order could different from the syntactic order only because of the parallel execution.
The problem I'm experiencing is an all target has dependencies on others that set a variable, then run matching dependencies.
Outcome - It will run the first dependency then stop.
Expected - Run both dependencies, setting the variable properly between each run
Is make smart enough to see pull and build were already ran and the dependency target itself has no execution, therefore it sees all dependencies as complete? Or I'm just abusing make in ways it should not be used?
Said makefile:
repo=rippio
image=default
pull:
#docker pull $(repo)/$(image):latest
build: pull
#sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$(image)/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$(image):custom .
#rm -f Dockerfile
node: image=node
node: | pull build
jdk8: image=jdk8
jdk8: | pull build
all: | node jdk8
TLDR
It is used to:
Pull the latest docker image
Run a generically designed Dockerfile against it to customize it
Tag it as :custom for internal use
Pretty handy for customizing images in a generic manner without managing many Dockerfiles.
Dockerfile template (Dockerfile.in), incase interested:
FROM {{repo}}/{{image}}:latest
... super secret sauce
UPDATE (ANSWER)
Thanks to #G.M., ended up with:
IMAGE_NAMES := node jdk8
TARGETS := $(patsubst %,build-%,$(IMAGE_NAMES))
repo=rippio
all: $(TARGETS)
build-%: pull-%
#$sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$*/' Dockerfile.in > Dockerfile-$* && \
$docker build -f=Dockerfile-$* -t=$(repo)/$*:custom .
#rm -f Dockerfile-$*
pull-%:
#$docker pull $(repo)/$*:latest
Which allows for:
Easy upkeep of 'all' targets, which constantly grows
Running parallel via make -j (note the Dockerfile-$* file pattern)
Much more beautiful than before
If you draw your dependency graph out long-hand you'll see that there are multiple paths from all to both pull and build -- one via each of node and jdk8. But make having reached/updated pull and build via one path will then assume that they are both up to date and, hence, not bother to update them further -- regardless of any change to target specific variables.
I think what you're trying to do (assuming I've understood correctly) might be more easily achieved using pattern rules.
IMAGE_NAMES := node jdk8
TARGETS := $(patsubst %,build-%,$(IMAGE_NAMES))
repo=rippio
all: $(TARGETS)
build-%: pull-%
#$sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$*/' Dockerfile.in > Dockerfile && \
$docker build -t=$(repo)/$*:custom .
#rm -f Dockerfile
pull-%:
#$docker pull $(repo)/$*:latest
Note: You currently have all build recipes using the same input/output file DockerFile. That will cause problems if you ever want to use parallel builds -- make -j etc. It might be wise to use the stem from the pattern rule match to uniquely identify the output file if that's possible.
Normally, if you invoke make with:
make all
and if none of pull, build, node, jdk8 are existing files, make should build pull and build. If you see only pull being made, it can be because you invoke make without specifying a goal. In this case make builds the first target it finds in the Makefile (pull in your case).
Anyway, there are several strange aspects in your Makefile: you use order-only prerequisites on what looks like phony targets and these phony targets are not declared as such.
I am not sure I fully understand what you are trying to do but maybe something like this would be a good starting point:
repo=rippio
image=default
.PHONY: all build node jdk8
all: node jdk8
node: image = node
jdk8: image = jdk8
build node jdk8:
#docker pull $(repo)/$(image):latest && \
sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$(image)/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$(image):custom . && \
rm -f Dockerfile
Note: if, instead of build you name the default target default you could even simplify further with:
repo=rippio
.PHONY: all default node jdk8
all: node jdk8
default node jdk8:
#docker pull $(repo)/$#:latest && \
sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$#/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$#:custom . && \
rm -f Dockerfile
This is a continuation of Makefile: run the same command with different arguments on different targets.
Code
COMMIT := $(shell git rev-parse HEAD)
images := base web proxy lb users api
$(images): base
#echo "Building $#"
docker build -f Dockerfile.$# --no-cache=true -t $#:$(COMMIT) .
build: $(images)
rebuild: $(images) # I want this to run with --no-cache=true
Essentially, build calls all image targets (base being the first one), and runs docker build with --no-cache=true for each one.
The problem
I would like to have a rebuild target which runs all image targets with --no-cache=false rather than --no-cache=true, without duplicating the code. I guess that the right way is to set a variable in rebuild and build whose scope would cover dependent targets like any of the images.
My question
How do I define a variable in a target whose scope covers all dependent targets?
Quite similar, like in mentioned question:
images := base web proxy lb users api
$(images):
#echo $# docker --no-cache=$(NO_CACHE)
build: NO_CACHE=true
rebuild: NO_CACHE=false
rebuild build: $(images)
You may want to set a default value for NO_CACHE in case you would like to call make base for example.
I am looking at a pre-existing, working, complex makefile for a project which will both build and deploy the code on multiple OS's.
I'm looking at some separate IDE support (Visual Studio) for the build process (i.e. half the make will already be done), so need to insert a phony target for the deploy action such that the old flow (make all) still works, including the deloy step, but that make deploy will just do the final deployment step for those using the IDE.
Not being familiar with make, I'm having difficulty seeing how/if make allows such an entry point and if so how to implement it.
The current code has:
$(BUILT_INS): git$X
$(QUIET_BUILT_IN)$(RM) $# && \
ln $< $# 2>/dev/null || \
ln -s $< $# 2>/dev/null || \
cp $< $#
whose actions are the deployment step.
So conceptually I think I need
.PHONY: deploy
$(BUILT_INS): git$X
deploy
deploy:
$(QUIET_BUILT_IN)$(RM) $# && \
ln $< $# 2>/dev/null || \
ln -s $< $# 2>/dev/null || \
cp $< $#
which is clearly not right, because the phony target can't be an action.
In summary; How to create an entry point into a makefile to do the rule's actions? (a critical desire is to avoid duplicating the action code)
Your "conceptually" solution has many more problems than using a target as a recipe (which you're right, won't work, but you could fix by using $(MAKE) deploy as the recipe to invoke a recursive make); the other thing is that this:
$(BUILT_INS):
... using $# ...
means run that recipe one time for each word in the BUILT_INS variable, and each time the automatic variable $# will be assigned to that word (the target).
Your replacement:
deploy:
... using $# ...
does an entirely different thing: it runs the recipe one time, with the value of $# set to deploy. Not going to work.
The simple answer to your question is that you just declare a new target deploy that lists the targets you want to run as prerequisites:
.PHONY: deploy
deploy: $(BUILT_INS)
Now when you run make deploy it will try to build the BUILT_INS targets, and run the install rule for each one.
However, my suspicion is that this will be a problem for you, depending on what the git$X prerequisite is... it might cause a big part of the rest of your makefile to run as well. But, you don't give any information about that so I can't say.
ETA Sure enough, git$X is a problem. So, it appears you want VS to generate your git$X file (which will be git.exe presumably). Then you want to run make deploy to copy it. The trick here is to keep make from rebuilding git$X when you use the deploy target. You can do something like this; replace the rule that builds git$X with:
.PHONY: deploy
deploy: $(BUILT_INS)
ifeq (,$(filter deploy,$(MAKECMDGOALS)))
git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS)
$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $# $(ALL_LDFLAGS) git.o \
$(BUILTIN_OBJS) $(LIBS)
endif
This creates a new target deploy which depends on the BUILT_INS targets, as I show above. Then, I enclose the rule to create git$X inside a test so that if you run make deploy that rule is not defined. Now make doesn't know how to build git$X at all, so if it doesn't exist then make deploy will fail, but if it does exist then make will copy it, without trying to rebuild it.
I inherited a project that has multiple top level Makefiles, one for each very-similar platform that the firmware image runs on. They have names like:
Makefile.apple.mak
Makefile.banana.mak
...
So every time I want to build a specific image, I run the command
Make -f Makefile.apple.mak // default is release
Much of the code is shared between the different images that get built. A problem with doing this is that sometimes when a change is made that compiles and works for the apple platform, it breaks the banana platform. There is a whole mess of #ifdefs that conditionally include or exclude code based on the platform specified by the Makefile. I know, it's a mess and badly in need of a refactor (did I mention I inherited this?)
In order to easily build all of the firmware images after making a change and declaring it good on one platform, I want to have a wrapping Makefile that invokes a build of one or all of the available firmware images.
I am currently looking at this brute-force approach. I know there's got to be a more efficient way to do this. I also don't know offhand how to pass additional arguments to this outer Makefile (i.e. make apple debug) if that is possible. Any help is appreciated.
.PHONY: all
all: cleanall releaseall
.PHONY: cleanall
cleanall:
make -f Makefile.apple.mak clean # Can I instead call apple target with an arg?
make -f Makefile.banana.mak clean # Can I instead call banana target with an arg?
.PHONY: releaseall
releaseall:
make -f Makefile.apple.mak
make -f Makefile.banana.mak
.PHONY: apple
apple:
make -f Makefile.apple.mak # how do I pass in the <clean | debug | release> args?
.PHONY: banana
banana:
make -f Makefile.banana.mak # how do I pass in ...
(BTW, you never want to use make to invoke a recursive make. Always use $(MAKE))
The way to do this is construct target names encoding the image and the target for that image you want to build. Then you can use make prerequisites.
So, for example, you can do something like this:
IMAGES := apple banana ...
all: cleanall releaseall
cleanall: $(IMAGES:%=%.clean)
releaseall: $(IMAGES)
$(IMAGES):
$(MAKE) -f Makefile.$#.mak
$(IMAGES:%=%.clean):
$(MAKE) -f Makefile.$(basename $#).mak clean
.PHONY: all cleanall releaseall $(IMAGES) $(IMAGES:%=%.clean)
This also lets you run make apple.clean to clean just the apple directory, or whatever.
You can add more targets like debug, etc. as well.