This is a continuation of Makefile: run the same command with different arguments on different targets.
Code
COMMIT := $(shell git rev-parse HEAD)
images := base web proxy lb users api
$(images): base
#echo "Building $#"
docker build -f Dockerfile.$# --no-cache=true -t $#:$(COMMIT) .
build: $(images)
rebuild: $(images) # I want this to run with --no-cache=true
Essentially, build calls all image targets (base being the first one), and runs docker build with --no-cache=true for each one.
The problem
I would like to have a rebuild target which runs all image targets with --no-cache=false rather than --no-cache=true, without duplicating the code. I guess that the right way is to set a variable in rebuild and build whose scope would cover dependent targets like any of the images.
My question
How do I define a variable in a target whose scope covers all dependent targets?
Quite similar, like in mentioned question:
images := base web proxy lb users api
$(images):
#echo $# docker --no-cache=$(NO_CACHE)
build: NO_CACHE=true
rebuild: NO_CACHE=false
rebuild build: $(images)
You may want to set a default value for NO_CACHE in case you would like to call make base for example.
Related
I have some "phony" targets that I occasionally want to run in a specific order, but are not strictly dependent on each other. For example:
.PHONY: image-build
image-build: Dockerfile
docker build --tag foobar --file $< .
.PHONY: image-tag
image-tag: image-build
docker tag foobar foobar:latest
.PHONY: image-push
image-push:
docker push foobar:latest
I don't always want to run image-tag before image-push. However, if I do happen to run them together, as in make image-tag image-push, I want to ensure that image-tag runs before image-push.
That is, whenever image-tag and image-push both appear in the dependency graph, I want image-tag to be executed first, but when image-push is in the dependency graph and image-tag is not, I do not want image-tag to be added to the dependency graph.
Is it possible to enforce this rule in GNU Make?
For now, I have created a workaround command:
.PHONY: image-deploy
image-deploy: image-tag
$(MAKE) image-push
You can do this:
image-push: $(filter image-tag,$(MAKECMDGOALS))
docker push foobar:latest
The filter will expand to nothing if image-tag is not specified as a goal target (on the command line), else it will expand to image-tag.
How to integrate an external, black-box build system like go build into Make?
I don't think this question is specific to Go, but I will use it as an example. The go build system tracks the relation between inputs and outputs (= the dependencies) internally, and avoids rebuilding the output if no inputs have changed.
I have a Makefile which contains targets based on shell scripts and targets based on invoking go build, for example:
my-exe:
go build <some-url>
intermediate: my-exe
<expensive shell script>
test: intermediate test-data
<some test>
Requirements:
Let the make goal be test.
When I touch the test-data, I don't want to run the intermediate target.
When I touch the go source code, I want to run all steps.
Options considered:
It is possible to list the go source files as dependencies of my-exe. However, my go source folder contains files for multiple targets, and I would have to somehow list the right files/folders in my Makefile. I could also overshoot and list all go source files as dependencies in the Makefile.
If I turn my-exe into a phony target, then requirement 2. is fulfilled but 1. is broken.
Renaud's solution will work. But as in my comment above, I think all you need to do is take advantage of the FORCE target trick described here: https://www.gnu.org/software/make/manual/html_node/Force-Targets.html
Change your makefile to this:
FORCE:
my-exe: FORCE
go build <some-url>
intermediate: my-exe
<expensive shell script>
test: intermediate test-data
<some test>
By adding a prerequisite that can never be satisfied to my-exe you will force it to always be built. Assuming that the go build ... command doesn't actually update the my-exe target if nothing changed and does update it when something changed, this makefile will work the way you want.
Let's abstract all this a bit with 2 targets: always and expensive. We want to always build always because it is cheap and it relies on a external build system. But we want to build expensive only if building always changed something. The solution consists in declaring always as phony, such that it is always remade, but not declaring it as a prerequisite of expensive, else it will always be remade too. We thus need a third target, relay that is not phony, that is also cheap and that will really change only if always did something.
In the following the GO make variable is used to emulate the external build system. If it is set to a non-empty string always changes, else it stays the same.
expensive: relay
#echo "$#"
#touch "$#"
relay: always
#echo "$#"
#if ! [ -f "$#" ] || [ "$<" -nt "$#" ]; then touch "$#"; fi
.PHONY: always
always:
#echo "$#"
#if ! [ -f "$#" ] || [ -n "$(GO)" ]; then touch "$#"; fi
And that's it:
$ make
always
relay
expensive
$ make
always
relay
$ make GO=1
always
relay
expensive
If we turn the external build step into a phony target, it will always be built. However, any target which depends on a phony target is also always built. GNU make documentation:
A phony target should not be a prerequisite of a real target file; if it is, its recipe will be run every time make goes to update that file.
There are two ways to use this:
Make the external build step depend on a phony target:
.PHONY: always-rebuild
my-exe: always-rebuild
go build -o my-exe <some-url> # creates my-exe
intermediate: my-exe
<expensive shell script>
test: intermediate test-data
<some test>
(GNU) make evaluates the timestamp of my-exe after the target my-exe has run. If the target did not change the timestamp, then the succeeding steps (intermediate, test) are not run.
Introduce a dummy target to remove the "phony" property:
.PHONY: external_my-exe
external_my-exe:
go build -o my-exe <some-url> # creates my-exe
my-exe: external_my-exe
# do nothing!
true
intermediate: my-exe
<expensive shell script>
test: intermediate test-data
<some test>
external_my-exe is always built (if it occurs in the dependency tree of the make goal).
my-exe is always built because it depends on a phony target.
intermediate depends on an actual file (my-exe), so it is only run if the file's timestamp changed.
i have something like this in my makefile :
JAR = jar1.jar
run:
java -cp $(JAR)
what i want to do is for the makefile to do run multiple times but with each iteration, it uses another jar, they're all called jarx.jar with x going from 1 to 10 for example
is it possible to do it without passing the jar name inside the statement? the example i gave is quite simple for the sake of simplicity but the actual makefile i'm working with is already quite complicated...
make uses filesystem objects to keep state. A common solution to keeping track of things which are not directly visible as files in the current directory is to create a local flag file.
JARS := first.jar 2nd.jar thirdjar.jar
.PHONY: all run
all: $(patsubst %, .made-%,$(JARS))
run: all
.made-%.jar: %.jar
java -cp $<
touch $#
So the existence of .made-first.jar signals to Make that this target has been performed for the prerequisite first.jar, etc.
If you have a clean target, or at least realclean, it should probably clean up all the flag files.
The problem I'm experiencing is an all target has dependencies on others that set a variable, then run matching dependencies.
Outcome - It will run the first dependency then stop.
Expected - Run both dependencies, setting the variable properly between each run
Is make smart enough to see pull and build were already ran and the dependency target itself has no execution, therefore it sees all dependencies as complete? Or I'm just abusing make in ways it should not be used?
Said makefile:
repo=rippio
image=default
pull:
#docker pull $(repo)/$(image):latest
build: pull
#sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$(image)/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$(image):custom .
#rm -f Dockerfile
node: image=node
node: | pull build
jdk8: image=jdk8
jdk8: | pull build
all: | node jdk8
TLDR
It is used to:
Pull the latest docker image
Run a generically designed Dockerfile against it to customize it
Tag it as :custom for internal use
Pretty handy for customizing images in a generic manner without managing many Dockerfiles.
Dockerfile template (Dockerfile.in), incase interested:
FROM {{repo}}/{{image}}:latest
... super secret sauce
UPDATE (ANSWER)
Thanks to #G.M., ended up with:
IMAGE_NAMES := node jdk8
TARGETS := $(patsubst %,build-%,$(IMAGE_NAMES))
repo=rippio
all: $(TARGETS)
build-%: pull-%
#$sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$*/' Dockerfile.in > Dockerfile-$* && \
$docker build -f=Dockerfile-$* -t=$(repo)/$*:custom .
#rm -f Dockerfile-$*
pull-%:
#$docker pull $(repo)/$*:latest
Which allows for:
Easy upkeep of 'all' targets, which constantly grows
Running parallel via make -j (note the Dockerfile-$* file pattern)
Much more beautiful than before
If you draw your dependency graph out long-hand you'll see that there are multiple paths from all to both pull and build -- one via each of node and jdk8. But make having reached/updated pull and build via one path will then assume that they are both up to date and, hence, not bother to update them further -- regardless of any change to target specific variables.
I think what you're trying to do (assuming I've understood correctly) might be more easily achieved using pattern rules.
IMAGE_NAMES := node jdk8
TARGETS := $(patsubst %,build-%,$(IMAGE_NAMES))
repo=rippio
all: $(TARGETS)
build-%: pull-%
#$sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$*/' Dockerfile.in > Dockerfile && \
$docker build -t=$(repo)/$*:custom .
#rm -f Dockerfile
pull-%:
#$docker pull $(repo)/$*:latest
Note: You currently have all build recipes using the same input/output file DockerFile. That will cause problems if you ever want to use parallel builds -- make -j etc. It might be wise to use the stem from the pattern rule match to uniquely identify the output file if that's possible.
Normally, if you invoke make with:
make all
and if none of pull, build, node, jdk8 are existing files, make should build pull and build. If you see only pull being made, it can be because you invoke make without specifying a goal. In this case make builds the first target it finds in the Makefile (pull in your case).
Anyway, there are several strange aspects in your Makefile: you use order-only prerequisites on what looks like phony targets and these phony targets are not declared as such.
I am not sure I fully understand what you are trying to do but maybe something like this would be a good starting point:
repo=rippio
image=default
.PHONY: all build node jdk8
all: node jdk8
node: image = node
jdk8: image = jdk8
build node jdk8:
#docker pull $(repo)/$(image):latest && \
sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$(image)/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$(image):custom . && \
rm -f Dockerfile
Note: if, instead of build you name the default target default you could even simplify further with:
repo=rippio
.PHONY: all default node jdk8
all: node jdk8
default node jdk8:
#docker pull $(repo)/$#:latest && \
sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$#/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$#:custom . && \
rm -f Dockerfile
I inherited a project that has multiple top level Makefiles, one for each very-similar platform that the firmware image runs on. They have names like:
Makefile.apple.mak
Makefile.banana.mak
...
So every time I want to build a specific image, I run the command
Make -f Makefile.apple.mak // default is release
Much of the code is shared between the different images that get built. A problem with doing this is that sometimes when a change is made that compiles and works for the apple platform, it breaks the banana platform. There is a whole mess of #ifdefs that conditionally include or exclude code based on the platform specified by the Makefile. I know, it's a mess and badly in need of a refactor (did I mention I inherited this?)
In order to easily build all of the firmware images after making a change and declaring it good on one platform, I want to have a wrapping Makefile that invokes a build of one or all of the available firmware images.
I am currently looking at this brute-force approach. I know there's got to be a more efficient way to do this. I also don't know offhand how to pass additional arguments to this outer Makefile (i.e. make apple debug) if that is possible. Any help is appreciated.
.PHONY: all
all: cleanall releaseall
.PHONY: cleanall
cleanall:
make -f Makefile.apple.mak clean # Can I instead call apple target with an arg?
make -f Makefile.banana.mak clean # Can I instead call banana target with an arg?
.PHONY: releaseall
releaseall:
make -f Makefile.apple.mak
make -f Makefile.banana.mak
.PHONY: apple
apple:
make -f Makefile.apple.mak # how do I pass in the <clean | debug | release> args?
.PHONY: banana
banana:
make -f Makefile.banana.mak # how do I pass in ...
(BTW, you never want to use make to invoke a recursive make. Always use $(MAKE))
The way to do this is construct target names encoding the image and the target for that image you want to build. Then you can use make prerequisites.
So, for example, you can do something like this:
IMAGES := apple banana ...
all: cleanall releaseall
cleanall: $(IMAGES:%=%.clean)
releaseall: $(IMAGES)
$(IMAGES):
$(MAKE) -f Makefile.$#.mak
$(IMAGES:%=%.clean):
$(MAKE) -f Makefile.$(basename $#).mak clean
.PHONY: all cleanall releaseall $(IMAGES) $(IMAGES:%=%.clean)
This also lets you run make apple.clean to clean just the apple directory, or whatever.
You can add more targets like debug, etc. as well.