The problem I'm experiencing is an all target has dependencies on others that set a variable, then run matching dependencies.
Outcome - It will run the first dependency then stop.
Expected - Run both dependencies, setting the variable properly between each run
Is make smart enough to see pull and build were already ran and the dependency target itself has no execution, therefore it sees all dependencies as complete? Or I'm just abusing make in ways it should not be used?
Said makefile:
repo=rippio
image=default
pull:
#docker pull $(repo)/$(image):latest
build: pull
#sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$(image)/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$(image):custom .
#rm -f Dockerfile
node: image=node
node: | pull build
jdk8: image=jdk8
jdk8: | pull build
all: | node jdk8
TLDR
It is used to:
Pull the latest docker image
Run a generically designed Dockerfile against it to customize it
Tag it as :custom for internal use
Pretty handy for customizing images in a generic manner without managing many Dockerfiles.
Dockerfile template (Dockerfile.in), incase interested:
FROM {{repo}}/{{image}}:latest
... super secret sauce
UPDATE (ANSWER)
Thanks to #G.M., ended up with:
IMAGE_NAMES := node jdk8
TARGETS := $(patsubst %,build-%,$(IMAGE_NAMES))
repo=rippio
all: $(TARGETS)
build-%: pull-%
#$sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$*/' Dockerfile.in > Dockerfile-$* && \
$docker build -f=Dockerfile-$* -t=$(repo)/$*:custom .
#rm -f Dockerfile-$*
pull-%:
#$docker pull $(repo)/$*:latest
Which allows for:
Easy upkeep of 'all' targets, which constantly grows
Running parallel via make -j (note the Dockerfile-$* file pattern)
Much more beautiful than before
If you draw your dependency graph out long-hand you'll see that there are multiple paths from all to both pull and build -- one via each of node and jdk8. But make having reached/updated pull and build via one path will then assume that they are both up to date and, hence, not bother to update them further -- regardless of any change to target specific variables.
I think what you're trying to do (assuming I've understood correctly) might be more easily achieved using pattern rules.
IMAGE_NAMES := node jdk8
TARGETS := $(patsubst %,build-%,$(IMAGE_NAMES))
repo=rippio
all: $(TARGETS)
build-%: pull-%
#$sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$*/' Dockerfile.in > Dockerfile && \
$docker build -t=$(repo)/$*:custom .
#rm -f Dockerfile
pull-%:
#$docker pull $(repo)/$*:latest
Note: You currently have all build recipes using the same input/output file DockerFile. That will cause problems if you ever want to use parallel builds -- make -j etc. It might be wise to use the stem from the pattern rule match to uniquely identify the output file if that's possible.
Normally, if you invoke make with:
make all
and if none of pull, build, node, jdk8 are existing files, make should build pull and build. If you see only pull being made, it can be because you invoke make without specifying a goal. In this case make builds the first target it finds in the Makefile (pull in your case).
Anyway, there are several strange aspects in your Makefile: you use order-only prerequisites on what looks like phony targets and these phony targets are not declared as such.
I am not sure I fully understand what you are trying to do but maybe something like this would be a good starting point:
repo=rippio
image=default
.PHONY: all build node jdk8
all: node jdk8
node: image = node
jdk8: image = jdk8
build node jdk8:
#docker pull $(repo)/$(image):latest && \
sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$(image)/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$(image):custom . && \
rm -f Dockerfile
Note: if, instead of build you name the default target default you could even simplify further with:
repo=rippio
.PHONY: all default node jdk8
all: node jdk8
default node jdk8:
#docker pull $(repo)/$#:latest && \
sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$#/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$#:custom . && \
rm -f Dockerfile
Related
I have some "phony" targets that I occasionally want to run in a specific order, but are not strictly dependent on each other. For example:
.PHONY: image-build
image-build: Dockerfile
docker build --tag foobar --file $< .
.PHONY: image-tag
image-tag: image-build
docker tag foobar foobar:latest
.PHONY: image-push
image-push:
docker push foobar:latest
I don't always want to run image-tag before image-push. However, if I do happen to run them together, as in make image-tag image-push, I want to ensure that image-tag runs before image-push.
That is, whenever image-tag and image-push both appear in the dependency graph, I want image-tag to be executed first, but when image-push is in the dependency graph and image-tag is not, I do not want image-tag to be added to the dependency graph.
Is it possible to enforce this rule in GNU Make?
For now, I have created a workaround command:
.PHONY: image-deploy
image-deploy: image-tag
$(MAKE) image-push
You can do this:
image-push: $(filter image-tag,$(MAKECMDGOALS))
docker push foobar:latest
The filter will expand to nothing if image-tag is not specified as a goal target (on the command line), else it will expand to image-tag.
Second dependency in goal first-deploy not executed.
if I execute make first-deploy, make execute only first dependence - build-proxy-base with success result and exit.
Makefile:
UID=`id -u`
GID=`id -g`
NODE_N=node
PROXY_N=proxy
PY_N=py
DOCKER_D=docker
PROXY_D=skif-proxy
NODE_D=skif
PY_D=skif-proxy/template-service
# first is default
default: first-deploy docker-compose-up
docker-compose-up:
cd $(DOCKER_D) && \
docker-compose up --build && \
cd ..
build-docker-base:
docker build \
--build-arg UID=${UID} \
--build-arg GID=${GID} \
-f ${D}/Dockerfile.base \
--rm \
-t skif-${SERV_N}-base ${D}
build-node-base: D := ${NODE_D}
build-node-base: SERV_N := ${NODE_N}
build-node-base: build-docker-base
build-proxy-base:D=${PROXY_D}
build-proxy-base:SERV_N=${PROXY_N}
build-proxy-base: build-docker-base
build-py-base: D=${PY_D}
build-py-base: SERV_N=${PY_N}
build-py-base: build-docker-base
first-deploy: build-proxy-base build-py-base build-node-base
UPD
I understand that build-docker-base build only once, but how I can reuse it code in 2 goals with other paramethers?
Make definitely considered the second dependency. If it didn't "execute" then it means make decided nothing needed to be done to execute it.
I can see this:
build-proxy-base: build-docker-base
...
build-node-base: build-docker-base
Both of these targets list the same target as a prerequisite, and they don't do anything else. In make, a given target (like build-docker-base) will only ever be built one time by make per invocation. Once make builds it once, it will be considered up to date no matter how many other targets may depend on it.
So make considers build-node-base, sees that it was already brought up to date due to build-proxy-base, and decides there's nothing else to do.
Since you don't actually show us to build-node-base rule or explain what you're trying to do, there's not much more we can say for sure.
ETA Based on your updated makefile, it seems like you're trying to think of targets as if they were functions, that can be called by listing them as prerequisites. They are not, and they cannot.
It looks to me like you just want to run the same recipe multiple times with different parameters. I think you should just write it like this:
build-node-base build-proxy-base build-py-base:
docker build \
--build-arg UID=${UID} \
--build-arg GID=${GID} \
-f ${D}/Dockerfile.base \
--rm \
-t skif-${SERV_N}-base ${D}
build-node-base: D := ${NODE_D}
build-node-base: SERV_N := ${NODE_N}
build-proxy-base:D=${PROXY_D}
build-proxy-base:SERV_N=${PROXY_N}
build-py-base: D=${PY_D}
build-py-base: SERV_N=${PY_N}
Since there is no recipe for either build-proxy-base or build-node-base, and both are effectively phony, it's unclear how you are concluding that build-node-base is not, in fact, built.
I suspect that the actual observation is that target build-docker-base is built only once per make run, and that that build reflects the values of variables D and SERV_N set while building build-proxy-base. That has little to do with whether target build-node-base is built, and a lot to do with the fact that make builds each target at most once per run.
This is a continuation of Makefile: run the same command with different arguments on different targets.
Code
COMMIT := $(shell git rev-parse HEAD)
images := base web proxy lb users api
$(images): base
#echo "Building $#"
docker build -f Dockerfile.$# --no-cache=true -t $#:$(COMMIT) .
build: $(images)
rebuild: $(images) # I want this to run with --no-cache=true
Essentially, build calls all image targets (base being the first one), and runs docker build with --no-cache=true for each one.
The problem
I would like to have a rebuild target which runs all image targets with --no-cache=false rather than --no-cache=true, without duplicating the code. I guess that the right way is to set a variable in rebuild and build whose scope would cover dependent targets like any of the images.
My question
How do I define a variable in a target whose scope covers all dependent targets?
Quite similar, like in mentioned question:
images := base web proxy lb users api
$(images):
#echo $# docker --no-cache=$(NO_CACHE)
build: NO_CACHE=true
rebuild: NO_CACHE=false
rebuild build: $(images)
You may want to set a default value for NO_CACHE in case you would like to call make base for example.
I am looking at a pre-existing, working, complex makefile for a project which will both build and deploy the code on multiple OS's.
I'm looking at some separate IDE support (Visual Studio) for the build process (i.e. half the make will already be done), so need to insert a phony target for the deploy action such that the old flow (make all) still works, including the deloy step, but that make deploy will just do the final deployment step for those using the IDE.
Not being familiar with make, I'm having difficulty seeing how/if make allows such an entry point and if so how to implement it.
The current code has:
$(BUILT_INS): git$X
$(QUIET_BUILT_IN)$(RM) $# && \
ln $< $# 2>/dev/null || \
ln -s $< $# 2>/dev/null || \
cp $< $#
whose actions are the deployment step.
So conceptually I think I need
.PHONY: deploy
$(BUILT_INS): git$X
deploy
deploy:
$(QUIET_BUILT_IN)$(RM) $# && \
ln $< $# 2>/dev/null || \
ln -s $< $# 2>/dev/null || \
cp $< $#
which is clearly not right, because the phony target can't be an action.
In summary; How to create an entry point into a makefile to do the rule's actions? (a critical desire is to avoid duplicating the action code)
Your "conceptually" solution has many more problems than using a target as a recipe (which you're right, won't work, but you could fix by using $(MAKE) deploy as the recipe to invoke a recursive make); the other thing is that this:
$(BUILT_INS):
... using $# ...
means run that recipe one time for each word in the BUILT_INS variable, and each time the automatic variable $# will be assigned to that word (the target).
Your replacement:
deploy:
... using $# ...
does an entirely different thing: it runs the recipe one time, with the value of $# set to deploy. Not going to work.
The simple answer to your question is that you just declare a new target deploy that lists the targets you want to run as prerequisites:
.PHONY: deploy
deploy: $(BUILT_INS)
Now when you run make deploy it will try to build the BUILT_INS targets, and run the install rule for each one.
However, my suspicion is that this will be a problem for you, depending on what the git$X prerequisite is... it might cause a big part of the rest of your makefile to run as well. But, you don't give any information about that so I can't say.
ETA Sure enough, git$X is a problem. So, it appears you want VS to generate your git$X file (which will be git.exe presumably). Then you want to run make deploy to copy it. The trick here is to keep make from rebuilding git$X when you use the deploy target. You can do something like this; replace the rule that builds git$X with:
.PHONY: deploy
deploy: $(BUILT_INS)
ifeq (,$(filter deploy,$(MAKECMDGOALS)))
git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS)
$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $# $(ALL_LDFLAGS) git.o \
$(BUILTIN_OBJS) $(LIBS)
endif
This creates a new target deploy which depends on the BUILT_INS targets, as I show above. Then, I enclose the rule to create git$X inside a test so that if you run make deploy that rule is not defined. Now make doesn't know how to build git$X at all, so if it doesn't exist then make deploy will fail, but if it does exist then make will copy it, without trying to rebuild it.
How do I implement the make installcheck target when building a library? That is, I have a check target that creates a test program that links to the created library, called by some script in order to check if the library code functions properly, and I wish to perform the same checks but linking to the library after it has been installed; how can I implement this?
Since I can't find any guides on this, I will present the method that I have come up with, which has been patched together from reading the GNU automake manual and some general trial and error. It might be ugly, but it works...
If the check routines for building a library involve linking the library to a test program and seeing if that program works correctly, then to installcheck the library we need only do the same thing but link the test program to the installed library rather than the locally built library.
Let's call the library alpha (so we'll be creating libalpha.so and/or libalpha.a). Supposing alpha's source code is in the file alpha.cpp in the src directory, we'll create src/Makefile.am as usual:
# src/Makefile.am
lib_LTLIBRARIES = libalpha.la
libalpha_la_SOURCES = alpha.cpp
include_HEADERS = alpha.h
The check routine involves creating a binary beta that links to alpha. The source code for beta is in the file beta.cpp in the directory tests. The automake file tests/Makefile.am looks like this:
# tests/Makefile.am
check_PROGRAMS = beta
beta_SOURCES = beta.cpp
beta_CPPFLAGS = -I$(top_srcdir)/src
beta_LDADD = $(top_builddir)/src/libalpha.la
We will create our check and installcheck routines by creating "local" targets in tests/Makefile.am like so:
# tests/Makefile.am
# ...
check-local:
# recipe to run when 'make check' is called.
installcheck-local:
# recipe to run when 'make installcheck' is called.
The check and installcheck targets conflict because use of either target prevents the other target from executing properly (one target "taints" the build tree for the other target); in order for the other target to execute properly we need to remove beta and its object files and have the target recompile and re-link as it sees fit according to its nature (installcheck to installed files, check to local files).
We can solve this issue of tainted build trees by simply running make clean in the recipe of both targets. This will clearly remove the tainted builds, but is overzealous because we don't need to rebuild whenever we run the same target again. The build tree is only tainted whenever we run the other target.
We can only solve this complication by remembering which of the two targets had been called previously, which we can do via the creation/destruction of an intermediary file (let's call it taint). The check target is tainted whenever the file taint exists, which it resolves by cleaning, rebuilding and removing taint; and the installcheck target is tainted when the file taint does not exist, which it resolves by cleaning, rebuilding and creating taint.
Our targets will take the form:
# tests/Makefile.am
# ...
check-local:
# First, check to see if the build tree is tainted and rebuild if so
test ! -f taint || $(MAKE) $(AM_MAKEFLAGS) check_rebuild
# Then, run our check tests. Substitute with your shell scripts or testsuite files as appropriate
./beta
installcheck-local:
# First, check to see if the build tree is tainted and rebuild if so
test -f taint || $(MAKE) $(AM_MAKEFLAGS) installcheck_rebuild
# Then, run our installcheck tests. Substitute with your shell scripts or testsuite files as appropriate
./beta
The target check_rebuild needs to rebuild according to how check will run, and will look like this:
# tests/Makefile.am
# ...
.PHONY: check_rebuild
check_rebuild:
$(MAKE) $(AM_MAKEFLAGS) clean # remove tainted build tree
$(MAKE) $(AM_MAKEFLAGS) beta$(EXEEXT) # rebuild beta
rm -f taint # mark build tree as untainted
The target installcheck_rebuild likewise looks like this:
# tests/Makefile.am
# ...
.PHONY: installcheck_rebuild
installcheck_rebuild:
$(MAKE) $(AM_MAKEFLAGS) clean # remove tainted build tree
$(MAKE) $(AM_MAKEFLAGS) beta$(EXEEXT) \
beta_CPPFLAGS="-I$(DESTDIR)$(includedir)" \
beta_LDADD="$(DESTDIR)$(libdir)/libalpha.la" \
beta_DEPENDENCIES="$(DESTDIR)$(libdir)/libalpha.la"
echo 1 > taint # mark build tree as untainted
Note that rebuilding beta in installcheck_rebuild now requires the overriding of the automake variables so that they point to the installed library.
Finally, we need to add the taint file to DISTCLEANFILES so that running distcheck does not fail with "files left in build directory after distclean" errors.
And that should be it. The final tests/Makefile.am should look like this:
# tests/Makefile.am
check_PROGRAMS = beta
beta_SOURCES = beta.cpp
beta_CPPFLAGS = -I$(top_srcdir)/src
beta_LDADD = $(top_builddir)/src/libalpha.la
taint_file = .taint
check-local:
test ! -f $(taint_file) || $(MAKE) $(AM_MAKEFLAGS) check_rebuild
./beta # substitute with your actual test routines
installcheck-local:
test -f $(taint_file) || $(MAKE) $(AM_MAKEFLAGS) installcheck_rebuild
./beta # substitute with your actual test routines
.PHONY: check_rebuild
check_rebuild:
$(MAKE) $(AM_MAKEFLAGS) clean
$(MAKE) $(AM_MAKEFLAGS) beta$(EXEEXT)
rm -f $(taint_file)
.PHONY: installcheck_rebuild
installcheck_rebuild:
$(MAKE) $(AM_MAKEFLAGS) clean
$(MAKE) $(AM_MAKEFLAGS) beta$(EXEEXT) \
beta_CPPFLAGS="-I$(DESTDIR)$(includedir)" \
beta_LDADD="$(DESTDIR)$(libdir)/libalpha.la" \
beta_DEPENDENCIES="$(DESTDIR)$(libdir)/libalpha.la"
echo 1 > $(taint_file)
DISTCLEANFILES = $(taint_file)
Disclaimer
This has been checked on Linux for a "standard" build, but it may not work on other build environments or if you are trying to do something "exotic". Hopefully it should, but it is not something that I have bothered to check. If there are errors, the problem will likely be a missing or misused variable in one of my targets above.
For example, the check_rebuild target has the line:
check_rebuild:
# ...
$(MAKE) $(AM_MAKEFLAGS) beta$(EXEEXT)
# ...
The variables $(AM_MAKEFLAGS) and $(EXEEXT) are a part of how automake itself creates routines to populate in the Makefiles it creates, and had I neglected them in my targets above, then it might have caused the build to fail (or at least not function as expected).
I have tried to make sure the targets that I have suggested above are likewise canonically constructed, but I may have missed something out. Your best bet in case of build errors is to open the Makefiles generated by automake itself during the build, seeing how it is creating objects and mimicking those constructs in the corresponding Makefile.am files, as I have tried to do.
The other likely issue may be in the "hackish" way I've built the beta binary in the installcheck_rebuild target. Again, your best bet in diagnosing problems will be to see how automake is doing things in the Makefiles it generates, and trying to mimic that. Otherwise, a read of the automake manual will be in order, and failing that, you will likely need the help of people more knowledgeable than me. Good luck.