I am trying to create a docker pull in my Makefile using the following script:
ARCH=amd64
IMAGE=k8s.gcr.io/debian-base
TAG=v1.0.0
all:
docker pull $(shell echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:$(TAG)~g");
When I run make command, I am able to pull the image:
# make
docker pull k8s.gcr.io/debian-base-amd64:v1.0.0;
v1.0.0: Pulling from debian-base-amd64
39fafc05754f: Pull complete
Digest: sha256:5f25d97ece9076612b64bb551e12f1e39a520176b684e2d663ce1bd53c5d0618
Status: Downloaded newer image for k8s.gcr.io/debian-base-amd64:v1.0.0
I would like to add a little caveat to it. Where I want to create manifest for multiple arch images with multiple tags, something like:
ARCH=amd64 ppc64le
IMAGE=k8s.gcr.io/debian-base
export DOCKER_CLI_EXPERIMENTAL=enabled
all:
for tag in v1.0.0 v1.0.1 ; do \
docker manifest create $(IMAGE):$$tag $(shell echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:$$tag~g"); \
done
This unfortunately fails with the error:
# make
for tag in v1.0.0 v1.0.1 ; do \
docker manifest create k8s.gcr.io/debian-base:$tag k8s.gcr.io/debian-base-amd64: k8s.gcr.io/debian-base-ppc64le:; \
done
invalid reference format
invalid reference format
make: *** [Makefile:6: all] Error 1
If you see above the tag field doesn't get populated properly. Is this possible to do in Makefile? Should I be doing it some other way or some modification required here? TIA.
You appear to have an order-of-execution problem. You expect make to expand the $(shell) function each time execution of the loop in the rule's recipe reaches the point where the $(shell) reference appears lexically, but if you think about it, that cannot work. make hands off each command in the recipe to the shell for it to execute, at which point it is out of make's hands. Therefore, it must (and does) expand make function calls before passing the command to the shell.
The whole for loop in your recipe is and must be a single command for this purpose (that's what the trailing backslashes are about), so the $(shell) function is expanded before the loop variable is ever set. Within that command, $tag expands to nothing.
And aside from execution order, the execution of the code in the $(shell) function happens in its own shell, where the $tag variable wouldn't be set anyway.
There are several good alternatives:
Alternative 1: Get rid of the loop
You have multiple things you want to build. Great! That's right in make's wheelhouse. Let make help you. For example:
# Note: make syntax permits whitespace around the "=" in variable assignments
ARCH = amd64 ppc64le
IMAGE = k8s.gcr.io/debian-base
export DOCKER_CLI_EXPERIMENTAL = enabled
TAGS = v1.0.0 v1.0.1
all: $(TAGS)
$(TAGS):
docker manifest create $(IMAGE):$# $(shell echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:$#~g")
.PHONY: $(TAGS)
This uses the fact that a rule with multiple targets is applied separately for building each of those targets. It does not require an explicit iteration variable because within any make recipe, the automatic variable $# expands to the name of the target presently being built.
The $(shell) function call herein is still expanded before the command runs, but that is not a problem because the make variables within, including $#, are expanded first.
Alternative 2: Use shell command substitution instead of make's $(shell) function
It's honestly pretty obtuse to use $(shell) inside a recipe unless to intentionally make use of the order of execution properties attending that, because the shell feature on which that make function is modeled is almost always a simpler and more appropriate choice. For example:
ARCH = amd64 ppc64le
IMAGE = k8s.gcr.io/debian-base
export DOCKER_CLI_EXPERIMENTAL = enabled
all:
for tag in v1.0.0 v1.0.1 ; do \
docker manifest create $(IMAGE):$$tag $$(echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:$$tag~g"); \
done
After expansions, the command that make passes to the shell in that case is equivalent to*
for tag in v1.0.0 v1.0.1 ; do \
docker manifest create k8s.gcr.io/debian-base:$tag $(echo amd64 ppc64le | sed -e "s~[^ ]*~k8s.gcr.io/debian-base\-&:$tag~g"); \
done
In shell code, the construct $(any command) is called a "command substitution". The command inside the parentheses is executed, and its standard output is captured and substituted. Using this leaves no question about order of execution.
Alternative 3: both of the above
There's not much more to say than that it comes out like this:
ARCH = amd64 ppc64le
IMAGE = k8s.gcr.io/debian-base
export DOCKER_CLI_EXPERIMENTAL = enabled
TAGS = v1.0.0 v1.0.1
all: $(TAGS)
$(TAGS):
docker manifest create $(IMAGE):$# $$(echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:$#~g")
.PHONY: $(TAGS)
And, I guess, that this is the alternative I like best so far. I don't particularly care for make functions generally, as they are a GNU extension, and many of them tend to produce a blurred programming paradigm. Or maybe "a programming paradigm" would be better wording, as I don't usually think of writing makefiles as "programming" per se.
Alternative 4: also improve the command substitution command
sed is a bit overkill for just appending a string to multiple other strings, and its expression syntax is a bit arcane. I'm actually very fond of sed, but for use in a makefile I value clarity very highly. For that reason, something along these lines is probably what I would do myself:
ARCH = amd64 ppc64le
IMAGE = k8s.gcr.io/debian-base
export DOCKER_CLI_EXPERIMENTAL = enabled
TAGS = v1.0.0 v1.0.1
all: $(TAGS)
$(TAGS):
docker manifest create $(IMAGE):$# $$(for arch in $(ARCH); do echo "$(IMAGE)-$$arch:$#"; done)
.PHONY: $(TAGS)
* equivalent, but not identical, because make will perform the line joining instead of leaving that for the shell. I present the line-split version instead for easier reading.
Your Makefile should work fine if you add \ in front of the second $$tag. This is because the contents in $(shell is passed to a shell twice (once with $(shell call and once in docker manifest ... call).
ARCH=amd64 ppc64le
IMAGE=k8s.gcr.io/debian-base
export DOCKER_CLI_EXPERIMENTAL=enabled
all:
for tag in v1.0.0 v1.0.1 ; do \
docker manifest create $(IMAGE):$$tag $(shell echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:\$$tag~g"); \
done
Perhaps letting make to build up names would be simpler?
ARCH=amd64 ppc64le
IMAGE=k8s.gcr.io/debian-base
all:
for tag in v1.0.0 v1.0.1 ; do \
docker manifest create $(IMAGE):$$tag $(foreach a,$(ARCH),$(IMAGE)-$a:$$tag); \
done
Related
Considering that every command is run in its own shell, what is the best way to run a multi-line bash command in a makefile? For example, like this:
for i in `find`
do
all="$all $i"
done
gcc $all
You can use backslash for line continuation. However note that the shell receives the whole command concatenated into a single line, so you also need to terminate some of the lines with a semicolon:
foo:
for i in `find`; \
do \
all="$$all $$i"; \
done; \
gcc $$all
But if you just want to take the whole list returned by the find invocation and pass it to gcc, you actually don't necessarily need a multiline command:
foo:
gcc `find`
Or, using a more shell-conventional $(command) approach (notice the $ escaping though):
foo:
gcc $$(find)
As indicated in the question, every sub-command is run in its own shell. This makes writing non-trivial shell scripts a little bit messy -- but it is possible! The solution is to consolidate your script into what make will consider a single sub-command (a single line).
Tips for writing shell scripts within makefiles:
Escape the script's use of $ by replacing with $$
Convert the script to work as a single line by inserting ; between commands
If you want to write the script on multiple lines, escape end-of-line with \
Optionally start with set -e to match make's provision to abort on sub-command failure
This is totally optional, but you could bracket the script with () or {} to emphasize the cohesiveness of a multiple line sequence -- that this is not a typical makefile command sequence
Here's an example inspired by the OP:
mytarget:
{ \
set -e ;\
msg="header:" ;\
for i in $$(seq 1 3) ; do msg="$$msg pre_$${i}_post" ; done ;\
msg="$$msg :footer" ;\
echo msg=$$msg ;\
}
The ONESHELL directive allows to write multiple line recipes to be executed in the same shell invocation.
all: foo
SOURCE_FILES = $(shell find . -name '*.c')
.ONESHELL:
foo: ${SOURCE_FILES}
FILES=()
for F in $^; do
FILES+=($${F})
done
gcc "$${FILES[#]}" -o $#
There is a drawback though : special prefix characters (‘#’, ‘-’, and ‘+’) are interpreted differently.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html
Of course, the proper way to write a Makefile is to actually document which targets depend on which sources. In the trivial case, the proposed solution will make foo depend on itself, but of course, make is smart enough to drop a circular dependency. But if you add a temporary file to your directory, it will "magically" become part of the dependency chain. Better to create an explicit list of dependencies once and for all, perhaps via a script.
GNU make knows how to run gcc to produce an executable out of a set of .c and .h files, so maybe all you really need amounts to
foo: $(wildcard *.h) $(wildcard *.c)
What's wrong with just invoking the commands?
foo:
echo line1
echo line2
....
And for your second question, you need to escape the $ by using $$ instead, i.e. bash -c '... echo $$a ...'.
EDIT: Your example could be rewritten to a single line script like this:
gcc $(for i in `find`; do echo $i; done)
This question already has answers here:
Define make variable at rule execution time
(4 answers)
Closed 4 years ago.
How can one use the variable defined inside a make target
.PHONY: foo
VAR_GLOBAL=$(shell cat /tmp/global)
foo:
echo "local" > /tmp/local
VAR_LOCAL=$(shell cat /tmp/local)
echo ${VAR_GLOBAL}
echo ${VAR_LOCAL}
here is the execution output:
$ echo global > /tmp/global
$ make foo
echo "local" > /tmp/local
VAR_LOCAL=local
echo global
global
echo
As #KelvinSherlock pointed out this is a duplicate of another question
here is the specific solution for my question:
.PHONY: foo
VAR_GLOBAL=$(shell cat /tmp/global)
foo:
echo "local" > /tmp/local
$(eval VAR_LOCAL := $(shell cat /tmp/local))
echo ${VAR_GLOBAL}
echo ${VAR_LOCAL}
You probably want to use the override directive in a target-specific variable assignment, so try:
foo: override LS_LOCAL=$(shell ls /var | tail -1)
echo ${LS_GLOBAL}
echo ${LS_LOCAL}
If LS_LOCAL is never defined (even by builtin-rules) you might not need the override keyword.
BTW, you might avoid $(shell ls /var | tail -1) by using the wildcard function combined with the lastword function (perhaps combined with notdir function), so you might code $(lastword $(wildcard /var/*)) or $(notdir $(lastword $(wildcard /var/*))) instead . However, beware of the order of expansion, and of filenames with spaces. At last the shell function probably uses your $PATH variable (so strange things could happen if some weird ls program appears there before /bin/ls). Perhaps using $(shell /bin/ls /var | /usr/bin/tail -1) might be better.
Look also into Guile-extended make; consider perhaps some other build-automation tool like ninja and/or generating your Makefile (or other build configuration) with something like a configure script generated via autoconf or cmake.
Notice also that a command in recipe can be made of several physical backslashed lines (hence executed in the same shell). Maybe you might consider something like
export MY_VAR=$$(ls /var | tail); \
dosomething; \
use $$MY_VAR
inside some recipe.
As part of my makefile I need to download and build ZLib. However I want to ensure that when I download ZLib, it is correct by comparing the sha256 of the downloaded .tar.gz against the known correct sha256 value. This need to work on multiple platforms.
I have so far something like the following, however the value of ZLIB_SHA256_ACTUAL always seems to be blank when I compare it with ZLIB_SHA256, so my makefile always exits with an error because the checksums are not the same. I am newish to Makefiles, can someone tell me what I am doing wrong please?
ZLIB_VER = 1.2.11
ZLIB_SHA256 = c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1
SHA256_CMD = sha256sum
ifeq ($(PLATFORM), OS_MACOSX)
SHA256_CMD = openssl sha256 -r
endif
ifeq ($(PLATFORM), OS_SOLARIS)
SHA256_CMD = digest -a sha256
endif
libz.a:
-rm -rf zlib-$(ZLIB_VER)
curl -O -L http://zlib.net/zlib-$(ZLIB_VER).tar.gz
ZLIB_SHA256_ACTUAL = $(SHA256_CMD) zlib-$(ZLIB_VER).tar.gz
ifneq ($(ZLIB_SHA256), $(ZLIB_SHA256_ACTUAL))
$(error zlib-$(ZLIB_VER).tar.gz checksum mismatch, expected="$(ZLIB_SHA256)" actual="$(ZLIB_SHA256_ACTUAL)")
endif
tar xvzf zlib-$(ZLIB_VER).tar.gz
cd zlib-$(ZLIB_VER) && CFLAGS='-fPIC' ./configure --static && make
cp zlib-$(ZLIB_VER)/libz.a .
A makefile consists of two different programming languages in one file. Most of the file uses makefile syntax, that make understands and parses. But the recipes of the rules use shell syntax, which make doesn't try to interpret: it just passes the contents of the recipe to the shell to interpret.
The recipe is the part of the makefile indented with a TAB character, after a target definition. So in your example above, the target definition is libz.a: and all the lines after that which are indented with a TAB, are recipe lines. They are passed to the shell, not run by make.
The recipe is a single block of lines; you cannot intersperse recipe lines with makefile lines. Once make sees the first non-recipe line, that's the end of the recipe and make starts treating the remaining lines as if they were makefile lines.
Let's look at your rule:
libz.a:
-rm -rf zlib-$(ZLIB_VER)
curl -O -L http://zlib.net/zlib-$(ZLIB_VER).tar.gz
OK, this is fine: you've created a target libz.a and provided two command lines, which are valid shell commands, in your recipe.
ZLIB_SHA256_ACTUAL = $(SHA256_CMD) zlib-$(ZLIB_VER).tar.gz
OK, now you have problems; this is a make variable assignment, not a shell command, but since you've indented it with a TAB make will not interpret it: make will just pass it to the shell. That's not a valid shell command (in the shell, variable assignments cannot have spaces around the equal sign); this is trying to run a program named literally ZLIB_SHA256_ACTUAL and pass it the arguments = and the expansion of the SHA256_CMD variable. Even if this was recognized as a make assignment it wouldn't do what you want since it would just set the value of the variable to the string openssl sha256 -r zlib-1.2.11.tar.gz: you want to run that command and set the variable to the output.
Then the next lines:
ifneq ($(ZLIB_SHA256), $(ZLIB_SHA256_ACTUAL))
$(error zlib-$(ZLIB_VER).tar.gz checksum mismatch, expected="$(ZLIB_SHA256)" actual="$(ZLIB_SHA256_ACTUAL)")
endif
Again, this is wrong because these are make commands but you've put them into a recipe which means they'll be passed to the shell, but the shell doesn't know anything about them.
However, they never get the chance to be passed to the shell because the one thing make does with a recipe before it sends it off to the shell is expand all make variables and functions. So, when make expands this it runs the error function and that immediately fails and make never has a chance to try to run the recipe.
This is the tricky part of make. Maybe I've just confused you with all of the above stuff.
The short, simple answer is: you have to use shell commands to perform operations in a recipe. You cannot use make commands (like ifeq etc.), and if you want to set variables in a recipe they have to be shell variables, not make variables.
So, you want something like this, which uses shell syntax not make syntax for the variable assignment and test.
EDIT Note your SHA generation command doesn't print just the SHA it also prints the name of the file, so you can't compare them as strings: they'll never be the same. You need to do something fancier; there are many ways to go about it. Here I decided to use case to do the comparison:
libz.a:
-rm -rf zlib-$(ZLIB_VER)
curl -O -L http://zlib.net/zlib-$(ZLIB_VER).tar.gz
ZLIB_SHA256_ACTUAL=`$(SHA256_CMD) zlib-$(ZLIB_VER).tar.gz`; \
case "$$ZLIB_SHA256_ACTUAL " in \
($(ZLIB_SHA256)\ *) : ok ;; \
(*) echo zlib-$(ZLIB_VER).tar.gz checksum mismatch, expected=\"$(ZLIB_SHA256)\" actual=\"$$ZLIB_SHA256_ACTUAL\"; \
exit 1 ;; \
esac
tar xvzf zlib-$(ZLIB_VER).tar.gz
cd zlib-$(ZLIB_VER) && CFLAGS='-fPIC' ./configure --static && $(MAKE)
cp zlib-$(ZLIB_VER)/libz.a .
Note that each logical line in the recipe is passed to a new instance of the shell, so if you want to set a shell variable and test its value you have to combine physical lines into one logical line with the backslash/newline syntax.
Also, when running a sub-make in a recipe you should always use the variable $(MAKE) and never use just make.
Considering that every command is run in its own shell, what is the best way to run a multi-line bash command in a makefile? For example, like this:
for i in `find`
do
all="$all $i"
done
gcc $all
You can use backslash for line continuation. However note that the shell receives the whole command concatenated into a single line, so you also need to terminate some of the lines with a semicolon:
foo:
for i in `find`; \
do \
all="$$all $$i"; \
done; \
gcc $$all
But if you just want to take the whole list returned by the find invocation and pass it to gcc, you actually don't necessarily need a multiline command:
foo:
gcc `find`
Or, using a more shell-conventional $(command) approach (notice the $ escaping though):
foo:
gcc $$(find)
As indicated in the question, every sub-command is run in its own shell. This makes writing non-trivial shell scripts a little bit messy -- but it is possible! The solution is to consolidate your script into what make will consider a single sub-command (a single line).
Tips for writing shell scripts within makefiles:
Escape the script's use of $ by replacing with $$
Convert the script to work as a single line by inserting ; between commands
If you want to write the script on multiple lines, escape end-of-line with \
Optionally start with set -e to match make's provision to abort on sub-command failure
This is totally optional, but you could bracket the script with () or {} to emphasize the cohesiveness of a multiple line sequence -- that this is not a typical makefile command sequence
Here's an example inspired by the OP:
mytarget:
{ \
set -e ;\
msg="header:" ;\
for i in $$(seq 1 3) ; do msg="$$msg pre_$${i}_post" ; done ;\
msg="$$msg :footer" ;\
echo msg=$$msg ;\
}
The ONESHELL directive allows to write multiple line recipes to be executed in the same shell invocation.
all: foo
SOURCE_FILES = $(shell find . -name '*.c')
.ONESHELL:
foo: ${SOURCE_FILES}
FILES=()
for F in $^; do
FILES+=($${F})
done
gcc "$${FILES[#]}" -o $#
There is a drawback though : special prefix characters (‘#’, ‘-’, and ‘+’) are interpreted differently.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html
Of course, the proper way to write a Makefile is to actually document which targets depend on which sources. In the trivial case, the proposed solution will make foo depend on itself, but of course, make is smart enough to drop a circular dependency. But if you add a temporary file to your directory, it will "magically" become part of the dependency chain. Better to create an explicit list of dependencies once and for all, perhaps via a script.
GNU make knows how to run gcc to produce an executable out of a set of .c and .h files, so maybe all you really need amounts to
foo: $(wildcard *.h) $(wildcard *.c)
What's wrong with just invoking the commands?
foo:
echo line1
echo line2
....
And for your second question, you need to escape the $ by using $$ instead, i.e. bash -c '... echo $$a ...'.
EDIT: Your example could be rewritten to a single line script like this:
gcc $(for i in `find`; do echo $i; done)
This question already has answers here:
How to use shell commands in Makefile
(2 answers)
Closed 9 months ago.
I would like to use a loop to find some files and rename them:
for i in `find $# -name *_cu.*`;do mv $i "$(echo $i|sed s/_cu//)"
done
This works in the shell. But how can I do this in a makefile recipe?
There are two main things you need to know when putting non-trivial shell fragments into make recipes:
Commands in the recipe are (of course!) executed one at a time, where command means "tab-prefixed line in the recipe", possibly spread over several makefile lines with backslashes.
So your shell fragment has to be written all on one (possibly backslashed) line. Moreover it's effectively presented to the shell as a single line (the backslashed-newlines are not plain newlines so are not used as command terminators by the shell), so must be syntactically correct as such.
Both shell variables and make variables are introduced by dollar signs ($#, $i), so you need to hide your shell variables from make by writing them as $$i. (More precisely, any dollar sign you want to be seen by the shell must be escaped from make by writing it as $$.)
Normally in a shell script you would write separate commands on separate lines, but here you effectively only get a single line so must separate the individual shell commands with semicolons instead. Putting all this together for your example produces:
foo: bar
for i in `find $# -name *_cu.*`; do mv $$i "$$(echo $$i|sed s/_cu//)"; done
or equivalently:
foo: bar
for i in `find $# -name *_cu.*`; do \
mv $$i "$$(echo $$i|sed s/_cu//)"; \
done
Notice that the latter, even though it's laid out readably on several lines, requires the same careful use of semicolons to keep the shell happy.
I found this useful, trying to use for loops to build multiple files:
PROGRAMS = foo bar other
.PHONY all
all: $(PROGRAMS)
$(PROGRAMS):
gcc -o $# $#.c
It will compile foo.c, bar.c, other.c into foor bar other executables
I spend good time on this and finally had it working. I had an easy solution using the global variable in makefile available for all targets, however I don`t want that so this is how I did it.
target:
$(eval test_cont=$(shell sh -c "docker ps | grep test" | awk '{print $$1}'))
for container in $(test_cont);do \
docker cp ssh/id_rsa.pub $${container}:/root/.ssh/authorized_keys; \
docker exec -it $${container} chown root.root /root/.ssh/authorized_keys; \
done