Docker build fails when executed within Makefile - bash

So, I have a Docker build command that I have tested which works great
docker build \
-t app \
--no-cache --network host \
--build argssh_private_key="$(cat ~/.ssh/id_rsa)"\
--build-arg python_version="3.6.8" -f Dockerfile .
To ease the pain of the team learning Docker I encapsulated a few of the commands - build, start, stop - within a Makefile. However, within the Makefile I need to change the command slightly by modifying
$(cat ~/.ssh/id_rsa)
to
$(shell cat ~/.ssh/id_rsa)
When I execute the following:
make build
I receive the following message:
Step 13/20 : RUN git clone --depth 1 "${git_user}#${git_host}:${git_repo}" app
---> Running in d2eb41a71315
Cloning into 'app'...
Warning: Permanently added the ECDSA host key for IP address [ip_address] to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
However, I do not have the same issue when executing from the command-line. I I think it has something to do with the way the call the "cat" command but, I do not know a way to resolve.
Any ideas ?
Makefile:
APP_NAME=ccs_data_pipeline
DATA?="${HOME}/data"
DOCKER_FILE=Dockerfile
PYTHON_VERSION?=3.6.8
SRC?=$(shell dirname `pwd`)
PRIVATE_KEY?=$(shell echo $(shell cat ~/.ssh/id_rsa))
build: ## Build container for ccs data pipeline
docker build \
-t $(APP_NAME) \
--no-cache --network host \
--build-arg ssh_private_key="$(PRIVATE_KEY)" \
--build-arg python_version="$(PYTHON_VERSION)" \
-f $(DOCKER_FILE) .
start: ## Start the docker container
docker run \
-it -v $(DATA):/data \
--network host \
--rm \
--name="$(APP_NAME)" $(APP_NAME)
stop: ## Stop the docker container
docker stop $(APP_NAME); \
docker rm $(APP_NAME)

Please show your actual makefile, or at least the entire rule that is having the error. The single command you provided, with no context, is not enough to understand what you're doing or what might be wrong.
Note that it is often not correct to replace a shell operation like $(...) with a make shell command $(shell ...). However, sometimes it will work "by accident", where the real differences between those commands don't happen to matter.
In general you should never use $(shell ...) inside a recipe (I have no idea if this command appears in a recipe). Instead, you should escape all the dollar signs that you want to be passed verbatim to the shell when it runs your recipe:
$$(cat ~/.ssh/id_rsa)

Related

Docker behaving odd with bash environment vars

I know technically host networking isn't supported MacOS (see https://docs.docker.com/network/host/)
The host networking driver only works on Linux hosts, and is not
supported on Docker Desktop for Mac, Docker Desktop for Windows, or
Docker EE for Windows Server.
However it does actually seem to work. E.g. this works just fine:
docker run \
--name local-mysql \
-e MYSQL_ROOT_PASSWORD=foo \
-e MYSQL_DATABASE=baz \
--network="host" \
-d mysql:latest
However when I try to conditionally specify the host networking with a bash variable, it doesn't work, and I can't make sense of it. Consider the following test.sh:
#!/bin/bash
echo "Test 1"
docker rm -f local-mysql
docker run \
--name local-mysql \
-e MYSQL_ROOT_PASSWORD=foo \
-e MYSQL_USER=master \
-e MYSQL_PASSWORD=bar \
-e MYSQL_DATABASE=baz \
--network="host" \
-d mysql:latest
docker ps
sleep 5
echo "Test 2"
export NETWORKING='--network="host"'
docker rm -f local-mysql
docker run \
--name local-mysql \
-e MYSQL_ROOT_PASSWORD=foo \
-e MYSQL_USER=master \
-e MYSQL_PASSWORD=bar \
-e MYSQL_DATABASE=baz \
${NETWORKING} \
-d mysql:latest
docker ps
This yields:
% ./test.sh
Test 1
local-mysql
6bbd68f0564943b8fb66ed37f1e639b54719bdb3b88b4e13aeef0a11cae4090b
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6bbd68f05649 mysql:latest "docker-entrypoint.s…" Less than a second ago Up Less than a second local-mysql
Test 2
local-mysql
e286028ef9a1a27f4226beb60e766cc163c289239ba506f63a71a35adbc73ef3
docker: Error response from daemon: network "host" not found.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
I.e. when I hard code --network=host into the docker command, the container starts fine. But the exact same parameter in an environment variable fails to start with network "host" not found.
I'm honestly not sure if this is a failure of bash or docker, but I can't actually figure out what's going wrong.
-- EDIT --
Changing
export NETWORKING='--network="host"'
to
export NETWORKING='--network=host'
works. And for my purposes right now that's enough. But just to be thorough... Why? The working example has quotes in the value (--network="host"), so why does the shell expansion break the non-working example? What if I wanted something like --network="my host"?

Port forwarding through nested docker containers on Jenkins

My Jenkins pipeline uses the docker plugin that then runs a docker container from inside of that to set up a general test environment like this:
node('docker') {
sh """
cat > .Dockerfile.build <<EOF
FROM ruby:$rubyVersion
RUN apt-get update && apt-get install -y locales && localedef -i en_US -f UTF-8 en_US.UTF-8
ENV LANG=en_US.UTF-8 \\
LANGUAGE=en_US:en \\
LC_LANG=en_US.UTF-8 \\
LC_ALL=en_US.UTF-8
RUN \\
curl -sSL -o /tmp/docker.tgz https://download.docker.com/linux/static/stable/x86_64/docker-${dockerVersion}.tgz && \\
tar --strip-components 1 --directory /usr/local/bin/ --extract --file /tmp/docker.tgz
RUN \\
groupadd -g $gid docker && \\
useradd -d $env.HOME -u $uid build -r -m && \\
usermod -a -G docker build
EOF
""".stripIndent().trim()
}
Once the test environment container is up, I run another container that has my code and tests inside that previously made environment container. One of my tests includes making sure a firewall was set up through iptables that allow certain ports through. To test to see if my firewall is setup correctly, I simple run this from inside that container (now 3 docker containers deep):
def listener_response(port, host = 'localhost')
TCPSocket.open(host, port) do |socket|
socket.read(2)
end
rescue SystemCallError
nil
end
This is called by simply passing in the random port I used and the Jenkins docker node IP. When I run my test container, I do something like:
docker run -d -e DOCKER_HOST_IP=10.x.x.x -e RANDOM_OPEN_PORT=52459 -p 52459:52459 -v /var/run/docker.sock:/var/run/docker.sock
However, I still get a nil response from my test rather than an OK. Is there a way to port forward from the Jenkins host to my test environment to my test container?
Running the test environment with the option --network host seemed to solve the problem for me.

Makefile and docker attach. How to execute from other Makefile's task?

I have a Makefile:
attach:
docker run --rm \
-it \
alpine:3.8
run-inside: attach
uname -a
I can run make attach and type uname -a by hands
uname -a, is just for example
I want run-inside to run container, attach to it, execute command and stop container. Is it possible to do this? I need this because i'm setting up CI for my project, and i need to know how to run without copy/paste
I know i can do this:
run:
docker run --rm \
alpine:3.8 uname -a
But this way i'm duplicating docker command
A possible solution consists in using the -d (--detach) option of docker run, along with the docker exec command.
For example:
Makefile
IMAGE ?= alpine:3.8
NAME ?= foobar
RUN = docker exec $(NAME)
all: start run-inside stop
start:
docker run -d -i --name=$(NAME) --rm --init $(IMAGE)
run-inside:
$(RUN) cat /etc/os-release
$(RUN) uname -a
stop:
docker stop $(NAME)
.PHONY: all start run-inside stop
Regarding the options passed to docker run:
-d tells the Docker Engine to run the container in the background;
-i is necessary to keep the container running (while -t is useless here);
--name specifies the container's name;
--rm triggers the container's removal as soon as it is stopped (here, with docker stop);
--init is optional (it is especially handy when the entrypoint is a shell, so that the signal sent by docker stop can be processed immediately by the tini process, run as PID 1).
As an aside, relying on a Makefile is maybe unnecessary when configuring a Docker-based CI: it can work well but you might instead want to:
inline the docker commands at stake directly in a .travis.yml or .gitlab-ci.yml or so;
use a docker-compose.yml file and install docker-compose beforehand.

Cant't build Jenkins latest within Docker

******** UPDATE *********
Bash script has no errors, checked with https://www.shellcheck.net/
Adding to the Dockerfilethe line
RUN tty | sed -e "s:/dev/::"
Outputs:
No tty
Next line on Dockerfile always fails:
ENTRYPOINT ["/usr/local/bin/jenkins.sh"]
I leave an image in order to clarify. In short, I think I need to attach a tty in some way to the batch script, but dunno how to do it.
Thanks
------------------- OLD CONTENT -------------------
I need to update a Jenkins image to 2.138.2. An excerpt of the original Dockerfile is as follows:
FROM openjdk:8-jdk
RUN apt-get update && apt-get install -y git curl && rm -rf /var/lib/apt/lists/*
# ...
# Use tini as subreaper in Docker container to adopt zombie processes
COPY tini_pub.gpg ${JENKINS_HOME}/tini_pub.gpg
RUN curl -fsSL https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static-$(dpkg --print-architecture) -o /sbin/tini \
&& curl -fsSL https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static-$(dpkg --print-architecture).asc -o /sbin/tini.asc \
&& gpg --import ${JENKINS_HOME}/tini_pub.gpg \
&& gpg --verify /sbin/tini.asc \
&& rm -rf /sbin/tini.asc /root/.gnupg \
# ...
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/jenkins.sh"]
Using this Dockerfile FAILS due to gpg --import statement now needs to be fixed using --no-tty option. So that line remains as follows:
&& gpg --no-tty --import ${JENKINS_HOME}/tini_pub.gpg \
That's not fine since the execution of jenkins.sh now fails in several ways. The code of the script starts as follows:
#! /bin/bash -e
: "${JENKINS_WAR:="/usr/share/jenkins/jenkins.sh
This script is called from the Dockerfile in this line:
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/jenkins.sh"]
But now fails with several errors and seems to be impossible to process the file, nor removing the shebang line, nor removing the '-' or the '-e' option. The rest of the file is not processed fine if we change bash to other shell (not odd) nor removing the -e option (if I do that, the entrypoint does not find the jenkins.sh script).
Sumarizing, I've needed to remove a tty from gpg but doing that, I've lost access to bash scripting.
I've checked about the applied workaround, the workaround is described here ( (if I'm right, case is number 8, gpg might write to the tty at some point):
https://lists.gnupg.org/pipermail/gnupg-users/2017-April/058162.html
Is there any way to attach a tty to the entrypointor having any settings in the script in order to allow this work fine?
Thanks.
Finally runned on a Linux VM and no problems. Running it on Windows is the problem.

Run Maven in Docker container via current user

I want to run Maven to build a project in a Docker container. First, I came up with:
docker run -it --rm
-v $HOME/.m2:/root/.m2:rw
-v $PWD:$PWD:rw
-w $PWD
maven:alpine
mvn "$#"
This builds fine, but the problem here is that all files are now written and owned by the root user. I want them to be owned by the current user, myself.
So I tried this:
docker run -it --rm
--user $(id -u):$(id -g)
-v $HOME/.m2:/root/.m2:rw
-v $PWD:$PWD:rw
-w $PWD
maven:alpine
mvn "$#"
This did not work as expected. I believe I know why: now with --user $(id -u):$(id -g), we are indeed executing as myself, but now the mapping of -v $HOME/.m2:/root/.m2:rw becomes incorrect, there is no /root in place anymore.
So let's try this:
docker run -it --rm
--user $(id -u):$(id -g)
-v $HOME:$HOME:rw
-v $PWD:$PWD:rw
-w $PWD
maven:alpine
mvn "$#"
Now I am getting the following warning:
Can not write to /root/.m2/copy_reference_file.log. Wrong volume permissions? Carrying on ...
Also, Maven seems to be able to build (although I am having problems with accessing the Docker daemon during integration tests, but that might be better suited for another question), but I don't see any artifacts appearing in ~/.m2/repository on the host? They are also not in /root/.m2/repository (which does not exist, as expected) on the host. Where are they? What am I doing wrong?
Here is described how to run maven as non-root-user:
Maven needs the user home to download artifacts to, and if the user does not exist in the image an extra user.home Java property needs to be set.
Something in that direction should work:
docker run -it --rm \
--user $(id -u):$(id -g) \
-v ~/.m2:/var/maven/.m2:rw \
-e MAVEN_CONFIG=/var/maven/.m2 \
-v $PWD:$PWD:rw \
-w $PWD \
maven:alpine \
mvn -Duser.home=/var/maven "$#"

Resources