using the same Dockerfile to build images with different python version - makefile

I have a python script that I'd want to run on 2 different containers running python 2.7 and python 3.6.
I want to use the same docker file to build 2 different images, with the difference being the python version(i.e one time it's FROM python:3.6 and the other is FROM python:2.7, and do it through a makefile.
Makefile
.PHONY = all clean build run
all: build run
# DOCKER TASKS
# Build the container
build: ## Build the container for tests
docker build -t myscript:python2.7 .
docker build -t myscript:python3.6 .
run: ## Run container
docker run myscript:python2.7
docker run myscript:python3.6
I have a Dockerfile for building the image:
Dockerfile
FROM python:3.6
WORKDIR /usr/local/bin
RUN pip install pytest
COPY myscript.py test_regex_script.py ./
CMD ["pytest", "test_regex_script.py"]
Since it's the same Docker file it's tricky, I was thinking to create two different Dockerfiles each with a different python version (and all the rest the same), but was wondering if there's more elegant way to do it.
thanks

You can use a build argument for it.
Your Dockerfile should look like this:
ARG VERSION=3.6
FROM python:3.6
WORKDIR /usr/local/bin
RUN pip install pytest
COPY myscript.py test_regex_script.py ./
CMD ["pytest", "test_regex_script.py"]
To build for 3.6:
docker build -t python-app:3.6 .
To build for 2.7:
docker build -t python-app:2.7 --build-arg VERSION=2.7 .
Note that 3.6 is the default version in case you don't specify one (ARG VERSION=3.6). You can also not assign a default value in which case you have to always pass a value in the build argument (--build-arg VERSION=...)

you can try updating the script mentioned above like
ARG PYTHON_VERSION=python:3.8-slim-buster
FROM python:${PYTHON_VERSION}
WORKDIR /usr/local/bin
RUN pip install pytest
COPY myscript.py test_regex_script.py ./
CMD ["pytest", "test_regex_script.py"]

Make sure ARG is at the very top of your Dockerfile.
ARG PY_VERSION=3.8
ARG NODE_VERSION=16
FROM node:$NODE_VERSION as frontend
FROM python:$PY_VERSION as backend
The FROM's can be lower in the file.
Default (python:3.8 and node:16):
docker build \
-t my-image \
-f Dockerfile \
.
Override Python (use 3.10 instead of 3.8):
docker build \
-t my-image \
-f Dockerfile \
--build-arg PY_VERSION=3.10 \
.
Override Node:
docker build \
-t my-image \
-f Dockerfile \
--build-arg NODE_VERSION=17 \
.
Override Both:
docker build \
-t my-image \
-f Dockerfile \
--build-arg PY_VERSION=3.10 \
--build-arg NODE_VERSION=17 \
.
You can also use buildx / docker buildx build:
docker buildx build \
-t my-image \
-f Dockerfile \
--build-arg PY_VERSION=3.10 \
--build-arg NODE_VERSION=17 \
.

Related

Access argument from docker build inside script

So in dockerfile I am running entrypoint:
ARG WP_IMAGE=latest
FROM wordpress:$WP_IMAGE
ARG VERSION
RUN curl -o /usr/local/bin/wp https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar \
&& chmod +x /usr/local/bin/wp
RUN apt update && apt install -y vim
ADD ./bin/ /
RUN chmod +x /*.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["apache2-foreground"]
And I have this script entrypoint.sh:
#!/bin/bash
/usr/local/bin/docker-entrypoint.sh php-fpm || /configure.sh
exec "$#"
And there is configure.sh script and inside this script I want to access this argument from Dockerfile VERSION.
This is how I build my docker docker-compose build --build-arg WP_IMAGE=latest --build-arg VERSION=7.0 && docker-compose up -d.
You can use ENV keyword in Dockerfile like:
ARG VERSION
ENV VERSION=${VERSION}
Now the script running in the image can access VERSION from the environment.
The ENV instruction sets the environment variable to the value
. The environment variables set using ENV will persist when a
container is run from the resulting image.

Go get "get" unexpected EOF

Thank you for visiting here.
First of all, I apologize for my bad English, maybe a little wrong, hope you can help me.
Then I had a little problem when deploying a new CI/CD system on k8s platform (v1.23.5+1) with Gitlab runner (14.9.0) and dind (docker:dind)
When deploying CI to Golang apps with private repositories at https://gitlab.domain.com, (I did the go env -w GOPRIVATE configuration), I had a problem with the go mod tidy command. Specifically getting the unexpected EOF error. I've tried go mod tidy -v but it doesn't seem to give any more info.
I did a lot of work to figure out the problem. Specifically, I have done wget and git clone commands with my private repository and they are still able to download successfully. I tried adding a private repository at https://gitlab.com in go.mod, they can still be retrieved without any errors.
And actually, without using my new runner, I can still git clone and go mod tidy in another vps.
All of this leaves me wondering where am I actually getting the error? Is it my gitlab or my k8s gitlab runner
This is runner output
go: downloading gitlab.domain.com/nood/fountain v0.0.12
unexpected EOF
Cleaning up project directory and file based variables
ERROR: Job failed: command terminated with exit code 1
This is my .gitlab-ci.yml
image: docker:latest
stages:
- build
- deploy
variables:
GTV_ECR_REPOSITORY_URL: repo.domain.com
PROJECT: nood
APP_NAME: backend-super-system
APP_NAME_ECR: backend-super-system
IMAGE_TAG: $GTV_ECR_REPOSITORY_URL/$PROJECT/$APP_NAME_ECR
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
services:
- name: docker:dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh", "--tls=false"]
build:
stage: build
allow_failure: false
script:
- echo "Building image."
- docker pull $IMAGE_TAG || echo "Building runtime from scratch"
- >
docker build
--cache-from $IMAGE_TAG
-t $IMAGE_TAG --network host .
- docker push $IMAGE_TAG
Dockerfile
FROM golang:alpine3.15
LABEL maintainer="NoodExe <nood.pr#gmail.com>"
WORKDIR /app
ENV BIN_DIR=/app/bin
RUN apk add --no-cache gcc build-base git
ADD . .
RUN chmod +x scripts/env.sh scripts/build.sh \
&& ./scripts/env.sh \
&& ./scripts/build.sh
# stage 2
FROM alpine:latest
WORKDIR /app
ENV BIN_DIR=/app/bin
ENV SCRIPTS_DIR=/app/scripts
ENV DATA_DIR=/app/data
# Build Args
ARG LOG_DIR=/var/log/nood
# Create log directory
RUN mkdir -p ${BIN_DIR} \
mkdir -p ${SCRIPTS_DIR} \
mkdir -p ${DATA_DIR} \
mkdir -p ${LOG_DIR} \
&& apk update \
&& addgroup -S nood \
&& adduser -S nood -G nood \
&& chown nood:nood /app \
&& chown nood:nood ${LOG_DIR}
USER nood
COPY --chown=nood:nood --from=0 ${BIN_DIR} /app
COPY --chown=nood:nood --from=0 ${DATA_DIR} ${DATA_DIR}
COPY --chown=nood:nood --from=0 ${SCRIPTS_DIR} ${SCRIPTS_DIR}
RUN chmod +x ${SCRIPTS_DIR}/startup.sh
ENTRYPOINT ["/app/scripts/startup.sh"]
scripts/env.sh
#!/bin/sh
go env -w GOPRIVATE=gitlab.domain.com/*
git config --global --add url."https://nood_deploy:rvbsosecret_Hizt97zQSn#gitlab.domain.com".insteadOf "https://gitlab.domain.com"
scripts/build.sh
#!/bin/sh
grep -v "replace\s.*=>.*" go.mod > tmpfile && mv tmpfile go.mod
go mod tidy
set -e
BIN_DIR=${BIN_DIR:-/app/bin}
mkdir -p "$BIN_DIR"
files=`ls *.go`
echo "****************************************"
echo "******** building applications **********"
echo "****************************************"
for file in $files; do
echo building $file
go build -o "$BIN_DIR"/${file%.go} $file
done
Thank you for still being here :3
This is a known issue with installing go modules from gitlab in nested locations. The issue describes several workarounds/solutions. One solution is described as follows:
create a gitlab Personal Access Token with at least read_api and read_repository scopes.
create a .netrc file:
machine gitlab.com
login yourname#gitlab.com
password yourpersonalaccesstoken
use go get --insecure to get your module
do not use the .gitconfig insteadOf workaround
For self-hosted instances of GitLab, there is also the additional option of using the go proxy, which is what I do to resolve this problem.
For additional context, see this answer to What's the proper way to "go get" a private repository?

Running docker as sibling

I am trying to run a container (hello-world) as a sibling from another container (dev).
But, container script is not able to access "Docker". I am getting
Docker not found error
Here is what I am doing: dev Dockerfile downloads the docker image Like
ENV DOCKER_VERSION=19.03.8
RUN curl -sfL -o docker.tgz
"https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_VERSION}.tgz" && \
tar -xzf docker.tgz docker/docker --strip=1 --directory /usr/local/bin && \
rm docker.tgz
RUN ["chmod","+x","./script.sh"]
ENTRYPOINT ["sh","./script.sh"]
script.sh is:
#!/bin/bash
docker run hello-world
Docker Build command:
docker build -t dev .
Docker run command:
docker run -v /var/run/docker.sock:/var/run/docker.sock <container_image>

Docker build fails when executed within Makefile

So, I have a Docker build command that I have tested which works great
docker build \
-t app \
--no-cache --network host \
--build argssh_private_key="$(cat ~/.ssh/id_rsa)"\
--build-arg python_version="3.6.8" -f Dockerfile .
To ease the pain of the team learning Docker I encapsulated a few of the commands - build, start, stop - within a Makefile. However, within the Makefile I need to change the command slightly by modifying
$(cat ~/.ssh/id_rsa)
to
$(shell cat ~/.ssh/id_rsa)
When I execute the following:
make build
I receive the following message:
Step 13/20 : RUN git clone --depth 1 "${git_user}#${git_host}:${git_repo}" app
---> Running in d2eb41a71315
Cloning into 'app'...
Warning: Permanently added the ECDSA host key for IP address [ip_address] to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
However, I do not have the same issue when executing from the command-line. I I think it has something to do with the way the call the "cat" command but, I do not know a way to resolve.
Any ideas ?
Makefile:
APP_NAME=ccs_data_pipeline
DATA?="${HOME}/data"
DOCKER_FILE=Dockerfile
PYTHON_VERSION?=3.6.8
SRC?=$(shell dirname `pwd`)
PRIVATE_KEY?=$(shell echo $(shell cat ~/.ssh/id_rsa))
build: ## Build container for ccs data pipeline
docker build \
-t $(APP_NAME) \
--no-cache --network host \
--build-arg ssh_private_key="$(PRIVATE_KEY)" \
--build-arg python_version="$(PYTHON_VERSION)" \
-f $(DOCKER_FILE) .
start: ## Start the docker container
docker run \
-it -v $(DATA):/data \
--network host \
--rm \
--name="$(APP_NAME)" $(APP_NAME)
stop: ## Stop the docker container
docker stop $(APP_NAME); \
docker rm $(APP_NAME)
Please show your actual makefile, or at least the entire rule that is having the error. The single command you provided, with no context, is not enough to understand what you're doing or what might be wrong.
Note that it is often not correct to replace a shell operation like $(...) with a make shell command $(shell ...). However, sometimes it will work "by accident", where the real differences between those commands don't happen to matter.
In general you should never use $(shell ...) inside a recipe (I have no idea if this command appears in a recipe). Instead, you should escape all the dollar signs that you want to be passed verbatim to the shell when it runs your recipe:
$$(cat ~/.ssh/id_rsa)

Run Maven in Docker container via current user

I want to run Maven to build a project in a Docker container. First, I came up with:
docker run -it --rm
-v $HOME/.m2:/root/.m2:rw
-v $PWD:$PWD:rw
-w $PWD
maven:alpine
mvn "$#"
This builds fine, but the problem here is that all files are now written and owned by the root user. I want them to be owned by the current user, myself.
So I tried this:
docker run -it --rm
--user $(id -u):$(id -g)
-v $HOME/.m2:/root/.m2:rw
-v $PWD:$PWD:rw
-w $PWD
maven:alpine
mvn "$#"
This did not work as expected. I believe I know why: now with --user $(id -u):$(id -g), we are indeed executing as myself, but now the mapping of -v $HOME/.m2:/root/.m2:rw becomes incorrect, there is no /root in place anymore.
So let's try this:
docker run -it --rm
--user $(id -u):$(id -g)
-v $HOME:$HOME:rw
-v $PWD:$PWD:rw
-w $PWD
maven:alpine
mvn "$#"
Now I am getting the following warning:
Can not write to /root/.m2/copy_reference_file.log. Wrong volume permissions? Carrying on ...
Also, Maven seems to be able to build (although I am having problems with accessing the Docker daemon during integration tests, but that might be better suited for another question), but I don't see any artifacts appearing in ~/.m2/repository on the host? They are also not in /root/.m2/repository (which does not exist, as expected) on the host. Where are they? What am I doing wrong?
Here is described how to run maven as non-root-user:
Maven needs the user home to download artifacts to, and if the user does not exist in the image an extra user.home Java property needs to be set.
Something in that direction should work:
docker run -it --rm \
--user $(id -u):$(id -g) \
-v ~/.m2:/var/maven/.m2:rw \
-e MAVEN_CONFIG=/var/maven/.m2 \
-v $PWD:$PWD:rw \
-w $PWD \
maven:alpine \
mvn -Duser.home=/var/maven "$#"

Resources