My scenario is as follow
I need to add "project" folder to docker container for production build but for development build I want to mount local volume to project folder of container
eg. ADD project /var/www/html/project in production
Nothing in development (I can copy a dummy folder in development)
If I copy whole project folder to container in development then any changes in project folder will invalidate the docker cache of layers after the add command. It will take time to build docker image in development.
I want to use same docker file for both environment
To achieve that I used ADD $PROJECT_DIR /var/www/html/project in docker file, where $PROJECT_DIR is environment variable
Setting the environment variable in docker file like ENV PROJECT_DIR project or ENV CONFIG_FILE_PATH dummy-folder adds correct folders to container, but it needs me to change docker file each time.
I can also pass "build-arg" parameter when building docker image like
docker build -t myproject --build-arg "BUILD_TYPE=PROD" --build-arg "PROJECT_DIR=project" .
As variables BUILD_TYPE and PROJECT_DIR are related, I want to set CONFIG_FILE_PATH variable based on BUILD_TYPE. This will prevent case of me forgetting to change one parameter.
For setting the PROJECT_DIR variable I written following script "set_config_path.sh"
if [ $BUILD_TYPE="PROD" ]; then
PROJECT_DIR="project";
else
PROJECT_DIR="dummy-folder";
fi
I then run the script in dockerfile using
RUN . /root/set_project_folder.sh
Doing this, set_project_folder.sh script can access BUILD_TYPE variable but PROJECT_DIR is not reflected back in docker file
When running the set_project_folder.sh in my local machine's terminal, the PROJECT_DIR variable is changed but it is not working with dockerfile
Is there anyway we can change environment variable from subshell script e.g "set_config_path.sh" in above questions?
If it is possible, It can be used in many use cases to make docker build dynamic
Am I doing anything wrong here?
OR
Is there another good way to achieve this?
You can use something like below
FROM alpine
ARG BUILD_TYPE=prod
ARG CONFIG_FILE_PATH=config-$BUILD_TYPE.yml
RUN echo "BUILD_TYPE=$BUILD_TYPE CONFIG_FILE_PATH=$CONFIG_FILE_PATH"
CMD echo "BUILD_TYPE=$BUILD_TYPE CONFIG_FILE_PATH=$CONFIG_FILE_PATH"
The output would be like
Step 4/4 : RUN echo "BUILD_TYPE=$BUILD_TYPE CONFIG_FILE_PATH=$CONFIG_FILE_PATH"
---> Running in b5de774d9ebe
BUILD_TYPE=prod CONFIG_FILE_PATH=config-prod.yml
But if you run the image
$ docker run 9df23a126bb1
BUILD_TYPE= CONFIG_FILE_PATH=
This is because build args are not persisted as environment variables. If you want to persists these variables in the image also then you need to add below
ENV BUILD_TYPE=$BUILD_TYPE CONFIG_FILE_PATH=$CONFIG_FILE_PATH
And now docker run will also output
$ docker run c250a9d1d109
BUILD_TYPE=prod CONFIG_FILE_PATH=config-prod.yml
Related
I'm running CI jobs on a self-hosted GitLab instance plus 10 GitLab Runners.
For this minimal example, two Runners are needed:
Admin-01
A shell runner with Docker installed.
It can execute e.g. docker build ... to create new images, which are then pushed to the private Docker registry (also self-hosted / part of the GitLab installation)
Docker-01
A docker runner, which executes the previously build image.
On a normal bare-metal, virtual machine or shell runner, I would modify e.g. ~/.profile to execute commands before before_script or script sections are executed. In my use case I need to set new environment variables and source some configuration files provided by the tools I want to run in an image. Yes, environment variables could be set differently, but there seams to be no way to source Bash scripts automatically before before_script or script sections are executed.
When sourcing the Bash source file manually, it works. I also notice, that I have to source it again in script block. So I assume the Bash session is ended between before_script block to script block. Of cause, it's no nice solution to manually source the tools Bash configuration script in every .gitlab-ci.yml file manually by the image users.
myjobn:
# ...
before_script:
- source /root/profile.additions
- echo "PATH=${PATH}"
# ...
script:
- source /root/profile.additions
- echo "PATH=${PATH}"
# ...
The mentioned modifications for e.g. shell runners does not work in images executed by GitLab Runner. It feels like the Bash in the container is not started as login shell.
The minimal example image is build as follows:
fetch debian:bullseye-slim from Docker Hub
use RUN commands in Dockerfile to modify with some echo outputs
/etc/profile
/root/.bashrc
/root/.profile
# ...
RUN echo "echo GREETINGS FROM /ROOT/PROFILE" >> /root/.profile \
&& echo "echo GREETINGS FROM /ETC/PROFILE" >> /etc/profile \
&& echo "echo GREETINGS FROM /ROOT/BASH_RC" >> /root/.bashrc
When the job starts, non of the echos is printing messages, while a cat shows, the echo commands have been put at the right places while building the image.
At next I tried to modify
SHELL ["/bin/bash", "-l", "-c"]
But I assume, this has only effects in RUN commands in the Dockerfile, but not on an executed container.
CMD ["/bin/bash", "-l"]
I see no behavior changes.
Question:
How to start the Bash in the Docker image managed by GitLab Runner as login shell so it ready configuration scripts?
How to modify the environment in a container before before_script or script runs. Modifying means environment variables and execution / sourcing a configuration script or patched default script like ~/.profile.
How does GitLab Runner execute a job with Docker?
This is not documented by GitLab in the official documentation ...
What I know so far, it jumps between Docker images specified by GitLab and user defined images and shares some directories/volumes or so.
Note:
Yes, the behavior can be achieved with some Docker arguments in docker run, but as I wrote GitLab Runner is managing the container. Alternatively, how to configure, how GitLab Runner launches the images? To my knowledge, there is no configuration option available / documented for this situation.
A shell runner with Docker installed. It can execute e.g. docker build ...
Use docker-in-docker or use kaniko. https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
Shell executor is like "the last resort", where you want specifically to make changes to the server, or you are deploying your application "into" this server.
How to start the Bash in the Docker image managed by GitLab Runner as login shell so it ready configuration scripts?
Add ENTRYPOING bash -l to your image. Or set the entrypoint from gitlab-ci.yml. See docker documentation on ENTRYPOINT and gitlab-ci.yml documentation on image: entrypoint: .
How to modify the environment in a container before before_script or script runs.
Build the image with modified environment. Consult Dockerfile documentation on ENV statements.
Or set the environment from gitlab-ci.yml file. Read documentation on variables: in gitlab-ci.
How to prepare the shell environment in an image executed by GitLab Runner?
Don't. The idea is that the environment is reproducible, ergo, there should be no changes beforehand. Add variables: in gitlab-ci file and use base images if possible.
How does GitLab Runner execute a job with Docker?
This is not documented by GitLab in the official documentation ...
Gitlab is open-source.
What I know so far, it jumps between Docker images specified by GitLab and user defined images and shares some directories/volumes or so.
Yes, first a gitlab-runner-helper is executed - it has git and git-lfs and basically clones the repository and downloads and uploads the artifacts. Then the container specified with image: is run, cloned repository is copied into it and a specially prepared shell script is executed in it.
Ok, this seems easy enough for Linux containers but I am trying to get this done using Windows Containers and its annoying that its so difficult.
I have a Windows Dockerfile which builds a project and part of the build process is to reversion the C# AssemblyInfo.cs files so that the built assemblies have a build version from the CI environment (Devops)
I am using a Powershell script https://github.com/microsoft/psi/blob/master/Build/ApplyVersionToAssemblies.ps1, it expects 2 Environment variables, one which I can hardcode so is not a problem, but the BUILD_BUILDNUMBER environment variable needs to be injected from Devops build system.
I have tried the following, none of which work
ARG BUILD_BUILDNUMBER
ENV BUILD_BUILDNUMBER=$BUILD_BUILDNUMBER
RUN ApplyVersionToAssemblies.ps1
and running
docker build -f Dockerfile --build-arg BUILD_BUILDNUMBER=1.2.3.4 .
also
RUN SETX BUILD_BUILDNUMBER $BUILD_BUILDNUMBER
RUN SETX BUILD_BUILDNUMBER %BUILD_BUILDNUMBER%
and a few other combinations that I dont recall, what I ended up doing which works but seems like a hack is to pass the BUILDNUMBER as a file via a COPY and then modifying the the Powershell script to read that into its local variable
So for the moment it works but I would really like to know how this is supposed to work via ARG and ENV for Windows Container builds
Windows Containers definitely feel like Linux containers poor cousin :)
Example for CMD in Docker Windows Containers:
ARG FEED_ACCESSTOKEN
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS="{\"endpointCredentials\": [{\"endpoint\":\"https://URL.com/_packaging/Name/nuget/v3/index.json\", \"username\":\"PATForPackages\", \"password\":\"${FEED_ACCESSTOKEN}\"}]}"
SHELL ["cmd", "/S", "/C"]
RUN echo %VSS_NUGET_EXTERNAL_FEED_ENDPOINTS%
so i have downloaded Docker Desktop and until now i have tested out containers and stuff just executing regular commands (docker ps, docker images..., docker run...) inside my zsh terminal and it works fine but now i am in a position where i want to create a directory inside docker host so that i can put my dockerfile inside, but if i run mkdir directory-name it is going to create the directory inside my mac not docker! so what command can i use to indicate that i want the directory to be created on docker not on my own mac machine?
While your docker container is running, you can start a new shell session inside using docker exec.
docker exec -it mycontainer bash
-i means interactive - so you can type
-t allocates a pseudo-TTY - just know that you need the argument
Then inside this bash, you can create folders and files all you want and they will be placed inside your running container. Note that whenever you remove the container (e.g. to update its image), these changes will be entirely lost. For persistence, use docker volumes.
Say you have the following directory structure:
.
├── Dockerfile
└── simple-web-app
Your Dockerfile:
FROM scratch
ADD simple-web-app simple-web-app
Then you would run
docker build .
We would like to leverage the excellent catalogue of DockerFiles on DockerHub, but the team is not in a position to use Docker.
Is there any way to run a DockerFile as if it were a shell script against a machine?
For example, if I chose to run the Docker container ruby:2.4.1-jessie against a server running only Debian Jessie, I'd expect it to ignore the FROM directive but be able to set the environment from ENV and run the RUN commands from this Dockerfile: Github docker-library/ruby:2.4.1-jessie
A dockerfile assumes to be executed in an empty container or an image on which it builds (using FROM). The knowledge about the environment (specifically the file system and all the installed software) is important and running something similar outside of docker might have side effects because files are at places where no files are expected.
I wouldn't recomend it
My objective is to have a self contained Go Workspace per project.
Is it possible to retrieve a Go workspace and Go environment variables from a running Docker container to be used by an IDE or Text Editor for development?
I have already tried mapping a volume to the container with the go tools and dependencies. But that requires always setting the GOPATH to the current workspace, and requires to have the go tools and dependencies on the host.
You can at least set and pass those environment variable when launching your container:
docker run -e "GOPATH=/a/mounted/path" -v [host-src:]container-dest --rm -it <yourImage>
By using -v, you allow your host to share a folder with your container.