I`d like to know is there good way to move folder/file, which is outside the building context, to
Inside the build context when docker-compose build.
Is that possible to solve by using init.sh or startup.sh by using Docker-compose?
When I build this Dockerfile,
….
# set assets to inside docker container
COPY ../../frontend/src/assets /var/www/assets
….
And I did docker-compose build
However I got error about this
Step 19/21 : COPY ../../frontend/src/assets /var/www/assets
ERROR: Service 'test' failed to build: COPY failed: Forbidden path outside the build context: ../../frontend/src/assets ()
If I execute “ cp -rf ../../frontend/src/assets ./“ before build and change path of folder in Dockerfile, this is no problem,
But if I could, I want to make this less operation.
Related
I want to copy folder D:\test from my host OS (Windows) to my docker image.
My Docker file is D:\Programs>
Docker file
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir test
COPY D:/test to /root/test
#USING 'COPY' here instead of 'ADD' because of https://stackoverflow.com/questions/24958140/what-is-the-difference-between-the-copy-and-add-commands-in-a-dockerfile
From folder D:\Programs> I run command docker build . -t test
But I get error:
COPY failed: file not found in build context or excluded by .dockerignore: stat test: file does not exist
I tried commands COPY D:/test to /root/test and COPY ./test to /root/test
I know this error occurs because the path I copy from has to be relative from the build context (the . in docker build .). It cannot be an arbitrary path on the system outside of the build context.
I thought by using . I'd be in context D:\ from my build context D:\Programs>, but I guess not. What should my COPY command look like?
I checked here already:
https://forums.docker.com/t/copy-files-from-windows-host-to-ubuntu-container/28757
Dockerfile COPY from a Windows file system to a docker container
Dockerfile: Copy directory from Windows host to docker container
UPDATE 1
I placed the test folder in D:\Programs where my Dockerfile also resides, so I now have structure:
D:\Programs
\Test
Dockerfile
I then ran the build command again where I tried COPY test to /root/test and COPY ./test to /root/test, but both fail with the same aforementioned error.
I assume you've only added to in COPY D:/test to /root/test for abbreviation but I recommend checking that anyway.
If you really need what's in D:/Programs and can't move it to a folder inside the current build context there are two general options as I see it:
changing the build context - I'd suggest setting the build context as D:/ and adding every folder in the D:/ directory other than Programs to the .dockerignore file, then you can reference the Dockerfile from some subdirectory somewhere in D:/ and copy your files at build time as you wanted
creating a bind mount in runtime - you can map the D:/Programs directory to a folder inside the container (/my-programs) at runtime then copy the files from that folder to /root/test or just use the bind mount folder (/my-programs)
If in the future it will be available to use bind mounts in build time you'd be able to use the second option while creating the docker image
The only way I know how to do this is to change the dockerfile location during build, keeping the copied folder in context.
As below:
'test' folder located at D:/test
Set dockerfile COPY command as such:
COPY test /root/test
Say dockerfile is at D:/Programs/dockerfile
Navigate to D:/ in CLI
docker build -f ./Programs/dockerfile .
The . keeps the "test" folder in the build context. As far as I know, you cannot go to a parent directory etc. with docker build, so instead you want to stay in the parent folder and go down to the dockerfile.
If you run the docker build ... from D:\Programs that folder is the docker build context. All files used in the Dockerfile must be there.
d:\files is outside d:\programs, so it will be never be found.
You need to copy files content to the places where you run the command and you can use this copy line
COPY samplefile1.txt /root/test
I'm building a new image and copy contents from host OS folder D:\Programs\scrapy into it like so: docker build . -t scrapy
Dockerfile
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir scrapy
COPY scrapy to /root/scrapy
Now when I add new contents to the host OS folder "D:\Programs\scrapy" I want to also add it to image folder "root/scrapy", but I DON'T want to build a completely new image (it takes quite a while).
So how can I keep the existing image and just overwrite the contents of the image folder "root/scrapy".
Also: I don't want to copy the new contents EACH time I run the container (so NOT at run-time), I just want to have a SEPARATE command to add more files to an existing image and then run a new container based on that image at another time.
I checked here: How to update source code without rebuilding image (but not sure if OP tries to do the same as me)
UPDATE 1
Checking What is the purpose of VOLUME in Dockerfile and docker --volume format for Windows
I tried the commands below, all resulting in error:
docker: Error response from daemon: invalid volume specification: ''. See 'docker run --help'.
Where <pathiused> is for example D:/Programs/scrapy:/root/scrapy
docker run -v //D/Programs/scrapy:/root/scrapy scrapy
docker run -v scrapy:/root/scrapy scrapy
docker run -it -v //D/Programs/scrapy:/root/scrapy scrapy
docker run -it -v scrapy:/root/scrapy scrapy
UPDATE WITH cp command based on #Makariy's feedback
docker images -a gives:
REPOSITORY TAG IMAGE ID CREATED SIZE
scrapy latest e35e03c8cbbd 29 hours ago 5.71GB
<none> <none> 2089ad178feb 29 hours ago 5.71GB
<none> <none> 6162a0bec2fc 29 hours ago 5.7GB
<none> <none> 116a0c593544 29 hours ago 5.7GB
mcr.microsoft.com/windows/servercore ltsc2019 d1724c2d9a84 5 weeks ago 5.7GB
I run docker run -it scrapy and then docker container ls which gives:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1fcda458a14c scrapy "c:\\windows\\system32…" About a minute ago Up About a minute thirsty_bassi
If I run docker cp D:\Programs\scrapy scrapy:/root/scrapy I get:
Error: No such container:path: scrapy:\root
So in a separate PowerShell instance I then run docker cp D:\Programs\scrapy thirsty_bassi:/root/scrapy whichs show no output in PowerShell whatsoever, so I think it should've done something.
But then in my container instance when I goto /root/scrapy folder I only see the files that were already added when the image was built, not the new ones I wanted to add.
Also, I think I'm adding files to the container here, but is there no way to add it to the image instead? Without rebuilding the whole image?
UPDATE 2
My folder structure:
D:\Programs
Dockerfile
\image_addons
Dockerfile
\scrapy
PS D:\Programs>docker build . -t scrapybase
Successfully built 95676d084e28
Successfully tagged scrapybase:latest
PS D:\Programs\image_addons> docker build -t scrapy .
Step 2/2 : COPY scrapy to /root/scrapy
COPY failed: file not found in build context or excluded by .dockerignore: stat to: file does not exist
Dockerfile A
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
WORKDIR /root/scrapy
Dockerfile B
FROM scrapybase
COPY scrapy to /root/scrapy
You also can use docker cp, to manually copy files from your host to running container
docker cp ./path/to/file containername:/another/path
Docs
answer if you want it quick and dirty
docker run -it -v c:/programs/test:/root/test ubuntu:latest cat /root/test/myTestFile.txt
to update one file quickly:
If you don't have to build your code (I don't know what language you are using) you can build some base image with the initial code and when you want to change only one file (again I'm assuming you don't need to compile your project again for that, otherwise if you do that is not possible to due the nature of compiled programming language):
FROM previous-version-image:latest
COPY myfile dest/to/file
then because your CMD and ENTRYPOINT are saved from the previous stages no need to declare them. (if you don't remember use docker history <docker-image-name> to view virtual dockerfile for image to this stage).
Notice though not to repetitively use this method or you'll get a very big image with many useless layers. Use this only for quick testing and debugging.
explanation
Usually people use it for frontend development on docker containers but the basic idea persists, you create the basic working image with the dependencies installed and the directory layout setup with the last Dockerfile command being the development server start command.
example:
Dockerfile:
# pull the base image
FROM node:slim
# set the working directory
WORKDIR /app
# add `/app/node_modules/.bin` to $PATH
ENV PATH /app/node_modules/.bin:$PATH
# copy dependencies files
COPY package.json ./
COPY package-lock.json ./
# install app dependencies
RUN npm install
# add app
COPY . ./
# start development server
CMD ["npm", "start"]
startup command:
docker run -it --rm \
-v ${PWD}:/app \ <mount current working directory in host to container in path /app>
-v /app/node_modules \ <or other dependency directory if exists>
-p 80:3000 \ <ports if needs exposing>
ps-container:dev
I'm not sure if that use case will 100% work for you because it needs the code to be mounted using bind-mount all the time and when needed to be exported will have to be exported as the image and the source code directory, on the other hand, it allows you to make quick changes without waiting for the image to be built each time you add something new and in the end build the final image that contains all that's needed.
more relatable example to question provided code:
As you can see there is a file on the host machine that contains some text
the command that uses bind-mount to have access to the file:
docker run -it -v c:/programs/test:/root/test ubuntu:latest cat /root/test/myTestFile.txt
hope you find something that works for you from what I've provided here.
thanks to this tutorial and this example for starting examples and information.
EDIT:
Let's say your original Dockerfile looks like this:
FROM python:latest
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD python /app/app.py
This will build your initial image on top of we'll add layers and change the python files.
The next Dockerfile we'd use (let's call it Dockerfile.fix file) would copy the file we want to change instead of the ones already in the image
FROM previous-image-name
COPY app.py .
Now with after building with this Dockerfile the final image Dockerfile would look (sort of) like so:
FROM python:latest
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD python /app/app.py
FROM previous-image-name
COPY app.py .
And each time we'll want to change the file we'll use the second Dockerfile
There's no way you can change a Docker image without (at least partially) rebuilding it. But you don't have to rebuild all of it, you can just rebuild the layer copying your scrapy content.
You can optimize your build to have two images:
First image is your static image you don't want to rebuild each time. Let's call it scrapy-base.
Second and final image is based on first image scrapy-base and will only exist for the purpose of copying your dynamic scrapy content
scrapy-base's Dockerfile:
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir scrapy
And build it like:
docker build -t scrapy-base .
This command only needs to be run once. You won't have to build this image if you only change the content of local scrapy folder. (as you can see, the build does not use it at all)
scrapy's Dockerfile:
FROM scrapy-base
COPY scrapy /root/scrapy
With build command:
docker build -t scrapy .
This second build command will re-use the previous static image and only copy content without having to rebuild the entire image. Even with lots of files it should be pretty quick. You don't need to have a running container.
For your scenario :
docker run -v D:/test:/root/test your-image
A lots of valuable details available in this thread
I have a deploy script that I am trying to use for my server for CD but I am running into issues writing the bash script to complete some of my required steps such as running npm and the migration commands.
How would I go about getting into a container bash, from this script, running the commands below and then exiting to finish pulling up the changes?
Here is the script I am trying to automate:
cd /Project
docker-compose -f docker-compose.prod.yml down
git pull
docker-compose -f docker-compose.prod.yml build
# all good until here because it opens bash and does not allow more commands to run
docker-compose -f docker-compose.prod.yml run --rm web bash
npm install # should be run inside of web bash
python manage.py migrate_all # should be run inside of web bash
exit # should be run inside of web bash
# back out of web bash
docker-compose -f docker-compose.prod.yml up -d
Typically a Docker image is self-contained, and knows how to start itself up without any user intervention. With some limited exceptions, you shouldn't ever need to docker-compose run interactive shells to do post-deploy setup, and docker exec should be reserved for emergency debugging.
You're doing two things in this script.
The first is to install Node packages. These should be encapsulated in your image; your Dockerfile will almost always look something like
FROM node
WORKDIR /app
COPY package*.json .
RUN npm ci # <--- this line
COPY . .
CMD ["node", "index.js"]
Since the dependencies are in your image, you don't need to re-install them when the image starts up. Conversely, if you change your package.json file, re-running docker-compose build will re-run the npm install step and you'll get a clean package tree.
(There's a somewhat common setup that puts the node_modules directory into an anonymous volume, and overwrites the image's code with a bind mount. If you update your image, it will get the old node_modules directory from the anonymous volume and ignore the image updates. Delete these volumes: and use the code that's built into the image.)
Database migrations are a little trickier since you can't run them during the image build phase. There are two good approaches to this. One is to always have the container run migrations on startup. You can use an entrypoint script like:
#!/bin/sh
python manage.py migrate_all
exec "$#"
Make this script be executable and make it be the image's ENTRYPOINT, leaving the CMD be the command to actually start the application. On every container startup it will run migrations and then run the main container command, whatever it may be.
This approach doesn't necessarily work well if you have multiple replicas of the container (especially in a cluster environment like Docker Swarm or Kubernetes) or if you ever need to downgrade. In these cases it might make more sense to manually run migrations by hand. You can do that separately from the main container lifecycle with
docker-compose run web \
python manage.py migrate_all
Finally, in terms of the lifecycle you describe, Docker images are immutable: this means that it's safe to rebuild new images while the old ones are running. A minimum-downtime approach to the upgrade sequence you describe might look like:
git pull
# Build new images (includes `npm install`)
docker-compose build
# Run migrations (if required)
docker-compose run web python manage.py migrate_all
# Restart all containers
docker-compose up --force-recreate
My scenario is as follow
I need to add "project" folder to docker container for production build but for development build I want to mount local volume to project folder of container
eg. ADD project /var/www/html/project in production
Nothing in development (I can copy a dummy folder in development)
If I copy whole project folder to container in development then any changes in project folder will invalidate the docker cache of layers after the add command. It will take time to build docker image in development.
I want to use same docker file for both environment
To achieve that I used ADD $PROJECT_DIR /var/www/html/project in docker file, where $PROJECT_DIR is environment variable
Setting the environment variable in docker file like ENV PROJECT_DIR project or ENV CONFIG_FILE_PATH dummy-folder adds correct folders to container, but it needs me to change docker file each time.
I can also pass "build-arg" parameter when building docker image like
docker build -t myproject --build-arg "BUILD_TYPE=PROD" --build-arg "PROJECT_DIR=project" .
As variables BUILD_TYPE and PROJECT_DIR are related, I want to set CONFIG_FILE_PATH variable based on BUILD_TYPE. This will prevent case of me forgetting to change one parameter.
For setting the PROJECT_DIR variable I written following script "set_config_path.sh"
if [ $BUILD_TYPE="PROD" ]; then
PROJECT_DIR="project";
else
PROJECT_DIR="dummy-folder";
fi
I then run the script in dockerfile using
RUN . /root/set_project_folder.sh
Doing this, set_project_folder.sh script can access BUILD_TYPE variable but PROJECT_DIR is not reflected back in docker file
When running the set_project_folder.sh in my local machine's terminal, the PROJECT_DIR variable is changed but it is not working with dockerfile
Is there anyway we can change environment variable from subshell script e.g "set_config_path.sh" in above questions?
If it is possible, It can be used in many use cases to make docker build dynamic
Am I doing anything wrong here?
OR
Is there another good way to achieve this?
You can use something like below
FROM alpine
ARG BUILD_TYPE=prod
ARG CONFIG_FILE_PATH=config-$BUILD_TYPE.yml
RUN echo "BUILD_TYPE=$BUILD_TYPE CONFIG_FILE_PATH=$CONFIG_FILE_PATH"
CMD echo "BUILD_TYPE=$BUILD_TYPE CONFIG_FILE_PATH=$CONFIG_FILE_PATH"
The output would be like
Step 4/4 : RUN echo "BUILD_TYPE=$BUILD_TYPE CONFIG_FILE_PATH=$CONFIG_FILE_PATH"
---> Running in b5de774d9ebe
BUILD_TYPE=prod CONFIG_FILE_PATH=config-prod.yml
But if you run the image
$ docker run 9df23a126bb1
BUILD_TYPE= CONFIG_FILE_PATH=
This is because build args are not persisted as environment variables. If you want to persists these variables in the image also then you need to add below
ENV BUILD_TYPE=$BUILD_TYPE CONFIG_FILE_PATH=$CONFIG_FILE_PATH
And now docker run will also output
$ docker run c250a9d1d109
BUILD_TYPE=prod CONFIG_FILE_PATH=config-prod.yml
I'm a beginner in working with docker especially docker compose. Currently, creation my initial easy docker environment, I run into the first error and I've no clue why.
I tried to search for a solution in stackoverflow but found nothing that could help me.
Starting my docker with "docker-compose up" I get the following error:
$ docker-compose up
Removing errorinstance_app_1
Recreating 8a358dfcb306_8a358dfcb306_8a358dfcb306_errorinstance_app_1 ...
Recreating 8a358dfcb306_8a358dfcb306_8a358dfcb306_errorinstance_app_1 ... error
ERROR: for 8a358dfcb306_8a358dfcb306_8a358dfcb306_errorinstance_app_1 Cannot start service app: oci runtime error: container_linux.go:265: starting container process caused "exec: \"./run.sh\": stat ./run.sh: no such file or directory"
ERROR: for app Cannot start service app: oci runtime error: container_linux.go:265: starting container process caused "exec: \"./run.sh\": stat ./run.sh: no such file or directory"
ERROR: Encountered errors while bringing up the project.
So. Following my folder structure:
Project
docker-compose.yml
Docker
Java
Dockerfile
src
run.sh
Following my docker-compose.yml:
version: '2'
services:
app:
build:
dockerfile: ./Docker/Java/Dockerfile
context: .
volumes:
- ./src:/usr/local/etc/
working_dir: /usr/local/etc/
command: ./run.sh
And following my docker file:
FROM java:7-jdk-alpine
# WORKDIR /usr/local/etc
run.sh
echo "Hello world."
Yes, I know that I could do that solution only in a docker-compose file. But in the future I need to extend the Dockerfile.
Can someone help me respectively does anyone see the issue?
The problem is with the base docker image you are using in dockerfile:
FROM java:7-jdk-alpine
You are trying to start container by running run.sh bash script. But the above image doesn't support bash itself
For reference, you can see the documentation of above image in docker hub page here. Quoting the necessary portion here:
java:alpine
...
To minimize image size, it's uncommon for additional related tools
(such as git or bash) to be included in Alpine-based images. Using
this image as a base, add the things you need in your own Dockerfile
(see the alpine image description for examples of how to install
packages if you are unfamiliar).
That's about the problem.
Now, I can think of 2 solutions:
Just use java:7-jdk as base image instead of java:7-jdk-alpine
Install bash on top of the base image java:7-jdk-alpine by changing dockerfile to:
FROM java:7-jdk-alpine
RUN apk update && apk upgrade && apk add bash
#WORKDIR /usr/local/etc
*source of steps to install bash in alpine linux is here
It looks like docker compose can't find your run.sh file. This file needs to be included in your docker image.
Change your Dockerfile to the following, then rebuild the image with docker build -t <YOUR_IMAGE_NAME> ..
FROM java:7-jdk-alpine
ADD run.sh /usr/local/etc/run.sh
Once your image is rebuilt, run docker-compose up again.
The easiest way to tackle the problem is to execute a bash session in the container, then inside the container, you have to check if the file exists in the
indicated path if the file is not in the path, it must be included when you create the image into the docker file or through a volume inside de docker-compose.
Another thing to check is the relative path you are using. It will be clear when you check the existence of the file inside de docker container
docker exec -it CONTAINER_NAME bash
I recommend you to create a volume in the docker compose file, as it is the easier way, and also the best way.
there is a question that I want to do you, why are you putting the Dockerfile file inside a Java path?
It is not a good idea o guideline to follow
The correct way is to put your dockerfile file into an environment folder, in such a way the dockerfile file is not related to the java source of your application
I got this Error quite a lot and after a lot of investigation, it looked like some images were corrupted.
Deleting those and rebuilding solve the problem. It was not docker installation or configuration itself.