I want to copy folder D:\test from my host OS (Windows) to my docker image.
My Docker file is D:\Programs>
Docker file
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir test
COPY D:/test to /root/test
#USING 'COPY' here instead of 'ADD' because of https://stackoverflow.com/questions/24958140/what-is-the-difference-between-the-copy-and-add-commands-in-a-dockerfile
From folder D:\Programs> I run command docker build . -t test
But I get error:
COPY failed: file not found in build context or excluded by .dockerignore: stat test: file does not exist
I tried commands COPY D:/test to /root/test and COPY ./test to /root/test
I know this error occurs because the path I copy from has to be relative from the build context (the . in docker build .). It cannot be an arbitrary path on the system outside of the build context.
I thought by using . I'd be in context D:\ from my build context D:\Programs>, but I guess not. What should my COPY command look like?
I checked here already:
https://forums.docker.com/t/copy-files-from-windows-host-to-ubuntu-container/28757
Dockerfile COPY from a Windows file system to a docker container
Dockerfile: Copy directory from Windows host to docker container
UPDATE 1
I placed the test folder in D:\Programs where my Dockerfile also resides, so I now have structure:
D:\Programs
\Test
Dockerfile
I then ran the build command again where I tried COPY test to /root/test and COPY ./test to /root/test, but both fail with the same aforementioned error.
I assume you've only added to in COPY D:/test to /root/test for abbreviation but I recommend checking that anyway.
If you really need what's in D:/Programs and can't move it to a folder inside the current build context there are two general options as I see it:
changing the build context - I'd suggest setting the build context as D:/ and adding every folder in the D:/ directory other than Programs to the .dockerignore file, then you can reference the Dockerfile from some subdirectory somewhere in D:/ and copy your files at build time as you wanted
creating a bind mount in runtime - you can map the D:/Programs directory to a folder inside the container (/my-programs) at runtime then copy the files from that folder to /root/test or just use the bind mount folder (/my-programs)
If in the future it will be available to use bind mounts in build time you'd be able to use the second option while creating the docker image
The only way I know how to do this is to change the dockerfile location during build, keeping the copied folder in context.
As below:
'test' folder located at D:/test
Set dockerfile COPY command as such:
COPY test /root/test
Say dockerfile is at D:/Programs/dockerfile
Navigate to D:/ in CLI
docker build -f ./Programs/dockerfile .
The . keeps the "test" folder in the build context. As far as I know, you cannot go to a parent directory etc. with docker build, so instead you want to stay in the parent folder and go down to the dockerfile.
If you run the docker build ... from D:\Programs that folder is the docker build context. All files used in the Dockerfile must be there.
d:\files is outside d:\programs, so it will be never be found.
You need to copy files content to the places where you run the command and you can use this copy line
COPY samplefile1.txt /root/test
Related
I'm building a new image like so docker build . -t test and copy contents from host OS folder into it:
Dockerfile
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir test
COPY test /root/test
#USING 'COPY' here instead of 'ADD' because of https://stackoverflow.com/questions/24958140/what-is-the-difference-between-the-copy-and-add-commands-in-a-dockerfile
I get this output in Powershell:
PS D:\Programs> docker build . -t test
Sending build context to Docker daemon 1.644GB
Step 1/7 : FROM mcr.microsoft.com/windows/servercore:ltsc2019
ltsc2019: Pulling from windows/servercore
4612f6d0b889: Pull complete
c67ded6868b6: Pull complete
Digest: sha256:1be9c8378b8c32d31538253a4b96169f5138a5f6d4c0a04e8d8f9a80b9ac8097
Status: Downloaded newer image for mcr.microsoft.com/windows/servercore:ltsc2019
---> d1724c2d9a84
Step 2/7 : SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
---> Running in 1bfefefbe433
Removing intermediate container 1bfefefbe433
---> 37de702deb33
Step 3/7 : RUN mkdir root
---> Running in e26d6b49ced7
Directory: C:\
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 9/8/2021 1:07 PM root
Removing intermediate container e26d6b49ced7
---> 451c4d3f3ea1
Step 4/7 : RUN cd root
---> Running in 74a228f8118f
Removing intermediate container 74a228f8118f
---> 3f175ac67f1d
Step 5/7 : WORKDIR /root
---> Running in 5f783d5b2332
Removing intermediate container 5f783d5b2332
---> 68b24e033f87
Step 6/7 : RUN mkdir test
---> Running in 5771bb7a593a
Directory: C:\root
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 9/8/2021 1:08 PM test
Removing intermediate container 5771bb7a593a
---> 35fa0b2af157
Step 7/7 : COPY test /root/test
---> 60eab8242865
Successfully built 60eab8242865
Successfully tagged test:latest
It creates EIGHT(!) folders on the host OS in folder "\docker\windowsfilter"
This seems excessive, it looks like a folder for each step in my Dockerfile, so what are all these folders for and could I prevent creation/autodelete the unneeded ones to keep everything tidy?
Host OS: Windows Server 2019 standard Version 1809 17763.2114
Docker: version 20.10.4, build 110e091
Please note that I do NOT use the Docker Desktop application, see here how I installed Docker.
Update 1
I updated my Dockerfile based on #Noam's suggestion like so:
#RUN mkdir root
#RUN cd root
#WORKDIR /root
WORKDIR /root #creates root directory if not exists, then enters it
After this change, 6 folders (instead of 8) are created in windowsfilter, so my hunch that Docker creates a folder for each command in Dockerfile seems to be correct.
Hallo #Flo this issue seems to be a long time issue specifically with docker desktop,
Causes:
One option that some times seems to explain it is upgrading from older versions of docker desktop and the software not cleaning up old directories.
Another I that docker saves built image layers there so if you have many versions of the same image your dangling ones could reside there requesting a run of docker image prune or maybe even docker system prune
Solutions:
An opensource cleanup script has been published and it seems to help a great deal, its called docker-ci-zap and for the most part it appears to handle it quite well.
if the issue still persists there is a fix suggested in the docker for desktop github issue page is adding -removing suffix to the folder names in the windowsfilter directory followed by a restart to allow docker to cleanup the unneeded folders
if it's still causing issues try returning to factory deafults and if that doesn't do it either maybe a fresh docker for desktop install might seal the deal though even then some experienced some difficulties so eventually if it really bothers and you need to use it, reinstalling windows usually looks like the final blow.
suggestion for Dockerfile improvments
in your Dockerfile the lines describing the folders might cause problems up a head du to unabsolute paths and improper use of the Dockerfile command.
replace:
RUN mkdir root
RUN cd root
WORKDIR /root
with:
WORKDIR /root
it achives the same purpose (creating and entering the new (if didn't exist yet) directory)
what does docker-ci-zap do?
You can get a rough sense by looking at the code used to compile the exe file that resides in the GitHub repo:
it uses the hcsshim driver and its DestroyLayer function that deletes the layer files from the host disk, or as the docs put it:
DestroyLayer will remove the on-disk files representing the layer
with the given path, including that layer's containing folder, if
any.
I'm building a new image and copy contents from host OS folder D:\Programs\scrapy into it like so: docker build . -t scrapy
Dockerfile
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir scrapy
COPY scrapy to /root/scrapy
Now when I add new contents to the host OS folder "D:\Programs\scrapy" I want to also add it to image folder "root/scrapy", but I DON'T want to build a completely new image (it takes quite a while).
So how can I keep the existing image and just overwrite the contents of the image folder "root/scrapy".
Also: I don't want to copy the new contents EACH time I run the container (so NOT at run-time), I just want to have a SEPARATE command to add more files to an existing image and then run a new container based on that image at another time.
I checked here: How to update source code without rebuilding image (but not sure if OP tries to do the same as me)
UPDATE 1
Checking What is the purpose of VOLUME in Dockerfile and docker --volume format for Windows
I tried the commands below, all resulting in error:
docker: Error response from daemon: invalid volume specification: ''. See 'docker run --help'.
Where <pathiused> is for example D:/Programs/scrapy:/root/scrapy
docker run -v //D/Programs/scrapy:/root/scrapy scrapy
docker run -v scrapy:/root/scrapy scrapy
docker run -it -v //D/Programs/scrapy:/root/scrapy scrapy
docker run -it -v scrapy:/root/scrapy scrapy
UPDATE WITH cp command based on #Makariy's feedback
docker images -a gives:
REPOSITORY TAG IMAGE ID CREATED SIZE
scrapy latest e35e03c8cbbd 29 hours ago 5.71GB
<none> <none> 2089ad178feb 29 hours ago 5.71GB
<none> <none> 6162a0bec2fc 29 hours ago 5.7GB
<none> <none> 116a0c593544 29 hours ago 5.7GB
mcr.microsoft.com/windows/servercore ltsc2019 d1724c2d9a84 5 weeks ago 5.7GB
I run docker run -it scrapy and then docker container ls which gives:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1fcda458a14c scrapy "c:\\windows\\system32…" About a minute ago Up About a minute thirsty_bassi
If I run docker cp D:\Programs\scrapy scrapy:/root/scrapy I get:
Error: No such container:path: scrapy:\root
So in a separate PowerShell instance I then run docker cp D:\Programs\scrapy thirsty_bassi:/root/scrapy whichs show no output in PowerShell whatsoever, so I think it should've done something.
But then in my container instance when I goto /root/scrapy folder I only see the files that were already added when the image was built, not the new ones I wanted to add.
Also, I think I'm adding files to the container here, but is there no way to add it to the image instead? Without rebuilding the whole image?
UPDATE 2
My folder structure:
D:\Programs
Dockerfile
\image_addons
Dockerfile
\scrapy
PS D:\Programs>docker build . -t scrapybase
Successfully built 95676d084e28
Successfully tagged scrapybase:latest
PS D:\Programs\image_addons> docker build -t scrapy .
Step 2/2 : COPY scrapy to /root/scrapy
COPY failed: file not found in build context or excluded by .dockerignore: stat to: file does not exist
Dockerfile A
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
WORKDIR /root/scrapy
Dockerfile B
FROM scrapybase
COPY scrapy to /root/scrapy
You also can use docker cp, to manually copy files from your host to running container
docker cp ./path/to/file containername:/another/path
Docs
answer if you want it quick and dirty
docker run -it -v c:/programs/test:/root/test ubuntu:latest cat /root/test/myTestFile.txt
to update one file quickly:
If you don't have to build your code (I don't know what language you are using) you can build some base image with the initial code and when you want to change only one file (again I'm assuming you don't need to compile your project again for that, otherwise if you do that is not possible to due the nature of compiled programming language):
FROM previous-version-image:latest
COPY myfile dest/to/file
then because your CMD and ENTRYPOINT are saved from the previous stages no need to declare them. (if you don't remember use docker history <docker-image-name> to view virtual dockerfile for image to this stage).
Notice though not to repetitively use this method or you'll get a very big image with many useless layers. Use this only for quick testing and debugging.
explanation
Usually people use it for frontend development on docker containers but the basic idea persists, you create the basic working image with the dependencies installed and the directory layout setup with the last Dockerfile command being the development server start command.
example:
Dockerfile:
# pull the base image
FROM node:slim
# set the working directory
WORKDIR /app
# add `/app/node_modules/.bin` to $PATH
ENV PATH /app/node_modules/.bin:$PATH
# copy dependencies files
COPY package.json ./
COPY package-lock.json ./
# install app dependencies
RUN npm install
# add app
COPY . ./
# start development server
CMD ["npm", "start"]
startup command:
docker run -it --rm \
-v ${PWD}:/app \ <mount current working directory in host to container in path /app>
-v /app/node_modules \ <or other dependency directory if exists>
-p 80:3000 \ <ports if needs exposing>
ps-container:dev
I'm not sure if that use case will 100% work for you because it needs the code to be mounted using bind-mount all the time and when needed to be exported will have to be exported as the image and the source code directory, on the other hand, it allows you to make quick changes without waiting for the image to be built each time you add something new and in the end build the final image that contains all that's needed.
more relatable example to question provided code:
As you can see there is a file on the host machine that contains some text
the command that uses bind-mount to have access to the file:
docker run -it -v c:/programs/test:/root/test ubuntu:latest cat /root/test/myTestFile.txt
hope you find something that works for you from what I've provided here.
thanks to this tutorial and this example for starting examples and information.
EDIT:
Let's say your original Dockerfile looks like this:
FROM python:latest
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD python /app/app.py
This will build your initial image on top of we'll add layers and change the python files.
The next Dockerfile we'd use (let's call it Dockerfile.fix file) would copy the file we want to change instead of the ones already in the image
FROM previous-image-name
COPY app.py .
Now with after building with this Dockerfile the final image Dockerfile would look (sort of) like so:
FROM python:latest
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD python /app/app.py
FROM previous-image-name
COPY app.py .
And each time we'll want to change the file we'll use the second Dockerfile
There's no way you can change a Docker image without (at least partially) rebuilding it. But you don't have to rebuild all of it, you can just rebuild the layer copying your scrapy content.
You can optimize your build to have two images:
First image is your static image you don't want to rebuild each time. Let's call it scrapy-base.
Second and final image is based on first image scrapy-base and will only exist for the purpose of copying your dynamic scrapy content
scrapy-base's Dockerfile:
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir scrapy
And build it like:
docker build -t scrapy-base .
This command only needs to be run once. You won't have to build this image if you only change the content of local scrapy folder. (as you can see, the build does not use it at all)
scrapy's Dockerfile:
FROM scrapy-base
COPY scrapy /root/scrapy
With build command:
docker build -t scrapy .
This second build command will re-use the previous static image and only copy content without having to rebuild the entire image. Even with lots of files it should be pretty quick. You don't need to have a running container.
For your scenario :
docker run -v D:/test:/root/test your-image
A lots of valuable details available in this thread
Visual Studio 2019 generates a docker file whose COPY statement looks like the following due to the way that the application's folders are structured:
FROM mcr.microsoft.com/dotnet/sdk:3.1 AS build
WORKDIR /src
COPY ["MyWebApp/MyWebApp.csproj", "MyWebApp/"]
COPY ["../../core/Logic/Logic.csproj", "../../core/Logic/"]
COPY ["../../core/Models/Models.csproj", "../../core/Models/"]
RUN dotnet restore "MyWebApp/MyWebApp.csproj"
COPY . .
WORKDIR "/src/MyWebApp"
RUN dotnet build "MyWebApp.csproj" -c Release -o /app/build
I keep getting the following error message when running docker compose command:
=> ERROR [build 4/9] COPY [../../core/Logic/Logic.csproj,
../../core/Logic/] 0.0s
=> ERROR [build 5/9] COPY [../../core/Models/Models.csproj, ../../core/Models/]
We cannot restructure the folders in VS 2019 as it has been set up in that way due to some reasons. The docker file resides in MyWebApp project's folder and the docker compose file is in the parent folder of MyWebApp folder where the solution resides. Even I moved the docker file adjacent to the docker compose file but it was not conclusive.
The following code snippet is the docker-compose file:
version: '3.4'
services:
mywebapp:
image: ${DOCKER_REGISTRY-}mywebapp
build:
context: .
dockerfile: MyWebApp/Dockerfile
ports:
- 8080:80
- 8443:443
What is the workaround or solution to address this problem?
If your file structure is:
docker-compose.yml
MyWebApp/MyWebApp.csproj
core/Logic/Logic.csproj
core/Models/Models.csproj
Your Dockerfile should be:
FROM mcr.microsoft.com/dotnet/sdk:3.1 AS build
WORKDIR /src
COPY ["MyWebApp/MyWebApp.csproj", "MyWebApp/"]
COPY ["core/Logic/Logic.csproj", "../../core/Logic/"]
COPY ["core/Models/Models.csproj", "../../core/Models/"]
RUN dotnet restore "MyWebApp/MyWebApp.csproj"
COPY . .
WORKDIR "/src/MyWebApp"
RUN dotnet build "MyWebApp.csproj" -c Release -o /app/build
Why?
Because the current working directory is always the context directory. Context can be set using docker-compose's build.context field. In your example, the context is the solution root. So the Dockerfile is executed from that directory, irrespective or where it's placed.
You cannot access files outside the context directory.
Eg. if context resolves to /a/b/c, Dockerfile can't COPY or use files in /a/b/d
i'm using logstash image and i have some ruby scripts that are located in the same directory as my dockerfile.
my goal is to create a script folder, and copy my scripts to it.
my problem is that when i access to the container instance, the folder is not created and no ruby file exist.
this is my dockerfile
FROM docker.elastic.co/logstash/logstash:${ELK_VERSION}
USER root
WORKDIR /usr/share/logstash
RUN mkdir scripts
COPY ./scripts/*.rb scripts
thanks in advance.
EDIT 1:
this is the files structure
I have a docker file that copies a file tree to the container using the command in the docker file:
COPY image /
Docker copies all the files BUT the .sh and .py etc files in the source folder ( on windows ) have execute permission turned on. When the container is run the copies present in the container DO NOT have execute permission turned on?
How do I deal with this other than explicitly chmod +x the files in the docker file when building the container (having to maintain a list of them etc) in this situation.