Why aren't files written in /root/.m2 in the maven3 docker image persistent during the build?
A simple dockerfile:
FROM maven:3-jdk-8
RUN touch /root/.m2/testfilehere && ls -a /root/.m2
RUN ls -a /root/.m2
CMD ["bash"]
Produces the following output.
$ docker build -t test --no-cache .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM maven:3-jdk-8
---> 42e3884987fb
Step 2 : RUN touch /root/.m2/testfilehere && ls -a /root/.m2
---> Running in 1c1dc5e9f082
.
..
testfilehere
---> 3da352119c4d
Removing intermediate container 1c1dc5e9f082
Step 3 : RUN ls -a /root/.m2
---> Running in df506db8c1dd
.
..
---> d07cc155b20e
Removing intermediate container df506db8c1dd
Step 4 : RUN stat /root/.m2/testfilehere
---> Running in af44f30aafe5
stat: cannot stat ‘/root/.m2/testfilehere’: No such file or directory
The command '/bin/sh -c stat /root/.m2/testfilehere' returned a non-zero code: 1
The file created at the first command is gone when the intermmediate container exists.
Also, this does not happen in the ubuntu image, just maven.
edit: using ADD hostfile /root/.m2/containerfile does work as a workaround, but is not what I want.
the maven docker image has an entrypoint
ENTRYPOINT ["/usr/local/bin/mvn-entrypoint.sh"]
On container started, entrypoint copy files from /usr/share/maven/ref into ${MAVEN_CONFIG} and erase your file
You can see script executed on startup by following this link
https://github.com/carlossg/docker-maven/blob/33eeccbb0ce15440f5ccebcd87040c6be2bf9e91/jdk-8/mvn-entrypoint.sh
That's because /root/.m2 is defined as a VOLUME in the image. When a container runs with a volume, the volume storage is not part of the UnionFS - so its data is not stored in the container's writable layer:
A data volume is a specially-designated directory within one or more containers that bypasses the Union File System.
The first RUN command creates a file in the volume, but that's in an intermediary container with its own volume. The file isn't saved to the image layer because it's in a volume.
The second RUN command is running in a new intermediary container which has its own volume. There's no content in the volume from the base image, so the volume is empty in the new container.
If you want to pre-populate the volume with data, you need to do it in the Dockerfile as you've seen.
There is a documentation for this in Docker Maven:
https://github.com/carlossg/docker-maven#packaging-a-local-repository-with-the-image
COPY settings.xml /usr/share/maven/ref/
After you run your Docker image, the settings.xml will appear in /root/.m2.
Related
I created a docker image using a script. The docker image got created successfully but when I am trying to run on the browser, I get the following message
I tried to resolve the error using webapps but it is popping up the same message.
It is happening probably because the directory /usr/local/tomcat/webapps is empty. as you can see in the description in the Docker Hub official image:
You can then go to http://localhost:8888 or http://host-ip:8888 in a browser (noting that it will return a 404 since there are no webapps loaded by default).
To solve that, you can just create a new image and copy and paste what you want inside of the folder /usr/local/tomcat/webapps.
For example, you can find inside of the folder /usr/local/tomcat/webapps.dist files of the tomcat webapp manager. If you try to see what is that you can run:
$ docker run -it tomcat:10 ls /usr/local/tomcat/webapps.dist
ROOT docs examples host-manager manager
Now that you know what is inside of the folder /usr/local/tomcat/webapps.dist, create your own Dockerfile and copy and paste this folder to /usr/local/tomcat/webapps:
FROM tomcat
RUN cp -R /usr/local/tomcat/webapps.dist/* /usr/local/tomcat/webapps
CMD ["catalina.sh", "run"]
Build the image:
$ docker build . -t custom-tomcat
Execute the image:
$ docker run -d -P custom-tomcat
Check the port opened:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
58390eab3fec custom-tomcat "catalina.sh run" 9 minutes ago Up 9 minutes 0.0.0.0:49163->8080/tcp, :::49163->8080/tcp lucid_joliot
Open your browser and check if it works:
I'm building a new image like so docker build . -t test and copy contents from host OS folder into it:
Dockerfile
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir test
COPY test /root/test
#USING 'COPY' here instead of 'ADD' because of https://stackoverflow.com/questions/24958140/what-is-the-difference-between-the-copy-and-add-commands-in-a-dockerfile
I get this output in Powershell:
PS D:\Programs> docker build . -t test
Sending build context to Docker daemon 1.644GB
Step 1/7 : FROM mcr.microsoft.com/windows/servercore:ltsc2019
ltsc2019: Pulling from windows/servercore
4612f6d0b889: Pull complete
c67ded6868b6: Pull complete
Digest: sha256:1be9c8378b8c32d31538253a4b96169f5138a5f6d4c0a04e8d8f9a80b9ac8097
Status: Downloaded newer image for mcr.microsoft.com/windows/servercore:ltsc2019
---> d1724c2d9a84
Step 2/7 : SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
---> Running in 1bfefefbe433
Removing intermediate container 1bfefefbe433
---> 37de702deb33
Step 3/7 : RUN mkdir root
---> Running in e26d6b49ced7
Directory: C:\
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 9/8/2021 1:07 PM root
Removing intermediate container e26d6b49ced7
---> 451c4d3f3ea1
Step 4/7 : RUN cd root
---> Running in 74a228f8118f
Removing intermediate container 74a228f8118f
---> 3f175ac67f1d
Step 5/7 : WORKDIR /root
---> Running in 5f783d5b2332
Removing intermediate container 5f783d5b2332
---> 68b24e033f87
Step 6/7 : RUN mkdir test
---> Running in 5771bb7a593a
Directory: C:\root
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 9/8/2021 1:08 PM test
Removing intermediate container 5771bb7a593a
---> 35fa0b2af157
Step 7/7 : COPY test /root/test
---> 60eab8242865
Successfully built 60eab8242865
Successfully tagged test:latest
It creates EIGHT(!) folders on the host OS in folder "\docker\windowsfilter"
This seems excessive, it looks like a folder for each step in my Dockerfile, so what are all these folders for and could I prevent creation/autodelete the unneeded ones to keep everything tidy?
Host OS: Windows Server 2019 standard Version 1809 17763.2114
Docker: version 20.10.4, build 110e091
Please note that I do NOT use the Docker Desktop application, see here how I installed Docker.
Update 1
I updated my Dockerfile based on #Noam's suggestion like so:
#RUN mkdir root
#RUN cd root
#WORKDIR /root
WORKDIR /root #creates root directory if not exists, then enters it
After this change, 6 folders (instead of 8) are created in windowsfilter, so my hunch that Docker creates a folder for each command in Dockerfile seems to be correct.
Hallo #Flo this issue seems to be a long time issue specifically with docker desktop,
Causes:
One option that some times seems to explain it is upgrading from older versions of docker desktop and the software not cleaning up old directories.
Another I that docker saves built image layers there so if you have many versions of the same image your dangling ones could reside there requesting a run of docker image prune or maybe even docker system prune
Solutions:
An opensource cleanup script has been published and it seems to help a great deal, its called docker-ci-zap and for the most part it appears to handle it quite well.
if the issue still persists there is a fix suggested in the docker for desktop github issue page is adding -removing suffix to the folder names in the windowsfilter directory followed by a restart to allow docker to cleanup the unneeded folders
if it's still causing issues try returning to factory deafults and if that doesn't do it either maybe a fresh docker for desktop install might seal the deal though even then some experienced some difficulties so eventually if it really bothers and you need to use it, reinstalling windows usually looks like the final blow.
suggestion for Dockerfile improvments
in your Dockerfile the lines describing the folders might cause problems up a head du to unabsolute paths and improper use of the Dockerfile command.
replace:
RUN mkdir root
RUN cd root
WORKDIR /root
with:
WORKDIR /root
it achives the same purpose (creating and entering the new (if didn't exist yet) directory)
what does docker-ci-zap do?
You can get a rough sense by looking at the code used to compile the exe file that resides in the GitHub repo:
it uses the hcsshim driver and its DestroyLayer function that deletes the layer files from the host disk, or as the docs put it:
DestroyLayer will remove the on-disk files representing the layer
with the given path, including that layer's containing folder, if
any.
I want to copy folder D:\test from my host OS (Windows) to my docker image.
My Docker file is D:\Programs>
Docker file
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir test
COPY D:/test to /root/test
#USING 'COPY' here instead of 'ADD' because of https://stackoverflow.com/questions/24958140/what-is-the-difference-between-the-copy-and-add-commands-in-a-dockerfile
From folder D:\Programs> I run command docker build . -t test
But I get error:
COPY failed: file not found in build context or excluded by .dockerignore: stat test: file does not exist
I tried commands COPY D:/test to /root/test and COPY ./test to /root/test
I know this error occurs because the path I copy from has to be relative from the build context (the . in docker build .). It cannot be an arbitrary path on the system outside of the build context.
I thought by using . I'd be in context D:\ from my build context D:\Programs>, but I guess not. What should my COPY command look like?
I checked here already:
https://forums.docker.com/t/copy-files-from-windows-host-to-ubuntu-container/28757
Dockerfile COPY from a Windows file system to a docker container
Dockerfile: Copy directory from Windows host to docker container
UPDATE 1
I placed the test folder in D:\Programs where my Dockerfile also resides, so I now have structure:
D:\Programs
\Test
Dockerfile
I then ran the build command again where I tried COPY test to /root/test and COPY ./test to /root/test, but both fail with the same aforementioned error.
I assume you've only added to in COPY D:/test to /root/test for abbreviation but I recommend checking that anyway.
If you really need what's in D:/Programs and can't move it to a folder inside the current build context there are two general options as I see it:
changing the build context - I'd suggest setting the build context as D:/ and adding every folder in the D:/ directory other than Programs to the .dockerignore file, then you can reference the Dockerfile from some subdirectory somewhere in D:/ and copy your files at build time as you wanted
creating a bind mount in runtime - you can map the D:/Programs directory to a folder inside the container (/my-programs) at runtime then copy the files from that folder to /root/test or just use the bind mount folder (/my-programs)
If in the future it will be available to use bind mounts in build time you'd be able to use the second option while creating the docker image
The only way I know how to do this is to change the dockerfile location during build, keeping the copied folder in context.
As below:
'test' folder located at D:/test
Set dockerfile COPY command as such:
COPY test /root/test
Say dockerfile is at D:/Programs/dockerfile
Navigate to D:/ in CLI
docker build -f ./Programs/dockerfile .
The . keeps the "test" folder in the build context. As far as I know, you cannot go to a parent directory etc. with docker build, so instead you want to stay in the parent folder and go down to the dockerfile.
If you run the docker build ... from D:\Programs that folder is the docker build context. All files used in the Dockerfile must be there.
d:\files is outside d:\programs, so it will be never be found.
You need to copy files content to the places where you run the command and you can use this copy line
COPY samplefile1.txt /root/test
I'm building a new image and copy contents from host OS folder D:\Programs\scrapy into it like so: docker build . -t scrapy
Dockerfile
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir scrapy
COPY scrapy to /root/scrapy
Now when I add new contents to the host OS folder "D:\Programs\scrapy" I want to also add it to image folder "root/scrapy", but I DON'T want to build a completely new image (it takes quite a while).
So how can I keep the existing image and just overwrite the contents of the image folder "root/scrapy".
Also: I don't want to copy the new contents EACH time I run the container (so NOT at run-time), I just want to have a SEPARATE command to add more files to an existing image and then run a new container based on that image at another time.
I checked here: How to update source code without rebuilding image (but not sure if OP tries to do the same as me)
UPDATE 1
Checking What is the purpose of VOLUME in Dockerfile and docker --volume format for Windows
I tried the commands below, all resulting in error:
docker: Error response from daemon: invalid volume specification: ''. See 'docker run --help'.
Where <pathiused> is for example D:/Programs/scrapy:/root/scrapy
docker run -v //D/Programs/scrapy:/root/scrapy scrapy
docker run -v scrapy:/root/scrapy scrapy
docker run -it -v //D/Programs/scrapy:/root/scrapy scrapy
docker run -it -v scrapy:/root/scrapy scrapy
UPDATE WITH cp command based on #Makariy's feedback
docker images -a gives:
REPOSITORY TAG IMAGE ID CREATED SIZE
scrapy latest e35e03c8cbbd 29 hours ago 5.71GB
<none> <none> 2089ad178feb 29 hours ago 5.71GB
<none> <none> 6162a0bec2fc 29 hours ago 5.7GB
<none> <none> 116a0c593544 29 hours ago 5.7GB
mcr.microsoft.com/windows/servercore ltsc2019 d1724c2d9a84 5 weeks ago 5.7GB
I run docker run -it scrapy and then docker container ls which gives:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1fcda458a14c scrapy "c:\\windows\\system32…" About a minute ago Up About a minute thirsty_bassi
If I run docker cp D:\Programs\scrapy scrapy:/root/scrapy I get:
Error: No such container:path: scrapy:\root
So in a separate PowerShell instance I then run docker cp D:\Programs\scrapy thirsty_bassi:/root/scrapy whichs show no output in PowerShell whatsoever, so I think it should've done something.
But then in my container instance when I goto /root/scrapy folder I only see the files that were already added when the image was built, not the new ones I wanted to add.
Also, I think I'm adding files to the container here, but is there no way to add it to the image instead? Without rebuilding the whole image?
UPDATE 2
My folder structure:
D:\Programs
Dockerfile
\image_addons
Dockerfile
\scrapy
PS D:\Programs>docker build . -t scrapybase
Successfully built 95676d084e28
Successfully tagged scrapybase:latest
PS D:\Programs\image_addons> docker build -t scrapy .
Step 2/2 : COPY scrapy to /root/scrapy
COPY failed: file not found in build context or excluded by .dockerignore: stat to: file does not exist
Dockerfile A
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
WORKDIR /root/scrapy
Dockerfile B
FROM scrapybase
COPY scrapy to /root/scrapy
You also can use docker cp, to manually copy files from your host to running container
docker cp ./path/to/file containername:/another/path
Docs
answer if you want it quick and dirty
docker run -it -v c:/programs/test:/root/test ubuntu:latest cat /root/test/myTestFile.txt
to update one file quickly:
If you don't have to build your code (I don't know what language you are using) you can build some base image with the initial code and when you want to change only one file (again I'm assuming you don't need to compile your project again for that, otherwise if you do that is not possible to due the nature of compiled programming language):
FROM previous-version-image:latest
COPY myfile dest/to/file
then because your CMD and ENTRYPOINT are saved from the previous stages no need to declare them. (if you don't remember use docker history <docker-image-name> to view virtual dockerfile for image to this stage).
Notice though not to repetitively use this method or you'll get a very big image with many useless layers. Use this only for quick testing and debugging.
explanation
Usually people use it for frontend development on docker containers but the basic idea persists, you create the basic working image with the dependencies installed and the directory layout setup with the last Dockerfile command being the development server start command.
example:
Dockerfile:
# pull the base image
FROM node:slim
# set the working directory
WORKDIR /app
# add `/app/node_modules/.bin` to $PATH
ENV PATH /app/node_modules/.bin:$PATH
# copy dependencies files
COPY package.json ./
COPY package-lock.json ./
# install app dependencies
RUN npm install
# add app
COPY . ./
# start development server
CMD ["npm", "start"]
startup command:
docker run -it --rm \
-v ${PWD}:/app \ <mount current working directory in host to container in path /app>
-v /app/node_modules \ <or other dependency directory if exists>
-p 80:3000 \ <ports if needs exposing>
ps-container:dev
I'm not sure if that use case will 100% work for you because it needs the code to be mounted using bind-mount all the time and when needed to be exported will have to be exported as the image and the source code directory, on the other hand, it allows you to make quick changes without waiting for the image to be built each time you add something new and in the end build the final image that contains all that's needed.
more relatable example to question provided code:
As you can see there is a file on the host machine that contains some text
the command that uses bind-mount to have access to the file:
docker run -it -v c:/programs/test:/root/test ubuntu:latest cat /root/test/myTestFile.txt
hope you find something that works for you from what I've provided here.
thanks to this tutorial and this example for starting examples and information.
EDIT:
Let's say your original Dockerfile looks like this:
FROM python:latest
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD python /app/app.py
This will build your initial image on top of we'll add layers and change the python files.
The next Dockerfile we'd use (let's call it Dockerfile.fix file) would copy the file we want to change instead of the ones already in the image
FROM previous-image-name
COPY app.py .
Now with after building with this Dockerfile the final image Dockerfile would look (sort of) like so:
FROM python:latest
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD python /app/app.py
FROM previous-image-name
COPY app.py .
And each time we'll want to change the file we'll use the second Dockerfile
There's no way you can change a Docker image without (at least partially) rebuilding it. But you don't have to rebuild all of it, you can just rebuild the layer copying your scrapy content.
You can optimize your build to have two images:
First image is your static image you don't want to rebuild each time. Let's call it scrapy-base.
Second and final image is based on first image scrapy-base and will only exist for the purpose of copying your dynamic scrapy content
scrapy-base's Dockerfile:
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir scrapy
And build it like:
docker build -t scrapy-base .
This command only needs to be run once. You won't have to build this image if you only change the content of local scrapy folder. (as you can see, the build does not use it at all)
scrapy's Dockerfile:
FROM scrapy-base
COPY scrapy /root/scrapy
With build command:
docker build -t scrapy .
This second build command will re-use the previous static image and only copy content without having to rebuild the entire image. Even with lots of files it should be pretty quick. You don't need to have a running container.
For your scenario :
docker run -v D:/test:/root/test your-image
A lots of valuable details available in this thread
I have a docker container inside which I have jar file that is of my springBoot application . How can I overwrite the application.property file of my app. Suppose if I want to change datasource url ,how can I do that through commandline?
Run docker ps and note the initials of id of your container.
Run docker exec -it <id> <bash or sh>, you will be inside your running container.
Go to your jar directory. Place your properties file in config/ as application.properties.
Rerun your app, but make sure to do it in background, so you can exit docker.
You can't restart the primary process (PID 1). For that usecase you'll need external volume and restart the container on property file change.
Create a volume: docker volume create vol1, you will get /var/lib/docker/volumes/vol1/
Change group of directory sudo chgrp 1001 /var/lib/docker/volumes/vol1/_data
Add write access to group: sudo chmod g+w /var/lib/docker/volumes/vol1/_data
Check the permissions of the directory: sudo ls -al /var/lib/docker/volumes/vol1/. They should be: drwxrwxr-x
Next time you run your container, mount the folder containing jar file to your volume as: docker run -d --name=webApp1 --mount source=vol1,destination=/path/to/app -p port:port name
You can change the property file in /var/lib/docker/volumes/vol1/_data/ and restart container: docker restart container_name
Disclaimer: Replacing properties file outside jar is not considered a good practice and is highly discouraged. So, these steps are only recommended for testing.