Environment:
Docker on windows
Code in questions: COPY configure_my_app.bat .
Expected: If I run docker build multiple times, I expect the image id to be same from various runs on docker build, given that there is no change in the Dockerfile
What happens: The image id after the above command changes for each run of docker build
I read it somewhere a while ago that this might have something to do with file attributes/permissions. I wonder, how can I get around this ? I want to use layers from cache if Dockerfile is not changed.
Related
I'm studing docker.
docker-compose is known as a role that conveniently runs multiple containers as one script.
First, since Dockerfile only handles one container, is it correct to think that Docker Compose is backwards compatible with Dockerfile?
I thought docker compose could cover everything, but I saw docker compose and docker files used together.
Let's take spring boot as an example.
Can I use only one docker-compose to run the db container required for the application, build the application, check the port being used, and run the jar file?
Or do I have to separate the dockerfils and roles and use the two?
When working with Docker, there are two concepts: Image and Container.
Images are like mini operating systems stored in a file which is built specifically with our application in mind. Think of it like a custom operating system which is sitting on your hard disk when your computer is switched off.
Containers are running instances of your image.
Imagine you had a shared hard disk (or even CD/DVD if you are old school) which had an operating system which can run on multiple machines. The files on the disk are the "image", and those files running on a machine are a "container".
This is how Docker works, you have files on the machine which are known as the image, and running instances of those files are referred to as the container. Images can also be uploaded and shared for other users to download and run on their machine too.
This brings us to Docker vs Docker Compose.
Docker is the underlying technology which manages (creates, stores or shares) images, and runs containers.
We provide the Dockerfile to tell Docker how to create our images. For example, we say: starts from the Python 3 base image, then install these requirements, then creates these folders, then switch to this user, etc... (I'm oversimplifying the actual steps, but this is just to explain the point).
Once we have done that, we can create an image using Docker, by running docker build .. If you run that, Docker will execute each step in our Dockerfile and store the result as an image on the system.
Once the image is build, you can run it manually using something like this:
docker run <IMAGE_ID>
However, if you need to setup volumes, you need to run them like this:
docker run -v /path/to/vol:/path/to/vol -p 8000:8000 <IMAGE_ID>
Often applications need multiple images to run. For example, you might have an application and a database, and you may also need to setup networks and shared volumes between them.
So you would need to write the above commands with the appropriate configurations and ID's for each container you want to run, every time you want to start your service...
As you might expect, this could become complex, tedious and difficult to manage very quickly...
This is where Docker Compose comes in...
Docker Compose is a tool used to define how Docker runs our images in a yaml file which is can be stored with our code and reused.
So, if we need to run our app image as a container, share port 8000 and map a volume, we can do this:
services:
app:
build:
context: .
ports:
- 8000:8000
volumes:
- app/:/app
Then, every time we need to start our app, we just run docker-compose up, and Docker Compose will handle all the complex docker commands behind the scenes.
So basically, the purpose of Docker Compose is to configure how our running service should work together to serve our application.
When we run docker-compose build, Docker Compose will run all the necessary docker build commands, to build all images needed for our project and tag them appropriately to keep track of them in the system.
In summary, Docker is the underlying technology used to create images and run them as containers, and Docker Compose is a tool that configures how Docker should run multiple containers to serve our application.
I hope this makes sense?
I suggest you to go deeper in what docker compose is reading at Difference between Docker Compose Vs Dockerfile.
I report part of what explained in the above article:
A Dockerfile is a simple text file that contains the commands a user could call to assemble an image whereas Docker Compose is a tool for defining and running multi-container Docker applications.
Docker Compose define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment. It get an app running in one command by just running docker-compose up.Docker compose uses the Dockerfile if one add the build command to your project’s docker-compose.yml. Your Docker workflow should be to build a suitable Dockerfile for each image you wish to create, then use compose to assemble the images using the build command.
About your question, I answer you that docker compose is so flexible that you can fragment your composition logic in multiple yaml files, and combining by the docker compose command line as you need.
Here an example:
# Build the docker infrastructure
docker-compose \
-f network.yaml \
-f database.yaml \
-f application.yaml
build
# Run the application
docker-compose \
-f application.yaml
up
to answer your question of spring boot application. you can build complete application through Compose alone as long as you know sequence and dependency but then question will arise are you using the compose power properly? as #Antonio Petricca already given well described answer for the compose
regarding compose file and DockerFile question: it depends on how you wrote your compose file. technically Both are different.
so in short:
1- Compose and DockerFile are two different things
2- compose can use multiple modular files even DockerFile and that is why it is so popular, you can break logic into multiple modules then use compose to build it. it also help you debug and iterate faster.
Hope its answer your doubt.
Hi I need to modify the a docker image from Autoware_AI repository after build it. The problem is:
A) I build the image running a .sh file:
cd $WORKING_DIRECTORY/docker/generic
./run.sh -t 1.14.0
It is specifically from Autoware: https://www.svlsimulator.com/docs/system-under-test/autoware-instructions/
B) I modify the scripts contained inside the packages contained in Autoware folder
C) When I exit the container, and later enter again the modifications are not there anymore, of coure, because the image is built from Dockerfile from scratch...
To find a solution I have tried 2 different approachs:
To modify the container and save it as described here: https://www.scalyr.com/blog/create-docker-image/
Issue: When using other terminal, trying to add .txt file for Autoware_AI running container, to modify the container, Autoware_AI container does not appear as active (but it is). Just other container are avaialable when I try to copy a file to Autoware_AI:
Commit Changes To a Docker image: https://phoenixnap.com/kb/how-to-commit-changes-to-docker-image
Issue: Problem to connect to Autoware_AI server and run the ros packages. This problem does not happen when building the original Docker file with .sh
The complete description of my problem as well as output of terminal attempts are better described here:
https://answers.ros.org/question/376512/fork-autoware_ai-repository-and-create-docker-image/?answer=376583#post-id-376583
https://get-help.robotigniteacademy.com/t/fork-autoware-ai-repository-and-create-docker-image/9533/4
I am kind of new in forking,changing docker images. I do not understand how to fix this, find a solution for create my custom docker image and make it functional.
Thanks very much in advance!
As David Maze suggested one feasible way would be change the Dockerfile, and then build my image from it. It is a good idea. In my case however additional steps were required, because I had a build.sh script that called different Dockerfiles to build the image, and besides that this build.sh file also installed some ROS packages and other dependencies.
Even if the build.sh needed to change. The modifications on Dockerfile contributed to solve 90% of the issue. My Dockerfile was:
enter image description here
And After the modification, the Dockerfile became:
enter image description here
To dowload the modified ROS codes to my image, instead of autoware repo ROS pkgs, I needed:
1 - Copy the autoware.ai.repos file from here:
https://raw.githubusercontent.com/Autoware-AI/autoware.ai/1.14.0/autoware.ai.repos
To my docker local folder (docker/generic) and unwrap them with vcs import command as the Dockerfile displays above...
2- Edit the autoware.ai.repos, in order to change the address of some of the repositories contained in autoware.ai.repos to my personal github:
I Removed the lines:
autoware/visualization:
type: git
url: https://github.com/Autoware-AI/visualization.git
version: 1.14.0
And replaced by:
autoware/visualization:
type: git
url: https://github.com/marcusvinicius178/visualization
Afterwards I followed the build instructions in case 3 here: https://github.com/Autoware-AI/autoware.ai/wiki/Generic-x86-Docker#run-an-autoware-docker-container
$ ./build.sh
$ ./run.sh -t local
I know that may exist a more professional way to work with Docker images, and build a new Dockerfile based on the original one. But I am not that expert in Docker and also in this way my problem was solved.
I needed to create a Docker image of a Springboot application and I achieved that by creating a Dockerfile and building it into an image. Then, I used "docker run" to bring up a container. This container is used for all the activities for which my application was written.
My problem, however, is that the JAR file that I have used needs constant changes and that requires me to rebuild the Docker image everytime. Furthermore, I need to take the contents of the earlier running Docker container and transfer it into a container created from the newly built image.
I know this whole process can be written as a Shell script and exected every time I have changes on my JAR file. But, is there any tool I can use to somehow automate it in a simple manner?
Here is my Dockerfile:
FROM java:8
WORKDIR /app
ADD ./SuperApi ./SuperApi
ADD ./config ./config
ADD ./Resources ./Resources
EXPOSE 8000
CMD java -jar SuperApi/SomeName.jar --spring.config.location=SuperApi/application.properties
If you have a JAR file that you need to copy into an otherwise static Docker image, you can use a bind mount to save needing to rebuild repeatedly. This allows for directories to be shared from the host into the container.
Say your project directory (the build location where the JAR file is located) on the host machine is /home/vishwas/projects/my_project, and you need to have the contents placed at /opt/my_project inside the container. When starting the container from the command line, use the -v flag:
docker run -v /home/vishwas/projects/my_project:/opt/my_project [...]
Changes made to files under /home/vishwas/projects/my_project locally will be visible immediately inside the container1, so no need to rebuild (and probably no need to restart) the container.
If using docker-compose, this can be expressed using a volumes stanza under the services listing for that container:
volumes:
- type: bind
source: /home/vishwas/projects/my_project
target: /opt/my_project
This works for development, but later on, it's likely you'll want to bundle the JAR file into the image instead of sharing from the host system (so it can be placed into production). When that time comes, just re-build the image and add a COPY directive to the Dockerfile:
COPY /home/vishwas/projects/my_project /opt/my_project
1: Worth noting that it will default to read/write, so the container will also be able to modify your project files. To mount as read-only, use: docker run -v /home/vishwas/projects/my_project:/opt/my_project:ro
You are looking for docker compose
You can build and start containers with a single command using compose.
I was following this post - the reference code is on GitHub. I have cloned the repository on my local.
The project has got a react app inside it. I'm trying to run it on my local following step 7 on the same post:
docker run -p 8080:80 shakyshane/cra-docker
This returns:
Unable to find image 'shakyshane/cra-docker:latest' locally
docker: Error response from daemon: pull access denied for shakyshane/cra-docker, repository does not exist or may require 'docker login'.
See 'docker run --help'.
I tried login to docker again but looks like since it belongs to #shakyShane I cannot access it.
I idiotically tried npm start too but it's not a simple react app running on node - it's in the container and containers are not controlled by npm
Looks like docker pull shakyshane/cra-docker:latest throws this:
Error response from daemon: pull access denied for shakyshane/cra-docker, repository does not exist or may require 'docker login'
So the question is how do I run this docker image on my local mac machine?
Well this is illogical but still sharing so future people like me don't get stuck.
The problem was that I was trying to run a docker image which doesn't exist.
I needed to build the image:
docker build . -t xameeramir/cra-docker
And then run it:
docker run -p 8080:80 xameeramir/cra-docker
In my case, my image had TAG specified with it and I was not using it.
REPOSITORY TAG IMAGE ID CREATED SIZE
testimage testtag 189b7354c60a 13 hours ago 88.3MB
Unable to find image 'testimage:latest' locally for this command docker run testimage
So specifying tag like this - docker run testimage:testtag worked for me
Posting my solution since non of the above worked.
Working on macbook M1 pro.
The issue I had is that the image was built as arm/64. And I was running the command:
docker run --platform=linux/amd64 ...
So I had to build the image for amd/64 platform in order to run it.
Command below:
docker buildx build --platform=linux/amd64 ...
In conclusion your docker image platform and docker run platform needs to be the same from what I experienced.
In my case, the docker image did exist on the system and still I couldn't run the container locally, so I used the exact image ID instead of image name and tag, like this:
docker run myContainer c29150c8588e
I received this error message when I typed the name/character wrong. That is, "name1\name2" instead of "name1/name2" (wrong slash).
In my case, I saw this error when I had logged in to the dockerhub in my docker desktop. The repo I was pulling was local to my enterprise. Once i logged out of dockerhub, the pull worked.
This just happened to me because my local docker vm on macos ran out of disk space.
I just deleted some old images using docker image prune and it started working correctly again.
shakyshane/cra-docker Does not exist in that user's repo https://hub.docker.com/u/shakyshane/
The problem is you are trying to run an imagen that does not exists. If you are executing a Dockerfile, the image was not created until Dockerfile pass with no errors; so when Dockerfile tries to run the image, it can't find it. Be sure you have no errors in the execution of your scripts.
The simplest answer can be the correct one!.. make sure you have permissions to execute the command, use:
sudo docker run -p 8080:80 shakyshane/cra-docker
In my case, I didn't realise there was a difference between docker run and docker start, and I kept using the run command when I should've been using the start command.
FYI, run is for building and creating the docker container, start is to just start a stopped container
Use -d
sudo docker run -d -p 8000:8000 rasa/duckling
learn about -d here
sudo docker run --help
At first, i build image on mac-m1-pro with this command docker build -t hello_k8s_world:0.0.1 ., when is run this image the issue appear.
After read Master Yi's answer, i realize the crux of the matter and rebuild my images like this docker build --platform=arm64 -t hello_k8s_world:0.0.1 .
Finally，it worked.
I'm new to Docker, so please allow me to describe the steps that I did. I'm using Docker (not Docker toolbox) on OS X. I built the image from Dockerfile using the following command
sudo docker build -t myImage .
Docker confirmed that building was successful.
Successfully built 7240e.....
However, I can't find the image anywhere. I looked at this question, but the answer is for Docker toolbox, and I don't have a folder /Users/<username>/.docker as suggested by the accepted answer.
You would be able to see your docker images by the below command:
docker images
And to check which all containers are running in docker:
docker ps -a
Local builds (in my case using buildkit) will create and cache the image layers but simply leave them in the cache rather than tell the docker daemon they're an actual image. To do that you need to use the --load flag.
$ docker buildx build -t myImage .
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
Doesn't show anything, but...
$ docker buildx build -t myImage --load .
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
myImage latest 538021e3d342 18 minutes ago 190MB
And there it is!
There actually is a warning about this in the output of the build command... but it's above all the build step logs so vanishes off your terminal without easily being seen.
To get list of Images
docker image ls
or
docker images
In addition to the correct responses above that discuss how to access your container or container image, if you want to know how the image is written to disk...
Docker uses a Copy on Write File System (https://en.wikipedia.org/wiki/Copy-on-write) and stores each Docker image as a series of read only layers and stores them in a list. The link below does a good job explaining how the image layers are actually stored on disk.
https://docs.docker.com/storage/storagedriver/
As already said, after the docker images
this command will show you all the images you have locally.
i.e "somth like that"
REPOSITORY TAG IMAGE ID CREATED SIZE
codestandars 1.0 a22daacf6761 8 minutes ago 622MB
bulletinboard 1.0 b73e8e68edc0 2 hours ago 681MB
ubuntu 18.04 cf0f3ca922e0 4 days ago 64.2MB
now you should
docker run -it and the IMAGE ID or the TAG related to the repository you want to run.
Command to list the docker images is :
docker images
The default docker images will show all top level images, their repository and tags, and their size.
An image will be listed more than once if it has multiple repository names or tags.
Click here for the screenshot for more details