I am using EB on AWS to deploy a dockerfile.
Currently I deploy to scripts:
The dockerfile and a run.sh file which starts a server.
The dockerfile roughly looks like this
FROM ubuntu:14.04
MAINTAINER xy
[...install a java server...]
ADD run.sh /run.sh
RUN chmod +x /*.sh
EXPOSE 8080
CMD ["/run.sh"]
run.sh starts the java server.
I would like to set the --no-cache flag for the docker. Where can I set that?
You can't specify docker build's --no-cache because eb doesn't allow you to.
A workaround is to build the image locally (using --no-cache). Then use docker push to publish your image to Docker hub public registry.
Your Dockerfile could be simplified (untested) down to:
FROM custom_java_server_build:latest
MAINTAINER xy
EXPOSE 8080
CMD ["/run.sh"]
It does sound like you're creating a large image, you might be able to mitigate this by turning the entire install sequence into a single RUN statement. Don't forget to delete all your temporary files too.
You use --no-cache only in the docker build step. If the run script doesn't build the image, then you need to find what is building it and tell that process to use no-cache.
Related
I'm building a new image and copy contents from host OS folder D:\Programs\scrapy into it like so: docker build . -t scrapy
Dockerfile
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir scrapy
COPY scrapy to /root/scrapy
Now when I add new contents to the host OS folder "D:\Programs\scrapy" I want to also add it to image folder "root/scrapy", but I DON'T want to build a completely new image (it takes quite a while).
So how can I keep the existing image and just overwrite the contents of the image folder "root/scrapy".
Also: I don't want to copy the new contents EACH time I run the container (so NOT at run-time), I just want to have a SEPARATE command to add more files to an existing image and then run a new container based on that image at another time.
I checked here: How to update source code without rebuilding image (but not sure if OP tries to do the same as me)
UPDATE 1
Checking What is the purpose of VOLUME in Dockerfile and docker --volume format for Windows
I tried the commands below, all resulting in error:
docker: Error response from daemon: invalid volume specification: ''. See 'docker run --help'.
Where <pathiused> is for example D:/Programs/scrapy:/root/scrapy
docker run -v //D/Programs/scrapy:/root/scrapy scrapy
docker run -v scrapy:/root/scrapy scrapy
docker run -it -v //D/Programs/scrapy:/root/scrapy scrapy
docker run -it -v scrapy:/root/scrapy scrapy
UPDATE WITH cp command based on #Makariy's feedback
docker images -a gives:
REPOSITORY TAG IMAGE ID CREATED SIZE
scrapy latest e35e03c8cbbd 29 hours ago 5.71GB
<none> <none> 2089ad178feb 29 hours ago 5.71GB
<none> <none> 6162a0bec2fc 29 hours ago 5.7GB
<none> <none> 116a0c593544 29 hours ago 5.7GB
mcr.microsoft.com/windows/servercore ltsc2019 d1724c2d9a84 5 weeks ago 5.7GB
I run docker run -it scrapy and then docker container ls which gives:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1fcda458a14c scrapy "c:\\windows\\system32…" About a minute ago Up About a minute thirsty_bassi
If I run docker cp D:\Programs\scrapy scrapy:/root/scrapy I get:
Error: No such container:path: scrapy:\root
So in a separate PowerShell instance I then run docker cp D:\Programs\scrapy thirsty_bassi:/root/scrapy whichs show no output in PowerShell whatsoever, so I think it should've done something.
But then in my container instance when I goto /root/scrapy folder I only see the files that were already added when the image was built, not the new ones I wanted to add.
Also, I think I'm adding files to the container here, but is there no way to add it to the image instead? Without rebuilding the whole image?
UPDATE 2
My folder structure:
D:\Programs
Dockerfile
\image_addons
Dockerfile
\scrapy
PS D:\Programs>docker build . -t scrapybase
Successfully built 95676d084e28
Successfully tagged scrapybase:latest
PS D:\Programs\image_addons> docker build -t scrapy .
Step 2/2 : COPY scrapy to /root/scrapy
COPY failed: file not found in build context or excluded by .dockerignore: stat to: file does not exist
Dockerfile A
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
WORKDIR /root/scrapy
Dockerfile B
FROM scrapybase
COPY scrapy to /root/scrapy
You also can use docker cp, to manually copy files from your host to running container
docker cp ./path/to/file containername:/another/path
Docs
answer if you want it quick and dirty
docker run -it -v c:/programs/test:/root/test ubuntu:latest cat /root/test/myTestFile.txt
to update one file quickly:
If you don't have to build your code (I don't know what language you are using) you can build some base image with the initial code and when you want to change only one file (again I'm assuming you don't need to compile your project again for that, otherwise if you do that is not possible to due the nature of compiled programming language):
FROM previous-version-image:latest
COPY myfile dest/to/file
then because your CMD and ENTRYPOINT are saved from the previous stages no need to declare them. (if you don't remember use docker history <docker-image-name> to view virtual dockerfile for image to this stage).
Notice though not to repetitively use this method or you'll get a very big image with many useless layers. Use this only for quick testing and debugging.
explanation
Usually people use it for frontend development on docker containers but the basic idea persists, you create the basic working image with the dependencies installed and the directory layout setup with the last Dockerfile command being the development server start command.
example:
Dockerfile:
# pull the base image
FROM node:slim
# set the working directory
WORKDIR /app
# add `/app/node_modules/.bin` to $PATH
ENV PATH /app/node_modules/.bin:$PATH
# copy dependencies files
COPY package.json ./
COPY package-lock.json ./
# install app dependencies
RUN npm install
# add app
COPY . ./
# start development server
CMD ["npm", "start"]
startup command:
docker run -it --rm \
-v ${PWD}:/app \ <mount current working directory in host to container in path /app>
-v /app/node_modules \ <or other dependency directory if exists>
-p 80:3000 \ <ports if needs exposing>
ps-container:dev
I'm not sure if that use case will 100% work for you because it needs the code to be mounted using bind-mount all the time and when needed to be exported will have to be exported as the image and the source code directory, on the other hand, it allows you to make quick changes without waiting for the image to be built each time you add something new and in the end build the final image that contains all that's needed.
more relatable example to question provided code:
As you can see there is a file on the host machine that contains some text
the command that uses bind-mount to have access to the file:
docker run -it -v c:/programs/test:/root/test ubuntu:latest cat /root/test/myTestFile.txt
hope you find something that works for you from what I've provided here.
thanks to this tutorial and this example for starting examples and information.
EDIT:
Let's say your original Dockerfile looks like this:
FROM python:latest
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD python /app/app.py
This will build your initial image on top of we'll add layers and change the python files.
The next Dockerfile we'd use (let's call it Dockerfile.fix file) would copy the file we want to change instead of the ones already in the image
FROM previous-image-name
COPY app.py .
Now with after building with this Dockerfile the final image Dockerfile would look (sort of) like so:
FROM python:latest
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD python /app/app.py
FROM previous-image-name
COPY app.py .
And each time we'll want to change the file we'll use the second Dockerfile
There's no way you can change a Docker image without (at least partially) rebuilding it. But you don't have to rebuild all of it, you can just rebuild the layer copying your scrapy content.
You can optimize your build to have two images:
First image is your static image you don't want to rebuild each time. Let's call it scrapy-base.
Second and final image is based on first image scrapy-base and will only exist for the purpose of copying your dynamic scrapy content
scrapy-base's Dockerfile:
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir scrapy
And build it like:
docker build -t scrapy-base .
This command only needs to be run once. You won't have to build this image if you only change the content of local scrapy folder. (as you can see, the build does not use it at all)
scrapy's Dockerfile:
FROM scrapy-base
COPY scrapy /root/scrapy
With build command:
docker build -t scrapy .
This second build command will re-use the previous static image and only copy content without having to rebuild the entire image. Even with lots of files it should be pretty quick. You don't need to have a running container.
For your scenario :
docker run -v D:/test:/root/test your-image
A lots of valuable details available in this thread
so i have downloaded Docker Desktop and until now i have tested out containers and stuff just executing regular commands (docker ps, docker images..., docker run...) inside my zsh terminal and it works fine but now i am in a position where i want to create a directory inside docker host so that i can put my dockerfile inside, but if i run mkdir directory-name it is going to create the directory inside my mac not docker! so what command can i use to indicate that i want the directory to be created on docker not on my own mac machine?
While your docker container is running, you can start a new shell session inside using docker exec.
docker exec -it mycontainer bash
-i means interactive - so you can type
-t allocates a pseudo-TTY - just know that you need the argument
Then inside this bash, you can create folders and files all you want and they will be placed inside your running container. Note that whenever you remove the container (e.g. to update its image), these changes will be entirely lost. For persistence, use docker volumes.
Say you have the following directory structure:
.
├── Dockerfile
└── simple-web-app
Your Dockerfile:
FROM scratch
ADD simple-web-app simple-web-app
Then you would run
docker build .
I have a deploy script that I am trying to use for my server for CD but I am running into issues writing the bash script to complete some of my required steps such as running npm and the migration commands.
How would I go about getting into a container bash, from this script, running the commands below and then exiting to finish pulling up the changes?
Here is the script I am trying to automate:
cd /Project
docker-compose -f docker-compose.prod.yml down
git pull
docker-compose -f docker-compose.prod.yml build
# all good until here because it opens bash and does not allow more commands to run
docker-compose -f docker-compose.prod.yml run --rm web bash
npm install # should be run inside of web bash
python manage.py migrate_all # should be run inside of web bash
exit # should be run inside of web bash
# back out of web bash
docker-compose -f docker-compose.prod.yml up -d
Typically a Docker image is self-contained, and knows how to start itself up without any user intervention. With some limited exceptions, you shouldn't ever need to docker-compose run interactive shells to do post-deploy setup, and docker exec should be reserved for emergency debugging.
You're doing two things in this script.
The first is to install Node packages. These should be encapsulated in your image; your Dockerfile will almost always look something like
FROM node
WORKDIR /app
COPY package*.json .
RUN npm ci # <--- this line
COPY . .
CMD ["node", "index.js"]
Since the dependencies are in your image, you don't need to re-install them when the image starts up. Conversely, if you change your package.json file, re-running docker-compose build will re-run the npm install step and you'll get a clean package tree.
(There's a somewhat common setup that puts the node_modules directory into an anonymous volume, and overwrites the image's code with a bind mount. If you update your image, it will get the old node_modules directory from the anonymous volume and ignore the image updates. Delete these volumes: and use the code that's built into the image.)
Database migrations are a little trickier since you can't run them during the image build phase. There are two good approaches to this. One is to always have the container run migrations on startup. You can use an entrypoint script like:
#!/bin/sh
python manage.py migrate_all
exec "$#"
Make this script be executable and make it be the image's ENTRYPOINT, leaving the CMD be the command to actually start the application. On every container startup it will run migrations and then run the main container command, whatever it may be.
This approach doesn't necessarily work well if you have multiple replicas of the container (especially in a cluster environment like Docker Swarm or Kubernetes) or if you ever need to downgrade. In these cases it might make more sense to manually run migrations by hand. You can do that separately from the main container lifecycle with
docker-compose run web \
python manage.py migrate_all
Finally, in terms of the lifecycle you describe, Docker images are immutable: this means that it's safe to rebuild new images while the old ones are running. A minimum-downtime approach to the upgrade sequence you describe might look like:
git pull
# Build new images (includes `npm install`)
docker-compose build
# Run migrations (if required)
docker-compose run web python manage.py migrate_all
# Restart all containers
docker-compose up --force-recreate
I needed to create a Docker image of a Springboot application and I achieved that by creating a Dockerfile and building it into an image. Then, I used "docker run" to bring up a container. This container is used for all the activities for which my application was written.
My problem, however, is that the JAR file that I have used needs constant changes and that requires me to rebuild the Docker image everytime. Furthermore, I need to take the contents of the earlier running Docker container and transfer it into a container created from the newly built image.
I know this whole process can be written as a Shell script and exected every time I have changes on my JAR file. But, is there any tool I can use to somehow automate it in a simple manner?
Here is my Dockerfile:
FROM java:8
WORKDIR /app
ADD ./SuperApi ./SuperApi
ADD ./config ./config
ADD ./Resources ./Resources
EXPOSE 8000
CMD java -jar SuperApi/SomeName.jar --spring.config.location=SuperApi/application.properties
If you have a JAR file that you need to copy into an otherwise static Docker image, you can use a bind mount to save needing to rebuild repeatedly. This allows for directories to be shared from the host into the container.
Say your project directory (the build location where the JAR file is located) on the host machine is /home/vishwas/projects/my_project, and you need to have the contents placed at /opt/my_project inside the container. When starting the container from the command line, use the -v flag:
docker run -v /home/vishwas/projects/my_project:/opt/my_project [...]
Changes made to files under /home/vishwas/projects/my_project locally will be visible immediately inside the container1, so no need to rebuild (and probably no need to restart) the container.
If using docker-compose, this can be expressed using a volumes stanza under the services listing for that container:
volumes:
- type: bind
source: /home/vishwas/projects/my_project
target: /opt/my_project
This works for development, but later on, it's likely you'll want to bundle the JAR file into the image instead of sharing from the host system (so it can be placed into production). When that time comes, just re-build the image and add a COPY directive to the Dockerfile:
COPY /home/vishwas/projects/my_project /opt/my_project
1: Worth noting that it will default to read/write, so the container will also be able to modify your project files. To mount as read-only, use: docker run -v /home/vishwas/projects/my_project:/opt/my_project:ro
You are looking for docker compose
You can build and start containers with a single command using compose.
I'm a beginner in working with docker especially docker compose. Currently, creation my initial easy docker environment, I run into the first error and I've no clue why.
I tried to search for a solution in stackoverflow but found nothing that could help me.
Starting my docker with "docker-compose up" I get the following error:
$ docker-compose up
Removing errorinstance_app_1
Recreating 8a358dfcb306_8a358dfcb306_8a358dfcb306_errorinstance_app_1 ...
Recreating 8a358dfcb306_8a358dfcb306_8a358dfcb306_errorinstance_app_1 ... error
ERROR: for 8a358dfcb306_8a358dfcb306_8a358dfcb306_errorinstance_app_1 Cannot start service app: oci runtime error: container_linux.go:265: starting container process caused "exec: \"./run.sh\": stat ./run.sh: no such file or directory"
ERROR: for app Cannot start service app: oci runtime error: container_linux.go:265: starting container process caused "exec: \"./run.sh\": stat ./run.sh: no such file or directory"
ERROR: Encountered errors while bringing up the project.
So. Following my folder structure:
Project
docker-compose.yml
Docker
Java
Dockerfile
src
run.sh
Following my docker-compose.yml:
version: '2'
services:
app:
build:
dockerfile: ./Docker/Java/Dockerfile
context: .
volumes:
- ./src:/usr/local/etc/
working_dir: /usr/local/etc/
command: ./run.sh
And following my docker file:
FROM java:7-jdk-alpine
# WORKDIR /usr/local/etc
run.sh
echo "Hello world."
Yes, I know that I could do that solution only in a docker-compose file. But in the future I need to extend the Dockerfile.
Can someone help me respectively does anyone see the issue?
The problem is with the base docker image you are using in dockerfile:
FROM java:7-jdk-alpine
You are trying to start container by running run.sh bash script. But the above image doesn't support bash itself
For reference, you can see the documentation of above image in docker hub page here. Quoting the necessary portion here:
java:alpine
...
To minimize image size, it's uncommon for additional related tools
(such as git or bash) to be included in Alpine-based images. Using
this image as a base, add the things you need in your own Dockerfile
(see the alpine image description for examples of how to install
packages if you are unfamiliar).
That's about the problem.
Now, I can think of 2 solutions:
Just use java:7-jdk as base image instead of java:7-jdk-alpine
Install bash on top of the base image java:7-jdk-alpine by changing dockerfile to:
FROM java:7-jdk-alpine
RUN apk update && apk upgrade && apk add bash
#WORKDIR /usr/local/etc
*source of steps to install bash in alpine linux is here
It looks like docker compose can't find your run.sh file. This file needs to be included in your docker image.
Change your Dockerfile to the following, then rebuild the image with docker build -t <YOUR_IMAGE_NAME> ..
FROM java:7-jdk-alpine
ADD run.sh /usr/local/etc/run.sh
Once your image is rebuilt, run docker-compose up again.
The easiest way to tackle the problem is to execute a bash session in the container, then inside the container, you have to check if the file exists in the
indicated path if the file is not in the path, it must be included when you create the image into the docker file or through a volume inside de docker-compose.
Another thing to check is the relative path you are using. It will be clear when you check the existence of the file inside de docker container
docker exec -it CONTAINER_NAME bash
I recommend you to create a volume in the docker compose file, as it is the easier way, and also the best way.
there is a question that I want to do you, why are you putting the Dockerfile file inside a Java path?
It is not a good idea o guideline to follow
The correct way is to put your dockerfile file into an environment folder, in such a way the dockerfile file is not related to the java source of your application
I got this Error quite a lot and after a lot of investigation, it looked like some images were corrupted.
Deleting those and rebuilding solve the problem. It was not docker installation or configuration itself.