Is there any easy way to convert a Dockerfile to a Bash script in order to install all the software on a real OS? The reason is that docker container I can not change and I would like afterwards change few things if they did not work out.
In short - no.
By parsing the Dockerfile with a tool such as dockerfile-parse you could run the individual RUN commands, but this would not replicate the Dockerfile's output.
You would have to be running the same version of the same OS.
The ADD and COPY commands affect the filesystem, which is in its own namespace. Running these outside of the container could potentially break your host system. Your host will also have files in places that the container image would not.
VOLUME mounts will also affect the filesytem.
The FROM image (which may in turn be descended from other images) may have other applications installed.
Writing Dockerfiles can be a slow process if there is a large installation or download step. To mitigate that, try adding new packages as a new RUN command (to take advantage of the cache) and add features incrementally, only optimising/compressing the layers when the functionality is complete.
You may also want to use something like ServerSpec to get a TDD approach to your container images and prevent regressions during development.
Best practice docs here, gotchas and the original article.
Basically you can make a copy of a Docker container's file system using “docker export”, which you can then write to a loop device:
docker build -t <YOUR-IMAGE> ...
docker create --name=<YOUR-CONTAINER> <YOUR-IMAGE>
dd if=/dev/zero of=disk.img bs=1 count=0 seek=1G
mkfs.ext2 -F disk.img
sudo mount -o loop disk.img /mnt
docker export <YOUR-CONTAINER> | sudo tar x -C /mnt
sudo umount /mnt
Convert a Docker container to a raw file system image.
More info here:
http://mr.gy/blog/build-vm-image-with-docker.html
You can of course convert a Dockerfile to bash script commands. Its just a matter of determining what the translation means. All docker installs, apply changes to a "file system layer" and that means all changes can be implemented in a real OS.
An example of this process is here:
https://github.com/thatkevin/dockerfile-to-shell-script
It is an example of how you would do the translation.
you can install application inside dockerfile like this
FROM <base>
RUN apt-get update -y
RUN apt-get install <some application> -y
Related
When running my Kali Linux docker container headerless-kali-linux without mounting volumes using the command:
docker run --rm -it headerless-kali-linux
The bash terminal within the container has colours and aesthetic features, which I want. They look like this:
However, when I mount my volumes to the container with the command:
docker run --rm -it -v %cd%\root:/root -v %cd%\postgresql:/var/lib/postgresql headerless-kali-linux
The aesthetic features go, as if volumes should somehow affect that, and I'm left with plain bash in my terminal. If anyone has any idea why this is happening, please let me know! Thanks in advance!
why this is happening
You are mounting /root so you are overwriting shell startup files, that could set up the configuration.
This problem seems to be fixed when creating docker volumes as opposed to using a local source in the C drive. You can also create a container first, not remove it, see its ID with docker ps -a and then populate local directories with docker cp <container ID>:/path/to/dir /path/to/local/volume. From here you can attach the volumes to the local sources with the -v /path/to/local/volume:/path/to/dir option when running.
I am doing a lot of media related tasks using content that is processed in different Docker containers and I would like some guidance on how to create a simple, streamlined workflow. Specifically I am not sure when to use shell scripts and when to use Dockerfiles.
The common sequence of events is:
Convert an input video to a sequence of images
Process that sequence of images
Convert the processed images back to a video
Save this video on the host
Each of these is run with a different docker run command. I want to script these processes so that I do not have to sit at the command line and type docker run image_name etc. etc. every single time I want to do this.
Step 1. is run as the following:
docker run -v $PWD:/temp/ jrottenberg/ffmpeg -i /temp/$SRC_VIDEO_DIR/$FILENAME /temp/$OUTPUT_IMAGE_DIR/$OUTPUT_IMAGE_BASENAME%06d.bmp
Step 2. is run as the following:
docker run -v $PWD/$OUTPUT_IMAGE_DIR:/notebook/raisr/test -v $PWD/$OUTPUT_LARGER_IMAGE_DIR:/notebook/raisr/results $DOCKER_IMAGE /bin/bash -c "source activate superres; cd /notebook/raisr; python test.py"
Step 3. is run as the following:
docker run -v $PWD:/temp/ jrottenberg/ffmpeg -framerate $FPS -i /temp/$OUTPUT_IMAGE_DIR/$OUTPUT_IMAGE_BASENAME%06d.bmp -c:v qtrle -pix_fmt rgb24 /temp/output_video/$OUTPUT_FILENAME
The easiest way to do this seems like to create a shell script with those three commands in them and just run that script. The annoying thing with that is that I have to edit the shell script and change the input path of the video file each time. Also calling 3 commands in Step 2. does not seem like a common practice.
I realize that I could use the RUN option in a Dockerfile to specify that those commands are automatically run when the docker file is built, but mounting volumes via a Dockerfile and getting data out of the running container and to the host has been a large pain. And some of these commands depend on a Docker image being installed locally, with a specific name (the variable $DOCKER_IMAGE in step 2. above.
So:
When you have multiple docker run commands, each relying on each others data, what is the correct, scalable way to run these commands?
When do people use shell scripts and when do people use Dockerfiles to manage multiple jobs?
Is it good practice to have a Dockerfile run a job that processes media or should this be done from a shell script as a docker run command?
I have a container which I am using interactively (docker run -it), in it, i have to run a pretty common set of commands, though not always in a set order, hence I cannot just run a script.
Thus, I would like for a way to have my commands in recursive search (Ctrl+R) be available in the Docker container.
Any idea how I can do this?
Let's mount the history file into the container from the host so it's contains will get preserved the container death.
# In some directory
touch bash_history
docker run -v ./bash_history:/root/.bash_history:Z -it fedora /bin/bash
I would recommend to have separate bash history to the one that you use on the host for the safety reasons.
I found helpful info in these questions:
Docker and .bash_history
Docker: preserve command history
https://superuser.com/questions/1158739/prompt-command-to-reload-from-bash-history
They use docker volume mounts however, which mean that the container commands affect the local (host PC) commands, which I do not want.
It seems I will have to copy ~/.bash_history from local into container which will make the history work 'one-way'.
UPDATE: Working:
COPY your_command_script.sh some_folder/my_history
ENV HISTFILE myroot/my_history
RUN PROMPT_COMMAND="history -a; history -r"
Explanation:
copy command script into a file in container
tell the shell to look at a different file for history
reload the history file
I would like to clone a dockerized application including all its data, which uses three containers in this example: 1) a web application container such as a CMS, 2) a database container and 3) a data-volume container (using docker volumes).
With docker-compose, I can easily create identical instances of these containers with just the initial data. But what, if I want to clone a set of running containers on the same server, including all its accumulated data, in a similar way as I would clone a KVM container? With KVM I would suspend or shutdown the VM, clone with something like virt-clone and then start the cloned guest, which has all the same data as the original.
One use case would be to create a clone/snapshot of a running development web-server before making major changes or before installing new versions of plugins.
With Docker, this does not seem to be so straightforward, as data is not automatically copied together with its container. Ideally I would like to do something simple like docker-compose clone and end up with a second set of containers identical to the first, including all their data. Neither Docker nor docker-compose provides a clone command (as of version 1.8), thus I would need to consider various approaches, like backing up & restoring the data/database or using a third party tool like Flocker.
Related to this is the question on how to do something similar to KVM snapshots of a dockerized app, with the ability to easily return to a previous state. Preferably the cloning, snapshotting and reverting should be possible with minimal downtime.
What would be the preferred Docker way of accomplishing these things?
Edit: Based on the first answer, I will make my question a little more specific in order to hopefully arrive at programmatic steps to be able to do something like docker-compose-clone and docker-compose-snapshot using a bash or python script. Cloning the content of the docker volumes seems to be the key to this, as the containers themselves are basically cloned each time I run docker-compose on the same yaml file.
Generally my full-clone script would need to
duplicate the directory containing the docker-compose file
temporarily stop the containers
create (but not necessarily run) the second set of containers
determine the data-volumes to be duplicated
backup these data-volumes
restore the data-volumes into the cloned data container
start the second set of containers
Would this be the correct way to go about it and how should I implement this? I'm especially not sure on how to do step 4 (determine the data-volumes to be duplicated) in a script, as the command docker volume ls will only be available in Docker 1.9.
How could I do something similar to KVM snapshots using this approach? (possibly using COW filesystem features from ZFS, which my Docker install is already using).
With docker you would keep all of your state in volumes. Your containers can be recreated from images as long as they re-use the same volumes (either from the host or a data-volume container).
I'm not aware of an easy way to export volumes from a data-volume container. I know that the docker 1.9 release is going to be adding some top-level apis for interacting with volumes, but I'm not sure if export will be available immediately.
If you're using a host volume, you could manage the state externally from docker.
Currently, I'm using the following script to clone a dockerized CMS web-application Concrete5.7, based on the approach outlined above. It creates a second set of identical containers using docker-compose, then it backs up just the data from the data volumes, and restores it to the data containers in the second set.
This could serve as an example for developing a more generalised script:
#!/bin/bash
set -e
# This script will clone a set of containers including all its data
# the docker-compose.yml is in the PROJ_ORIG directory
# - do not use capital letters or underscores for clone suffix,
# as docker-compose will modify or remove these
PROJ_ORIG="c5app"
PROJ_CLONE="${PROJ_ORIG}003"
# 1. duplicate the directory containing the docker-compose file
cd /opt/docker/compose/concrete5.7/
cp -Rv ${PROJ_ORIG}/ ${PROJ_CLONE}/
# 2. temporarily stop the containers
cd ${PROJ_ORIG}
docker-compose stop
# 3. create, run and stop the second set of containers
# (docker-compose does not have a create command)
cd ../${PROJ_CLONE}
docker-compose up -d
docker-compose stop
# 4. determine the data-volumes to be duplicated
# a) examine which containers are designated data containers
# b) then use docker inspect to determine the relevant directories
# c) store destination directories & process them for backup and clone
#
# In this appliaction we use two data containers
# (here we used DATA as part of the name):
# $ docker-compose ps | grep DATA
# c5app_DB-DATA_1 /true Exit 0
# c5app_WEB-DATA_1 /true Exit 0
#
# $ docker inspect ${PROJ_ORIG}_WEB-DATA_1 | grep Destination
# "Destination": "/var/www/html",
# "Destination": "/etc/apache2",
#
# $ docker inspect ${PROJ_ORIG}_DB-DATA_1 | grep Destination
# "Destination": "/var/lib/mysql",
# these still need to be determined manually from examining
# the docker-compose.yml or using the commands in 4.
DATA_SUF1="_WEB-DATA_1"
VOL1_1="/etc/apache2"
VOL1_2="/var/www/html"
DATA_SUF2="_DB-DATA_1"
VOL2_1="/var/lib/mysql"
# 5. Backup Data:
docker run --rm --volumes-from ${PROJ_ORIG}${DATA_SUF1} -v ${PWD}:/clone debian tar -cpzf /clone/clone${DATA_SUF1}.tar.gz ${VOL1_1} ${VOL1_2}
docker run --rm --volumes-from ${PROJ_ORIG}${DATA_SUF2} -v ${PWD}:/clone debian tar -cpzf /clone/clone${DATA_SUF2}.tar.gz ${VOL2_1}
# 6. Clone Data:
# existing files in volumes need to be deleted before restoring,
# as the installation may have created additional files during initial run,
# which do not get overwritten during restore
docker run --rm --volumes-from ${PROJ_CLONE}${DATA_SUF1} -v ${PWD}:/clone debian bash -c "rm -rf ${VOL1_1}/* ${VOL1_2}/* && tar -xpf /clone/clone${DATA_SUF1}.tar.gz"
docker run --rm --volumes-from ${PROJ_CLONE}${DATA_SUF2} -v ${PWD}:/clone debian bash -c "rm -rf ${VOL2_1}/* && tar -xpf /clone/clone${DATA_SUF2}.tar.gz"
# 7. Start Cloned Containers:
docker-compose start
# 8. Remove tar archives
rm -v clone${DATA_SUF1}.tar.gz
rm -v clone${DATA_SUF2}.tar.gz
Its been tested and works, but still has the following limitations:
the data-volumes to be duplicated need to be determined manually and
the script needs to be modified, depending on the number of data-containers and data-volumes
there is no snap-shot/restore capability
I welcome any suggestions for improvements (especially step 4.). Or, if someone would come up with a different, better approach I would accept that as an answer instead.
The application used in this example, together with the docker-compose.yml file can be found here.
On Windows, there is a port of docker's open source container project available from Windocks that does what you need. There are two options:
Smaller sized databases are copied into containers via an Add database command specified while building the image. After that every container built from that receives the database automatically.
For large databases, there is a cloning functionality. The databases are cloned during the creation of containers and the clones are done in seconds even for terabyte size DBs. Deleting a container also removes the clone automatically. Right now its only available for SQL Server though.
See here for more details on the database adding and cloning.
using ulimit command, i set core file size.
ulimit -c unlimited
and I compiled c source code using gcc - g option.
then a.out generated.
after command
./a.out
there is runtime error .
(core dumped)
but core file was not generated.(ex. core.294340)
how to generated core file?
First make sure the container will write the cores to an existing location in the container filesystem. The core generation settings are set in the host, not in the container. Example:
echo '/cores/core.%e.%p' | sudo tee /proc/sys/kernel/core_pattern
will generate cores in the folder /cores.
In your Dockerfile, create that folder:
RUN mkdir /cores
You need to specify the core size limit; the ulimit shell command would not work, cause it only affects at the current shell. You need to use the docker run option --ulimit with a soft and hard limit. After building the Docker image, run the container with something like:
docker run --ulimit core=-1 --mount source=coredumps_volume,target=/cores ab3ca583c907 ./a.out
where coredumps_volume is a volume you already created where the core will persist after the container is terminated. E.g.
docker volume create coredumps_volume
If you want to generate a core dump of an existing process, say using gcore, you need to start the container with --cap-add=SYS_PTRACE to allow a debugger running as root inside the container to attach to the process. (For core dumps on signals, see the other answer)
I keep forgetting how to do it exactly and keep stumbling upon this question which provides marginal help.
All in all it is very simple:
Run the container with extra params --ulimit core=-1 --privileged to allow coredumps:
docker run -it --rm \
--name something \
--ulimit core=-1 --privileged \
--security-opt seccomp=unconfined \
--entrypoint '/bin/bash' \
$IMAGE
Then, once in container, set coredump location and start your failing script:
sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t
myfailingscript.a
Enjoy your stacktrace
cd /tmp; gdb -c `ls -t /tmp | grep core | tail -1`
Well, let's resurrect an ancient thread.
If you're running Docker on Linux, then all of this is controlled by /proc/sys/kernel/core_pattern on your raw metal. That is, if you cat that file on bare metal and inside the container, they'll be the same. Note also the file is tricky to update. You have to use the tee method from some of the other posts.
echo core | sudo tee /proc/sys/kernel/core_pattern
If you change it in bare metal, it gets changed in your container. So that also means that behavior is going to be based on where you're running your containers.
My containers don't run apport, but my bare metal did, so I wasn't getting cores. I did the above (I had already solved the ulimit -c thing), and suddenly I get core files in the current directory.
The key to this is understanding that it's your environment, your bare metal, that controls the contents of that file.