Several articles have been extremely helpful in understanding Docker's volume and data management. These two in particular are excellent:
http://container-solutions.com/understanding-volumes-docker/
http://www.alexecollins.com/docker-persistence/
However, I am not sure if what I am looking for is discussed. Here is my understanding:
When running docker run -v /host/something:/container/something the host files will overlay (but not overwrite) the container files at the specified location. The container will no longer have access to the location's previous files, but instead only have access to the host files at that location.
When defining a VOLUME in a Dockerfile, other containers may share the contents created by the image/container.
The host may also view/modify a Dockerfile volume, but only after discovering the true mountpoint using docker inspect. (usually somewhere like /var/lib/docker/vfs/dir/cde167197ccc3e138a14f1a4f7c....). However, this is hairy when Docker has to run inside a Virtualbox VM.
How can I reverse the overlay so that when mounting a volume, the container files take precedence over my host files?
I want to specify a mountpoint where I can easily access the container filesystem. I understand I can use a data container for this, or I can use docker inspect to find the mountpoint, but neither solution is a good solution in this case.
The docker 1.10+ way of sharing files would be through a volume, as in docker volume create.
That means that you can use a data volume directly (you don't need a container dedicated to a data volume).
That way, you can share and mount that volume in a container which will then keep its content in said volume.
That is more in line with how a container is working: isolating memory, cpu and filesystem from the host: that is why you cannot "mount a volume and have the container's files take precedence over the host file": that would break that container isolation and expose to the host its content.
Begin your container's script with copying files from a read-only mount bind reflecting the host files to a work location in the container. End the script with copying necessary results from the container's work location back to the host using either the same or different mount point.
Alternatively to the end-of-the script command, run the container without automatically removing it at the end, then run docker cp CONTAINER_NAME:CONTAINER_DIR HOST_DIR, then docker rm CONTAINER_NAME.
Alternatively to copying results back to the host, keep them in a separate "named" volume, provided that the container had it mounted (type=volume,src=datavol,dst=CONTAINER_DIR/work). Use the named volume with other docker run commands to retrieve or use the results.
The input files may be modified in the host during development between the repeated runs of the container. Avoid shadowing them with the frozen files in the named volume. Beginning the container script with copying the input files from the host may help.
Using a named volume helps running the container read-only. (One may still need --tmpfs /tmp for temporary files or --tmpfs /tmp:exec if some container commands create and run executable code in the temporary location).
Related
I am learning docker. In my image build process i create a file called "appsettings.json" in a folder called "Config". I want this file to be editable by the user outside the container. The final goal is that a user can stop the container, make changes to the settings file and start the container again with the new settings file.
I am using windows containers on a windows 10 host. I created a new volume first:
docker volume create myvolume
After that I tried to start my container
docker run -v myvolume:C:/app/Config
However, it seems that the -v argument deletes all content in the Config folder. I was aware that bind mounts override folders in the container with folders on the host, but I thought this named volume will copy the appsettings file to the host.
What I could do is creating the volume first, starting the container and copying the file from within the container into the volume, but this seems to be an annoying overkill.
Is there any easier way or best practice to make files which result as part of the build process visible to the host file system?
you can't mount files from your host machine into volumes. Like Pandey Amit already mentioned in the comments, you have to mount the Config directory of your host machine directly into the container. To achieve this in Windows 10, you have to grant the access to your filesystem in the docker settings first.
Open Docker Dashboard -> Settings -> Resources -> FILE SHARING + add here the directory, which should be mounted in your Docker containers (fyi: all the subdirectories are included)
I recommend to restart docker itself, to take note of the new settings.
Once docker has restarted, run your container:
docker run -v C:/app/Config:/path/to/your/app/Config <image-name>
This will mount the content of C:\app\Config into the container and now you should be able to modify the content of appsettings.json even without having to restart your container (but this depends on the architecture of your application -> if it supports a live reload of appsettings.json)
UPDATE: you cannot mount files from your docker image on a host machine. if you want, you could instruct the user to create the file manually on the host machine and follow with the steps described above. But if the only thing you want achieve, is to allow the user to override application settings I would recommend you to work with environment variables. For example run container with overwritten setting from the host machine:
docker run -e "foo=bar" <image-name>
I am wondering if it is possible to RUN a remote file stored in an NFS share when building an image from a dockerfile.
Currently I am using the COPY command and then the RUN command to execute the files, however many of the files I need to create the image are extremely large.
Is it possible to execute files stored in an NFS share directly in the dockerfile without having to copy them all over?
You can only RUN files inside your container - so it needs to copied to your container.
What you can do is move the COPY commands to the beginning of your Dockerfile so that they are cached and don't need to be copied every time you change a command later in the Dockerfile.
You can RUN curl.... to grab the remote file ,then execute it sure.
But this will only run at image build time, not during lifecycle of the container
You could also mount the NFS volume to your host, then COPY the files.
Otherwise, remote execution is a pretty basic security flaw and shouldn't be possible under any circumstances
I've reviewed the documentation here:
https://docs.docker.com/docker-for-mac/install/#install-and-run-docker-for-mac
It doesn't say anything about boot2docker, although some other questions along these lines talk about this:
Mount volume to Docker image on OSX
So the question is – the Docker for Mac application provides File Sharing via Preferences -> File Sharing; how does one make use of these shared folders from the docker image (for example if one ssh's into the docker image)? When I say how, I don't mean "what are the use-cases", I mean "please show me an example of how to access a shared folder from the command line of the running container".
Ideally I'm trying to create a similar scenario to Vagrant's synched folders whereby I can edit files on my Host env, independently of the Docker Image but these are updated automatically to the Docker image on save.
UPDATE:
To be clear, the reason for asking this question is because I couldn't get the -v docker command to work. E.g.
docker run -v /Users/geoidesic/Documents/projects/arc/mysite/djangocms_demo:/home/djangocms/djangocms/djangocms_demo -d -p 8001:8000 --name test_shared_volumes bluszcz/djangocms
With the above command the container immediately stops, so if I run docker ps the list of running containers is empty.
However, if I run the container without the -v command, then it stays running as expected:
docker run -d -p 8001:8000 --name test_shared_volumes bluszcz/djangocms
Updated:
Well, if you want to share file/directory between host and container, you're gonna use Docker's bind-mount.
For example, if I want to share my host's /etc/resolv.conf to my container, I do the following:
docker run -v /etc/resolv.conf:/etc/resolv.conf <IMAGE>
In which the -v ... part tells the container to reuse host's /etc/resolve.conf. And whenever I edit this file, the changes will be immediately visible to the container.
In Linux, you can use this way to share almost any of your host files to containers. Unfortunately, this is not the case for Mac. As I mentioned in my old answer, by default you can only share /Users/, /Volumes/, /private/, and /tmp directly.
On my Mac, saying, I want to share the /data directory to a container. I run below command:
docker run -it --rm -v /data:/data busybox sh
Then it pops up an unhappy error:
docker: Error response from daemon: Mounts denied:
The path /data
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
So you see, this is where File Sharing comes up.
Then comes my answers to your questions:
File Sharing does not provide you a ready-to-use way to do the sharing as you have experienced in Vagrant;
To share file/folder between host and container, use Dockers bind-mount.
Hope that helps.
Old answer:
File Sharing is used by Docker's bind-mount feature. By default, you can bind-mount files in /Users/, /Volumes/, /private/, and /tmp directly. For other paths, you need to add them to Preferences -> File Sharing first.
Use cases for bind-mount:
Persisting data generated by the running container, so that you can backup or migrate data.
Sharing data amount multiple running containers.
Share host configuration files to containers.
Share source code between host and containers, to make debugging easier.
Note: For cases #1 and #2, consider using volumes instead of bind-mount.
I have a container that I start like
docker run -it --mount type=bind,source=/path/to/my/data,target=/staging -v myvol:/myvol buildandoid bash -l
It has two mounts, one bind mount that I use to get data into the container, and one named volume that I use to persist data. The container is used as a reproducable android (AOSP) build environment, so not your typical web service.
I would like to access the files on myvol from the Windows host. If I use an absolute path for the mount, e.g. -v /c/some/path:/myvol, I can do that, but I believe docker creates copies of all the files and keeps them in sync. I really want to avoid creating these files on the windows side (for space reasons, as it is several GB, and performance reasons, since NTFS doesn't seem to handle many little files well).
Can I somehow "mount" a container directory or a named volume on the host? So the exact reverse of a bind mount. I think alternatively I could install samba or sshd in the container, and use that, but maybe there is something built into docker / VirtualBox to achive this.
Use bind mounts.
https://docs.docker.com/engine/admin/volumes/bind-mounts/
By contrast, when you use a volume, a new directory is created within Docker’s storage directory on the host machine, and Docker manages that directory’s contents.
How can I access Docker containers Folder and files from Windows file explorer?
If you are running Docker Desktop on Windows, Docker containers don't run natively on the local filesystem, but instead on a hyper-v virtual machine or via WSL2.
Hyper-v (legacy)
In theory, if you were to stop the hyper-v vm, you could open up the vhdx, and if you had the right filesystem drivers, mount it and see the files inside. This is not possible to do while the virtual machine is running. By default the OS that runs for Linux container mode is named "Docker Desktop", but runs busybox.
The file could be found here:
C:\ProgramData\DockerDesktop\vm-data\DockerDesktop.vhdx
WSL2 (modern)
WSL things are slightly different, but not much. You are still effectively working with a virtual environment.
One of the nice advantages of WSL however, is that you can actually browse this file system naively with Windows Explorer.
By browsing to \\wsl$ you will be able to see the file systems of any distributions you have, including docker-desktop.
The docker filesystems on my machine seem to live in:
\\wsl$\docker-desktop-data\version-pack-data\community\docker\overlay2
However, the overlay 'merged' view, which shows the original file system with your changes, doesn't seem to work via windows explorer and gives you a blank window. You can however still see the 'diff' folder, which contains your changes.
You can open a terminal to either of these instances by using the wsl command, from powershell.
Access via Docker
If you wanted to have a look at this Docker OS and filesystem, one way would be to spin up a container, that has access to the OS at the root, something like:
docker run -it --mount type=bind,source=/,target=/host ubuntu /bin/bash
This should drop you into a Ubuntu docker container, with a Bash terminal, which has the root of the hyper-v container (/), mounted on the path '/host'. Looking inside, you will find the Busybox filesystem of the virtual machine that is running docker, and all the containers.
Due to how docker runs, you will be able to access the filesystems of each container. If you are using the overlay2 filesystem for you containers, you would likely find the filesystem layers here for each container:
/host/var/lib/docker/overlay2
If the files you want to browse through in windows explorer, you should be able to configure a samba export of this folder, that is accessible from the host machine, that is accessible while this container is running.
If the goal however is to be able to browse/edit files on the local OS, and have them update inside the container, normally the easiest way to do this, is to mount local directory into the container. This can be done similar to the example above, but you first need to go into the Docker Desktop settings, and enable the mounting of the shared drive into the host virtual machine, and then provide the volume argument when you spin up a container.
If you are using WSL2, there are a few more options available to you, as you can keep your projects inside the WSL layer, while interacting with them from the host OS or via docker. Best practice for this is still in flux, so I'm going to avoid giving direct advice here.
Another related question's reply answers this: https://stackoverflow.com/a/64418064/1115220
\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\
I'll give WordPress app as an example by showing a sample of the docker-compose.yaml file. In order to have project files shown in windows from docker container, you'll need to use ports and volumes
Notice volume and ports.
port 8000 from the local machine maps to 80 within the container.
as for volume, ./ current directory on windows maps to the container image files.
wordpress:
depends_on:
- db
image: wordpress:latest
volumes: ['./:/var/www/html']
ports:
- "8000:80"
restart: always
environment:
WORDPRESS_DB_HOST: db
WORDPRESS_DB_USER: wordpress
WORDPRESS_DB_PASSWORD: wordpress
WORDPRESS_DB_NAME: wordpress
When running Windows container on Windows Docker Desktop, I was able to see all image files here:
C:\ProgramData\Docker\windowsfilter
(requires admin rights to access, and it would be unwize to delete/modify anything there)
Further, with WizTree tool, it's easy to see real sizes of each image layer and even find which specific files contribute to layer's size.
You should use a mount volume. In your docker run .... command, you may specify a mount volume. The syntax is as follows:
-v /host/directory:/container/directory
An example:
docker run -it -v C:\Users\thomas\Desktop:/root/home --name my_container image1
This would allow the container to write files to /root/home and have them appear on the user thomas' desktop