Question
How can I see symlinks of docker linux-containers from a windows host? (Even if I have to place an intermediate linux machine exposing the filesystem via NFS or Samba)
Context
In a DEVEL environment, I have this structure in a certain remote filesystem in a Linux within the office:
/files/repos/app-1
/files/repos/app-2
/files/repos/lib-x
/files/repos/lib-y
both app-1 and app-2 use those libraries which are vendored and symlinked like this:
/files/repos/app-1/vendor/my-company/lib-x => /files/repos/lib-x
/files/repos/app-1/vendor/my-company/lib-y => /files/repos/lib-y
/files/repos/app-2/vendor/my-company/lib-x => /files/repos/lib-x
/files/repos/app-2/vendor/my-company/lib-y => /files/repos/lib-y
The developers need to be in Windows.
So the developers have their IDE pointing to some mounted unit, for example Z:\ where they see all the repos and projects.
This allows us the following:
Edit any of the projects from it's own folder, and run the unit-tests for that project, including running the lib-x and lib-y.
Develope any of the libraries and have them updated in the depending applications (note I say I am in DEVEL, not PRE or PROD).
From the IDE, pointing see the "complete structure" of any of the applications (for instance app-1) also see the classes of the lib-x and lib-y so the autocompletion and so works perfectly.
This has been working like this for nearly a decade and works perfectly.
Problems
The developers need the connection to the server to develop and we wanted to mutate to local dockers so we can make the devels work from home.
Going to docker
We now decided that we are not going to use anymore the office-servers and we are going to setup all the development within docker containers.
What does actually work
We just installed docker desktop in Windows and shared C:\repos from the host into the dockers.
We now have some devel machines FROM ubuntu:xxx and run them mounting the volumes.
We made the symlinks within the app-1 and app-2 to lib-x and lib-y from inside the linux containers.
This does work perfectly and also the repositories work fine if we run the applications in the local dockers
Problem with symlinks in linux container and windows host
The problem is now the IDE: While it reads the files in C:\repos\app-1, the symlink that has been created within the linux containers can't be seen from the host.
This makes the IDE to be unable to follow C:\repos\app-1\vendor\lib-x and all the code-completion helpers are broken.
I already know Windows does not support symlink compatible with linux symlinks.
This forces us to look for an alternate solution.
Solution we've though with Samba
Initially I thought that as well as in the old topology a linux server just shared the filesystem via samba and the windows could just read the symlinks contents as they were demapped at the serverside and not the clientside, I thought that I could run another docker machine with a samba server just to locally share the "things seen from the linux" into the Windows host again.
To do so, I setup this docker-compose:
version: "3.7"
services:
samba:
container_name: samba
hostname: samba
image: dperson/samba
volumes:
- //c/Users/xavi/Documents/repos/test_samba:/mount
ports:
- "139:139"
- "445:445"
command: samba.sh -s "test_samba;/mnt/repos/test_samba;yes;no;yes;all"
restart: always
But this conflicts as 445 is locally already used.
If I turn down the local SMB, then in the next reboot, docker is unable to share C:\ into docker (I was not consciuos it does this sharing via SMB, could it be turned into a NFS or so?)
If I map to another port, like 10445:445 then the client is unable to access it, as client samba ports in windows seem to be not configurable.
Mapping an IP
So I tried to map an IP:
version: "3.7"
services:
samba:
container_name: samba
hostname: samba
image: dperson/samba
volumes:
- //c/Users/xavi/Documents/repos/test_samba:/mount
ports:
- "139:139"
- "192.168.4.83:445:445"
command: samba.sh -s "test_samba;/mnt/repos/test_samba;yes;no;yes;all"
restart: always
networks:
samba:
ipv4_address: 192.168.4.83
networks:
samba:
ipam:
driver: default
config:
- subnet: "192.168.4.0/16"
But is seems that this still creates problems:
It seems the IP is only for internal docker networking but not seen from the host
It seems the original service still listens not to 127.0.0.1:445 but to 0.0.0.0:445 so still "blocking" the attachment to listen to 192.168.4.83:445
So question
How could I make a windows host to see the "demapped contents of symlinks" to make the IDE see the vendored content that is linked from inside docker linux containers?
TL;DR
Run git-bash as administrator.
Issue export MSYS=winsymlinks:nativestrict in git-bash.
From there on, ln -s works in windows.
Links are seen from inside the docker.
Details
We'll walk thru these steps:
Preparation: Prepare a temporary dir with some files within the abc directory.
See it fail: We'll try to make a symlink and see it fail.
Create symlink: We'll create the symlink in windows and see it. We'll point xyz to abc.
Run docker: We'll then run docker with ubuntu and change contents in xyz.
Check in ubuntu container: We'll see the changes also in abc from within the docker.
Check in windows host: Well check both abc and xyz from ouside the container.
1. Preparation
In a git-bash go to /c and create a temporary dir tmp.
Inside it, create an abc dir and throw some contents there.
cd /c
mkdir tmp
cd tmp/
mkdir abc
cd abc/
echo 1111 > old_1
echo 2222 > old_2
echo 3333 > old_3
Here's a sample session:
2. See it fail
First let's try the "normal" way and see it fail.
In a git-bash, navigate to /c/tmp
Then do a symlink making xyz to point to abc: ln -s abc xyz
See it fails, by ls-ing the tmpand see xyz is a regular dir.
To be sure, create new content in xyz and see it's not there in abc.
Try to create the link. It will not become a symlink, but rather create a copy of the directory.
cd /c/tmp/
ln -s abc xyz
Create new_bad in xyz and don't see it in abc.
cd xyz/
touch new_bad
cd ../abc/
ls -l
Clear the wrong xyz
rm -Rf xyz/
Here's a sample session:
3. Create symlink
Here it comes the real stuff. The inspiration comes from #Slayvin's answer here, as well as here Git Bash shell fails to create symbolic links and the official git-for-windows repo here https://github.com/git-for-windows/git/pull/156
First open a new git-bash in Administrator mode. The reason is that only admins can create links in windows.
Once you are a CLI admin, navigate to the destination and set this evironment variable:
export MSYS=winsymlinks:nativestrict
This will tell the runtime subsytem of git-bash to actually use the symlinks feature. As we are admins we'll succeed.
The do just "normal symlinks" as you would expect: ln -s abc xyz
It works!!! Now next move is to test within docker!
NOTE: As per Sebastian's answer here https://stackoverflow.com/a/40914277/1315009 you DON'T need to be administrator to create symlinks in git-bash if you enabled the developer tools. In the search-bar write for developers and enable it:
4. Run docker + 5. Check in docker
The bash with admin privileges is no longer needed. So we'll close it and re-instantiate a "normal" bash.
In it, run an ubuntu continainer with docker. Use -it to interact with the ubuntu's bash. Use winpty to allow -it to work.
Bind-mount the /c/tmp directory so both abc and xyz are reachable. I chose to mount it to /files.
From inside, cd /files and see that xyz is actually a symlink.
Create some new content in xyz
Run and see:
winpty docker run -it --rm --mount type=bind,source="c:\tmp",target=/files --name ubuntu-link ubuntu
cd /files/
ls -l
Create content:
cd xyz
echo "yeaaahh" > new_good
Check it's really a symlink by going to abc:
cd ..
cd abc/
cat new_good
Sample session:
6. Check in windows host
Step out from the docker. Stay in the git-bash.
Again: This git-bash does not need to be privileged. The only moment we had to be admin was to "create" the symlink in windows.
From the unprivileged bash, explore abc as well as xyz and see that there's the content we created from inside the docker, appearing in both the original directory and in the symlink.
Sample session:
Final check
We can finally go to a classical CMD to see how it looks like. We can see it's clearly indicated that it's a symlink for a directory and we also see the target there:
Golden touch
If you have the "developer tools" activates as stated above, the only missing thing is the ENV VAR.
We can set this by editing the .bashrc at your windows home:
By doing this we can just use git-bash completely normally and start creating the symlinks from windows without any overload.
Caution
The symlinks created this way work from windows and are seen from inside docker. But not the oposite. If you create symlinks inside the container they don't get created in windows.
Therefore, in mounted volumes, setup the symlinks always from git-bash and consume them from the container. If you create them from the container, they still can be consumed from the container. But won't be usable from windows.
Conclussion
It can be done fully from the linux flavour commands via git-bash. Only that you need to be admin to create the links and tell the git-bash runtime to use that feature. And that the link needs to be done from windows, instead from inside the ubuntu.
I encountered a similar problem with my setup: developing on Windows 10 (where both the IDE and Docker are running), and having the website running inside the container (Linux).
I used to work on a library that is required by the website, working on both projects in parallel. And to do so, the library directory was symlinked (in host/Windows) in the vendor path.
Something like:
+ my-website
↪ vendor
↪ company
↪ my-package (->symlink here)
↪ ...
↪ docker-compose.yml
+ external-packages
↪ company
↪ my-package (real files here)
But with Docker, that setup doesn't work anymore.
So the trick is to mount a volume in docker-compose like this:
volumes:
- ./:/my-app
- ../external-packages/company:/my-app/vendor/company
So the files in vendor are 'seen' by the web server (inside the container), and we can keep the symlink (made in windows) between the my-package folders, so the IDE sees them as well.
I hope this will help you.
Related
I am new to docker. I currently have docker-compose.yml where volume I want to mount my current directory to '/usr/share/data' in the container. So something like:
volumes:
- ${PWD}/data:/usr/share/data/
I am using windows with a Linux Subsystem and want to know why isn't it able to get the PWD variable. The same code on a Linux machine works fine. Please tell me how to set the environment PWD. Do I need to do it manually or can i do it using the same docker-compose file. Using . instead of ${PWD} gives no directory error and I have seen many forums saying that it is a windows problem.
You need to use ${PWD} instead of {$PWD}. You can also use dot . instead of ${PWD}.
In order to share Windows folders with Docker containers when running, you first need to configure the Shared Drives option in Docker settings:
Right click on docker app > Settings > Shared drives > Check D:
Then . and ${PWD} should work in compose:
volumes:
- ./data:/usr/share/data/
I've reviewed the documentation here:
https://docs.docker.com/docker-for-mac/install/#install-and-run-docker-for-mac
It doesn't say anything about boot2docker, although some other questions along these lines talk about this:
Mount volume to Docker image on OSX
So the question is – the Docker for Mac application provides File Sharing via Preferences -> File Sharing; how does one make use of these shared folders from the docker image (for example if one ssh's into the docker image)? When I say how, I don't mean "what are the use-cases", I mean "please show me an example of how to access a shared folder from the command line of the running container".
Ideally I'm trying to create a similar scenario to Vagrant's synched folders whereby I can edit files on my Host env, independently of the Docker Image but these are updated automatically to the Docker image on save.
UPDATE:
To be clear, the reason for asking this question is because I couldn't get the -v docker command to work. E.g.
docker run -v /Users/geoidesic/Documents/projects/arc/mysite/djangocms_demo:/home/djangocms/djangocms/djangocms_demo -d -p 8001:8000 --name test_shared_volumes bluszcz/djangocms
With the above command the container immediately stops, so if I run docker ps the list of running containers is empty.
However, if I run the container without the -v command, then it stays running as expected:
docker run -d -p 8001:8000 --name test_shared_volumes bluszcz/djangocms
Updated:
Well, if you want to share file/directory between host and container, you're gonna use Docker's bind-mount.
For example, if I want to share my host's /etc/resolv.conf to my container, I do the following:
docker run -v /etc/resolv.conf:/etc/resolv.conf <IMAGE>
In which the -v ... part tells the container to reuse host's /etc/resolve.conf. And whenever I edit this file, the changes will be immediately visible to the container.
In Linux, you can use this way to share almost any of your host files to containers. Unfortunately, this is not the case for Mac. As I mentioned in my old answer, by default you can only share /Users/, /Volumes/, /private/, and /tmp directly.
On my Mac, saying, I want to share the /data directory to a container. I run below command:
docker run -it --rm -v /data:/data busybox sh
Then it pops up an unhappy error:
docker: Error response from daemon: Mounts denied:
The path /data
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
So you see, this is where File Sharing comes up.
Then comes my answers to your questions:
File Sharing does not provide you a ready-to-use way to do the sharing as you have experienced in Vagrant;
To share file/folder between host and container, use Dockers bind-mount.
Hope that helps.
Old answer:
File Sharing is used by Docker's bind-mount feature. By default, you can bind-mount files in /Users/, /Volumes/, /private/, and /tmp directly. For other paths, you need to add them to Preferences -> File Sharing first.
Use cases for bind-mount:
Persisting data generated by the running container, so that you can backup or migrate data.
Sharing data amount multiple running containers.
Share host configuration files to containers.
Share source code between host and containers, to make debugging easier.
Note: For cases #1 and #2, consider using volumes instead of bind-mount.
I installed docker toolbox 1.11.2 and Laradock v.2 cloned from GitHub.
Everything seems to work except the laradock_workspace_1. When is generated it does not create files on the host machine (Windows 7 64-bit). In the docker-compose.yml I have tried playing with the volumes as suggested here
### Laravel Application Code Container ######################
volumes_source:
build: ./volumes/application
volumes:
- ../:/var/www/laravel
If I change the last line to ../.. then run docker-compose up, docker exec -it laradock_workspace_1 ls and I can see that it is traversing the folders on the host machine. I just don't see any files.
My goal here is to make the actual Laravel code external so I can edit them on the host machine and use git.
I can use the Kitematic app to make the changes I want but they seem lost if I do a docker-compose down. (and I get errors about things still being in use.)
I'm new to docker so any help is appreciated.
First, make sure your docker-machine is running. If it is, then follow below:
Open up Virtualbox GUI and right click your docker vm, and select settings, then go to Shared Folders.
Change the c\users to whatever folder your code lies in, like this:
This will mount your desired folder to /c/Users in the docker-machine vm.
After this, change the docker-compose.yml in the laradock folder to this:
### Laravel Application Code Container ######################
volumes_source:
build: ./volumes/application
volumes:
- /c/Users/pomodoro.xyz/code:/var/www/laravel
The logic behind this is, since we are running the docker in a VM, the docker-compose command looks for folder in the VM, not in the windows machines. Thats why we have provided the VM machine path to the docker-compose file.
For the sake of simplicity, use ubuntu image as an example.
I often find it easier to use docker-compose, particularly if there's a high chance I'll want to both mount-volumes and link the container to another container at some point in the future.
Create a folder for working in, say "ubuntu".
In the "ubuntu" folder, create another folder called "files"
Create a file in that folder called "docker-compose.yml". In this file, enter:
ubuntucontainer:
image: "ubuntu:latest"
ports:
- "80:80"
volumes:
- ./files:/files
Whenever you need to start the box, navigate to "ubuntu" and type docker-compose up. To stop again, use docker-compose stop.
The advantage of using docker compose is that if you ever want to link-up a database container this can be done easily by adding another container to the yaml file, and then in the ubuntucontainer container adding a links section.
Not to mention, docker-compose up is quite minimal on the typing.
(Also, forwarding the ports with 80:80 may not be strictly necessary, it depends on what you want the box to do.)
TL;DR version:
Open Docker Quickstart Terminal. If it is already open, run $ cd ~
Run this once: $ docker run -it -v /$(pwd)/ubuntu:/windows --name ubu ubuntu
To start every time: $ docker start -i ubu
You will get an empty folder named ubuntu in your Windows user directory. You will see this folder with the name windows in your ubuntu container.
Explanation:
cd ~ is for making sure you are in Windows user directory.
-it stands for interactive, so you can interact with the container in the terminal environment. -v host_folder:container_folder enables sharing a folder between the host and the container. The host folder should be inside the Windows user folder. /$(pwd) translates to //c/Users/YOUR_USER_DIR in Windows 10. --name ubu assigns the name ubu to the container.
-i stands for interactive
How can I access Docker containers Folder and files from Windows file explorer?
If you are running Docker Desktop on Windows, Docker containers don't run natively on the local filesystem, but instead on a hyper-v virtual machine or via WSL2.
Hyper-v (legacy)
In theory, if you were to stop the hyper-v vm, you could open up the vhdx, and if you had the right filesystem drivers, mount it and see the files inside. This is not possible to do while the virtual machine is running. By default the OS that runs for Linux container mode is named "Docker Desktop", but runs busybox.
The file could be found here:
C:\ProgramData\DockerDesktop\vm-data\DockerDesktop.vhdx
WSL2 (modern)
WSL things are slightly different, but not much. You are still effectively working with a virtual environment.
One of the nice advantages of WSL however, is that you can actually browse this file system naively with Windows Explorer.
By browsing to \\wsl$ you will be able to see the file systems of any distributions you have, including docker-desktop.
The docker filesystems on my machine seem to live in:
\\wsl$\docker-desktop-data\version-pack-data\community\docker\overlay2
However, the overlay 'merged' view, which shows the original file system with your changes, doesn't seem to work via windows explorer and gives you a blank window. You can however still see the 'diff' folder, which contains your changes.
You can open a terminal to either of these instances by using the wsl command, from powershell.
Access via Docker
If you wanted to have a look at this Docker OS and filesystem, one way would be to spin up a container, that has access to the OS at the root, something like:
docker run -it --mount type=bind,source=/,target=/host ubuntu /bin/bash
This should drop you into a Ubuntu docker container, with a Bash terminal, which has the root of the hyper-v container (/), mounted on the path '/host'. Looking inside, you will find the Busybox filesystem of the virtual machine that is running docker, and all the containers.
Due to how docker runs, you will be able to access the filesystems of each container. If you are using the overlay2 filesystem for you containers, you would likely find the filesystem layers here for each container:
/host/var/lib/docker/overlay2
If the files you want to browse through in windows explorer, you should be able to configure a samba export of this folder, that is accessible from the host machine, that is accessible while this container is running.
If the goal however is to be able to browse/edit files on the local OS, and have them update inside the container, normally the easiest way to do this, is to mount local directory into the container. This can be done similar to the example above, but you first need to go into the Docker Desktop settings, and enable the mounting of the shared drive into the host virtual machine, and then provide the volume argument when you spin up a container.
If you are using WSL2, there are a few more options available to you, as you can keep your projects inside the WSL layer, while interacting with them from the host OS or via docker. Best practice for this is still in flux, so I'm going to avoid giving direct advice here.
Another related question's reply answers this: https://stackoverflow.com/a/64418064/1115220
\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\
I'll give WordPress app as an example by showing a sample of the docker-compose.yaml file. In order to have project files shown in windows from docker container, you'll need to use ports and volumes
Notice volume and ports.
port 8000 from the local machine maps to 80 within the container.
as for volume, ./ current directory on windows maps to the container image files.
wordpress:
depends_on:
- db
image: wordpress:latest
volumes: ['./:/var/www/html']
ports:
- "8000:80"
restart: always
environment:
WORDPRESS_DB_HOST: db
WORDPRESS_DB_USER: wordpress
WORDPRESS_DB_PASSWORD: wordpress
WORDPRESS_DB_NAME: wordpress
When running Windows container on Windows Docker Desktop, I was able to see all image files here:
C:\ProgramData\Docker\windowsfilter
(requires admin rights to access, and it would be unwize to delete/modify anything there)
Further, with WizTree tool, it's easy to see real sizes of each image layer and even find which specific files contribute to layer's size.
You should use a mount volume. In your docker run .... command, you may specify a mount volume. The syntax is as follows:
-v /host/directory:/container/directory
An example:
docker run -it -v C:\Users\thomas\Desktop:/root/home --name my_container image1
This would allow the container to write files to /root/home and have them appear on the user thomas' desktop