Cannot create file with Jupyter Notebook running in Docker in Windows 10 - windows

I have installed Docker Toolbox for Windows v 18.09 in Windows 10 Version 10.0.19041 Build 19041 and am trying to run a docker container to run Jupiter Notebook with Pyspark.
I am using Windows Powershell to execute docker commands
docker run hello-world works fine so I can assume that Dockers has been installed correctly. Infact, I had go down to Toolbox v 18.09 before I could get hello world to work.
i use the following command to run the pyspark container
docker run -it --rm -p 8888:8888 --volume=//C/Users/prith/pydev://home/jovyan/work jupyter/pyspark-notebook
from the C:/Users/prith/pydev directory that is mapped to the work directory of the container. The // is slash is required because i am working with Windows. the notebook shows up in http://192.168.99.100:8888 as expected and I can login with the token.
Then the problem starts when i try to create a new notebook or even a text file, I get a permission denied error. Evidently the container cannot write to 'some' directory. I have used Windows filesystem properties to give Everyone all privileges on this particular directory and have also run the Powershell in Adminstrtor mode ( to simulate Ubuntu sudo ..) but nothing works.
Interestingly, I can write into the directory located above the work directory in the container but then I cannot access files written into that directory from Windows because I have mapped my local windows directory to /home/jovyan/work
What do what i want? I want to create Jupyter notebooks in the container and save them in Windows
I know all this works like a charm in Linux-Ubuntu, but unfortunately I am stuck with Windows 10. Please help.

It looks you forgot to add the directory you're trying to mount to FILE SHARING.
Please, do right-click on docker icon (in system tray) -> Settings -> Resources -> FILE SHARING
Then, add your local directory.
Finally, if it doesn't work, try to mount volume with --volume="C:\Users\prith\pydev":/home/joyvan/work

this command seems to work
docker run -it --rm -p 8888:8888 --volume='/c/Users/Public/PyDev'://home/jovyan/work jupyter/pyspark-notebook start-notebook.sh --NotebookApp.token=''

Related

How to change the location of docker installation? [duplicate]

I've just upgraded to Windows 10 Home May 2020, activated WSL2, and installed Docker Desktop.
WSL2 must be installed in my system disk, which is a small SSD. I don't want to fill it with docker images. How do I change the docker images path? I'd like to use a path in my big Windows filesystem.
The image location is somewhat confusing. I believe it is in /mnt/wsl/docker-desktop-data/.
How do I change the directory of docker images inside WSL2? May I change docker configuration to select a path inside /mnt/d, or mount a path from /mnt/d over docker data dirs?
The WSL 2 docker-desktop-data vm disk image would normally reside in:
%USERPROFILE%\AppData\Local\Docker\wsl\data\ext4.vhdx
Follow the following to relocate it to other drive/directory, with all existing docker data preserved (tested against Docker Desktop 2.3.0.4 (46911), and continued to work after updating the 3.1.0 (51484)):
First, shut down your docker desktop by right click on the Docker Desktop icon and select Quit Docker Desktop
Then, open your command prompt:
wsl --list -v
You should be able to see, make sure the STATE for both is Stopped.(wsl --shutdown)
NAME STATE VERSION
* docker-desktop Stopped 2
docker-desktop-data Stopped 2
Export docker-desktop-data into a file
wsl --export docker-desktop-data "D:\Docker\wsl\data\docker-desktop-data.tar"
Unregister docker-desktop-data from wsl, note that after this, your ext4.vhdx file would automatically be removed (so back it up first if you have important existing image/container):
wsl --unregister docker-desktop-data
Import the docker-desktop-data back to wsl, but now the ext4.vhdx would reside in different drive/directory:
wsl --import docker-desktop-data "D:\Docker\wsl\data" "D:\Docker\wsl\data\docker-desktop-data.tar" --version 2
Start the Docker Desktop again and it should work
You may delete the D:\Docker\wsl\data\docker-desktop-data.tar file (NOT the ext4.vhdx file) if everything looks good for you after verifying
Stop Docker Desktop
Relocate Docker folder from C:\Users\xxx\AppData\Local\Docker to new path
Make sure C:\Users\xxx\AppData\Local\Docker is no longer there
Open a cmd in administrator mode
Run the following command that will create a symbolic link in the cmd window with the appropriate from and to path
mklink /j "C:\Users\xxx\AppData\Local\Docker" "path to where you relocated your docker folder"
Restart Docker Desktop
Edit: re-register docker-desktop would set the default docker-data to C drive now, so we should only unregister docker-data as the accepted answer.
You can do
wsl --unregister docker-desktop-data
wsl --import docker-desktop-data D:\wsl\docker-desktop-data "C:\Program Files\Docker\Docker\resources\wsl\wsl-data.tar" --version=2
The tar file is the file used to install, and before it is your new destination.
This always work while the move-wsl or lxrunoffline didn't work for me on fast rings. And sometimes you have to unistall/install docker first
Extending #Attila Badi 's answer would be to also give the same treatment to the C:\ProgramData\Docker folder, which seems to be used for WSL / Windows Containers. Even moving the Docker data folders, would still leave you with a boot drive ProgramData\Docker folder of massive proportions - especially if you are unable or unwilling to clean the images. You cannot migrate it, or move it once installed. Using the Docker engine advanced settings works in Linux container mode, but not in windows and vice versa and has trouble starting.
Steps I followed:
Uninstall Docker. I know... Make sure you have saved what you need.
Create the primary space-eating docker folders, in a location you have a lot of space, e.g. :
D:\Data\Docker\ProgramData_Docker &
D:\Data\Docker\AppData_Local_Docker
Create linked folders, by running the below in a command window in administrator mode:
mklink /j "C:\Users\xxx\AppData\Local\Docker" "D:\Data\Docker\ProgramData_Docker"
mklink /j "C:\ProgramData\Docker" "D:\Data\Docker\AppData_Local_Docker"
Install Docker.
You should be able to merrily pull windows server images, but not clog up your boot drive.
UPDATE:
Trying to symlink the C:\ProgramData\Docker folder, may result in a security error, depending on the version running depending on the originally installed version.
Release notes for 4.13.0 refers to this feature, which my be a possible work-around (Thanks to #bhagerty and #Oly for the trail):
start /w “” “Docker Desktop Installer.exe” install --installation-dir=G:\Docker
(Source: ungureanuovidiu # https://forums.docker.com/t/docker-installation-directory/32773/17 )
For me docker won't start with junction.
Then I've used just directory symbolic link:
Docker stopped
Folder "wsl" moved to other location on disk "B"
RUben#AD-RUBEN C:\Users\RUben\AppData\Local\Docker
$ mklink /D wsl "B:\dev\wsl"
**symbolic link** created for wsl <<===>> B:\dev\wsl
Containers and Images are ready to use:
A nice tool:
DDoSolitary/LxRunOffline: A full-featured utility for managing Windows Subsystem for Linux (WSL)
https://github.com/DDoSolitary/LxRunOffline
LxRunOffline.exe move Move a distribution to a new directory.
Options:
-n arg Name of the distribution
-d arg The directory to move the distribution to.
for example:
quit docker desktop, then:
wsl --shutdown
LxRunOffline.exe move -n docker-desktop-data -d D:\vm\dockerdesktop\wsl\data
I found this tool from pxlrbt on github. It's using standard wsl import/export and pretty safe. Just moved both my docker-desktop-data distro to a different drive and it works well.
The best option is to update the registry. Follow the below steps
Shutdown the wsl. Use the command wsl --shutdown.
Move the entire C:\Users\%USERPROFILE%\AppData\Local\Docker directory to different drive for example D:\Docker.
Goto Registry editor location Computer\HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Lxss.
Find the registry entry with the BasePath set to C:\Users\%USERPROFILE%\AppData\Local\Docker\wsl\data. Update this D:\Docker\wsl\data.
Find another registry entry with the BasePath set to C:\Users\%USERPROFILE%\AppData\Local\Docker\wsl\distro. Update this D:\Docker\wsl\distro.
Restart wsl using: wsl -d Ubuntu.
In Windows 10 home, docker desktop creates the VM under ""C:\Users\xxx\AppData\Local\Docker" directory and it is this VM that contains the downloaded docker images. If you want to change the VM location from C: to a different directory you can do this by creating a junction on windows (prior to docker desktop installation) using a command like below:
mklink /j "C:\Users\xxx\AppData\Local\Docker" "D:\Users\xxx\AppData\Local\Docker"
Note that prior to executing the command the target directory structure should exist while you should delete the C:\Users\xxx\AppData\Local\Docker directory if it exists already else the command could fail. Now install docker desktop on windows 10 home and voila you can see stuff inside "D:\Users\xxx\AppData\Local\Docker" directory namely the docker VM hard disk image file that is going to contain all the downloaded docker images.
Using small SSD also you may want to relocate WSL swap file location.
https://learn.microsoft.com/en-us/windows/wsl/wsl-config

Write access in mounted volume in docker

I mounted a folder in docker ubuntu container
docker run -it --rm -v ${pwd}:/files ubuntu /bin/bash
but when i try to write to mounted folder get error
root#0974e91f50ec:/files# cat >a.txt
bash: a.txt: Permission denied
Is it possible to write to mounted folder?
I just tried your example. Here's what I ran on my windows machine.
And the file appears fine.
Also they appear fine in the host system (I created b.txt on the host to check if it appears fine in the container).
PS: Something you have to beware of is that on Windows you can only mount very specific directories (e.g. ~/Documents directory) at least when using Docker toolbox (Not sure about Docker Desktop CE). I think to mount folders from other directories there's bit of work to be done. So it might be a good idea to make sure that whatever you are trying to mount is in fact something allowed by your Docker on Windows.

How can i see path in windows for docker container? Mine is giving root#default:/home/some file name,where i can see this path in windows

Hey i'm trying to find out the path where i can see the file in windows. I install docker toolbox in D drive, But when i give command "docker exec -it container-name /bin/bash then it redirect me to root#default:/home/application-name.
Now my Question is where can i see home/application-name file path in windows?
That path is not visible to host system as its on a virtual disk mounted on docker. The Virtual Disk itself is a file

Docker: Mount volume from Windows host

I use 1.12 version of Docker on Windows, since I can't use the Hyper-V feature with the newer "native" version - so I have my quickstart terminal and communicate to docker host via the invisible underlying virtual box.
Now I have the problem, that I need to mount a local folder to a container, which worked successfully from within the docker-machine by adding
--volume="`pwd`:/root/data"
to the docker run command, but it does not when I launch the same command from my Windows quickstart terminal (even though pwd command works correctly in the terminal).
I tried to find the Windows specific settings for the directory and tested several combinations of format, but no luck. Can anyone help me out on how to correctly specify a Windows folder (e.g. C:\Users\alexander.ruehl) for the volume parameter?
You can use relative path for your volume : --volume="./mydata:/root/data"
Also make sure that you have given the permission for read/write to Docker.

Running docker shell on Windows

I've just installed Docker toolbox 1.9.1 (only docker itself plus Kitematic as I already have both VirtualBox and Git for Windows installed). Clicking on Docker Quickstart Terminal icon wasn't working so I had to associate it with C:\Program Files (x86)\Git\bin\bash.exe as suggested. Now it starts fine but I'm unable to run docker command:
Kitematic works just fine but I need the shell. How do I fix it?
Check where docker-machine.exe has been installed (or copy the latest released one in your %PATH%) and use that, instead of the quickstart.
From a regular CMD session:
# find the name of the machine created.
docker-machine ls
docker-machine env --shell cmd <nameOfTheMachine>
docker-machine ssh <nameOfTheMachine>
The shell you are after is the one provided by the Linux TinyCore boot2docker.iso image managed by VirtualBox, not the git-bash installed on your Windows host.
Here is an alternative option. Create the following docker.cmd batch:
cd "%ProgramFiles%\Docker Toolbox"
"%ProgramFiles(x86)%\Git\bin\bash.exe" --login -i "%ProgramFiles%\Docker Toolbox\start.sh"
Now you need to start the default VM in your Oracle VirtualBox Manager (created automatically when you installed Docker for Windows):
and then the batch:

Resources