How can I "mirror" my whole system to a Docker image on an automated way? I wouldn't like to create only a base-image, but all my installed applications to image.
You can find some example in "Create a base image".
You have two options:
use a distro command like debootstrap, which can install a distro base system into a subdirectory of another, already installed system.
But that is not exactly what you want since it wouldn't copy your own host distro content.
use the SCRATCH image, and copy the files you want in it.
Related
I've just upgraded to Windows 10 Home May 2020, activated WSL2, and installed Docker Desktop.
WSL2 must be installed in my system disk, which is a small SSD. I don't want to fill it with docker images. How do I change the docker images path? I'd like to use a path in my big Windows filesystem.
The image location is somewhat confusing. I believe it is in /mnt/wsl/docker-desktop-data/.
How do I change the directory of docker images inside WSL2? May I change docker configuration to select a path inside /mnt/d, or mount a path from /mnt/d over docker data dirs?
The WSL 2 docker-desktop-data vm disk image would normally reside in:
%USERPROFILE%\AppData\Local\Docker\wsl\data\ext4.vhdx
Follow the following to relocate it to other drive/directory, with all existing docker data preserved (tested against Docker Desktop 2.3.0.4 (46911), and continued to work after updating the 3.1.0 (51484)):
First, shut down your docker desktop by right click on the Docker Desktop icon and select Quit Docker Desktop
Then, open your command prompt:
wsl --list -v
You should be able to see, make sure the STATE for both is Stopped.(wsl --shutdown)
NAME STATE VERSION
* docker-desktop Stopped 2
docker-desktop-data Stopped 2
Export docker-desktop-data into a file
wsl --export docker-desktop-data "D:\Docker\wsl\data\docker-desktop-data.tar"
Unregister docker-desktop-data from wsl, note that after this, your ext4.vhdx file would automatically be removed (so back it up first if you have important existing image/container):
wsl --unregister docker-desktop-data
Import the docker-desktop-data back to wsl, but now the ext4.vhdx would reside in different drive/directory:
wsl --import docker-desktop-data "D:\Docker\wsl\data" "D:\Docker\wsl\data\docker-desktop-data.tar" --version 2
Start the Docker Desktop again and it should work
You may delete the D:\Docker\wsl\data\docker-desktop-data.tar file (NOT the ext4.vhdx file) if everything looks good for you after verifying
Stop Docker Desktop
Relocate Docker folder from C:\Users\xxx\AppData\Local\Docker to new path
Make sure C:\Users\xxx\AppData\Local\Docker is no longer there
Open a cmd in administrator mode
Run the following command that will create a symbolic link in the cmd window with the appropriate from and to path
mklink /j "C:\Users\xxx\AppData\Local\Docker" "path to where you relocated your docker folder"
Restart Docker Desktop
Edit: re-register docker-desktop would set the default docker-data to C drive now, so we should only unregister docker-data as the accepted answer.
You can do
wsl --unregister docker-desktop-data
wsl --import docker-desktop-data D:\wsl\docker-desktop-data "C:\Program Files\Docker\Docker\resources\wsl\wsl-data.tar" --version=2
The tar file is the file used to install, and before it is your new destination.
This always work while the move-wsl or lxrunoffline didn't work for me on fast rings. And sometimes you have to unistall/install docker first
Extending #Attila Badi 's answer would be to also give the same treatment to the C:\ProgramData\Docker folder, which seems to be used for WSL / Windows Containers. Even moving the Docker data folders, would still leave you with a boot drive ProgramData\Docker folder of massive proportions - especially if you are unable or unwilling to clean the images. You cannot migrate it, or move it once installed. Using the Docker engine advanced settings works in Linux container mode, but not in windows and vice versa and has trouble starting.
Steps I followed:
Uninstall Docker. I know... Make sure you have saved what you need.
Create the primary space-eating docker folders, in a location you have a lot of space, e.g. :
D:\Data\Docker\ProgramData_Docker &
D:\Data\Docker\AppData_Local_Docker
Create linked folders, by running the below in a command window in administrator mode:
mklink /j "C:\Users\xxx\AppData\Local\Docker" "D:\Data\Docker\ProgramData_Docker"
mklink /j "C:\ProgramData\Docker" "D:\Data\Docker\AppData_Local_Docker"
Install Docker.
You should be able to merrily pull windows server images, but not clog up your boot drive.
UPDATE:
Trying to symlink the C:\ProgramData\Docker folder, may result in a security error, depending on the version running depending on the originally installed version.
Release notes for 4.13.0 refers to this feature, which my be a possible work-around (Thanks to #bhagerty and #Oly for the trail):
start /w “” “Docker Desktop Installer.exe” install --installation-dir=G:\Docker
(Source: ungureanuovidiu # https://forums.docker.com/t/docker-installation-directory/32773/17 )
For me docker won't start with junction.
Then I've used just directory symbolic link:
Docker stopped
Folder "wsl" moved to other location on disk "B"
RUben#AD-RUBEN C:\Users\RUben\AppData\Local\Docker
$ mklink /D wsl "B:\dev\wsl"
**symbolic link** created for wsl <<===>> B:\dev\wsl
Containers and Images are ready to use:
A nice tool:
DDoSolitary/LxRunOffline: A full-featured utility for managing Windows Subsystem for Linux (WSL)
https://github.com/DDoSolitary/LxRunOffline
LxRunOffline.exe move Move a distribution to a new directory.
Options:
-n arg Name of the distribution
-d arg The directory to move the distribution to.
for example:
quit docker desktop, then:
wsl --shutdown
LxRunOffline.exe move -n docker-desktop-data -d D:\vm\dockerdesktop\wsl\data
I found this tool from pxlrbt on github. It's using standard wsl import/export and pretty safe. Just moved both my docker-desktop-data distro to a different drive and it works well.
The best option is to update the registry. Follow the below steps
Shutdown the wsl. Use the command wsl --shutdown.
Move the entire C:\Users\%USERPROFILE%\AppData\Local\Docker directory to different drive for example D:\Docker.
Goto Registry editor location Computer\HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Lxss.
Find the registry entry with the BasePath set to C:\Users\%USERPROFILE%\AppData\Local\Docker\wsl\data. Update this D:\Docker\wsl\data.
Find another registry entry with the BasePath set to C:\Users\%USERPROFILE%\AppData\Local\Docker\wsl\distro. Update this D:\Docker\wsl\distro.
Restart wsl using: wsl -d Ubuntu.
In Windows 10 home, docker desktop creates the VM under ""C:\Users\xxx\AppData\Local\Docker" directory and it is this VM that contains the downloaded docker images. If you want to change the VM location from C: to a different directory you can do this by creating a junction on windows (prior to docker desktop installation) using a command like below:
mklink /j "C:\Users\xxx\AppData\Local\Docker" "D:\Users\xxx\AppData\Local\Docker"
Note that prior to executing the command the target directory structure should exist while you should delete the C:\Users\xxx\AppData\Local\Docker directory if it exists already else the command could fail. Now install docker desktop on windows 10 home and voila you can see stuff inside "D:\Users\xxx\AppData\Local\Docker" directory namely the docker VM hard disk image file that is going to contain all the downloaded docker images.
Using small SSD also you may want to relocate WSL swap file location.
https://learn.microsoft.com/en-us/windows/wsl/wsl-config
I have installed Docker Desktop 3.0 on my mac - on an external drive.
I'm trying to uprade it to latest. I don't get prompted for them and I can't see any option to check for updates in the GUI.
I'd rather not uninstall incase it removes the containers and the data I have set up. But maybe I won't lose them?
can someone point me in the right direction?
thanks.
The location of the files dosn't play role in the upgrade. What plays the major role role is the source version and target version. There are two ways to upgrade the Docker for Mac, and there will be described in detail below:
Incremental upgrade
Upgrade with rebuild (Export / Import)
In the specific case of the upgrade from 3.03 to 3.3, the upgrade to 3.1 and 3.2 can be done as incremental, without need to export / import data, however the upgrade to 3.2.1 will require rebuild of the VM, so the export / import method will help to move the data to the new VM.
Location of data
The application dosn't contain the data for the containers. Docker for macOS is running on a virtual machine, and the data is stored on the virtual disk of the virtual machine.
The raw data (the virtual disk) of the virtual machine is located in
~/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw
Where the 0 is for the 0th VM. By default it's only one VM, but it's better to check. You can find this information from the GUI from Preferences > Resources > Disk Image Location. Simple copy of the content of the data folder will preserve the virtual machine.
Incremental Upgrade
It's always recommended to do incremental upgrade by following the official release versions between your current version and the target version. The releass can be found in the release notes.
The steps to upgrade are:
Stop the docker application (from the menu, find the docker icon and select Quit Docker Desktop
Make full backup of the ~/Library/Containers/com.docker.docker/Data
Rename the application (you might need it to go back, in case the upgrade is not successful)
Copy the new application in the destination and start it.
This steps will be repeated for each minor version between the source version and target version. Some versions can be skipped, however it's not recommended.
In case of problems you can use the troubleshooting guid.
Upgrades with rebuild of the VM
Be aware that upgrading the docker from certain older version to new version might require a rebuild of the virtual machine.
In addition, sometimes the upgrade is not working properly and you will need to uninstall the application as described in the troubleshooting page.
So before you proceed with the uninstall and rebuild of the VM, if you have data that it's important for you, it is highly recommendable to export the containers, volumes and images before the upgrade.
You will need to make a list of all of the containers, images and volumes you want to move across to the new VM. Following docker commands will help:
~# docker ps -a
~# docker volume ls
~# docker image ls
To export the information, you can use the docker export and docker image save. The following example will save the image or the container/volume to file called file.raw:
~# docker export <<volume name or contianer name>> > file.raw
~# docker image save <<image name>> > file.raw
Once the Docker is installed (the latest version) and the VM is rebuild, to import the data back from the files, you will need to use:
~# docker import file.raw
~# docker image load file.raw
I have a system that has no connection to the internet but does have local mirrors of the OS package manager repos, PyPI, Conda, etc. I'd like to build Singularity images on this system. It looks like the localimage bootstrap agent was designed specifically for this use case, but how would I download the base image from which to build my images? From an internet-connected machine, can I download an "official" base image from SyLabs, Singularity Hub, or somewhere else in my browser without installing Singularity?
The easiest would probably be to use singularity to build just the base image on the internet connected computer. For example, if you would normally bootstrap off ubuntu:latest from Docker hub:
sudo singularity build ubuntu_latest.sif docker://ubuntu:latest
Then copy it to the offline server and use that with Bootstrap: localimage.
I use 1.12 version of Docker on Windows, since I can't use the Hyper-V feature with the newer "native" version - so I have my quickstart terminal and communicate to docker host via the invisible underlying virtual box.
Now I have the problem, that I need to mount a local folder to a container, which worked successfully from within the docker-machine by adding
--volume="`pwd`:/root/data"
to the docker run command, but it does not when I launch the same command from my Windows quickstart terminal (even though pwd command works correctly in the terminal).
I tried to find the Windows specific settings for the directory and tested several combinations of format, but no luck. Can anyone help me out on how to correctly specify a Windows folder (e.g. C:\Users\alexander.ruehl) for the volume parameter?
You can use relative path for your volume : --volume="./mydata:/root/data"
Also make sure that you have given the permission for read/write to Docker.
I have a Linux x86 application inside a docker container and I want to run it under Windows. I don't want to force users to install Virtual Box. Ideally a qemu or similar virtualization tool can be used, since it is very tiny and requires no installation at all.
My approach was to use qemu for Windows and
boot2docker, so I can boot a minimal Linux with docker installed and than run my docker container within it.
This is the command I'm using to run it:
qemu-system-x86_64.exe -m 256 -cdrom boot2docker.iso
The boot goes well, but I have several problems:
at every boot the image goes trough all the configuration steps (generating keys for ssh, setting hostname, etc.) that can be skipped the second time the image runs; seems that the changes to the image are not persisted trough runs. I want to build an image that is already configured and needs only to boot;
to add my application inside the image I have to rebuild the whole boot2docker.iso image by using the steps described in How to build boot2docker.iso locally.
So, the question is: how can I use the base boot2docker.iso image and add some persisting data (such as configurations and my application)? Perhaps a read/write partition mounted from another file?
like the idea.
Maybe you can check MobaliveCD, it has a nice lightweight GUI and it embeds qemu system inside. I tried it for tinycore live cd iso (base of boot2docker), which works quite ok.
While it seems it doesn't support 64bit (which boot2docker needs), but the function fits for you need.
Your command
qemu-system-x86_64.exe -m 256 -cdrom boot2docker.iso
launches an ISO, what you want is
reserve some disk place for this iso in a .img
run this iso and install it in this .img
reboot
In a Linux you would start by doing
qemu-img create -f qcow2 /home/myuser/my_image.img 6G
There is docker-cli for Windows, it seems to be what you look for, see
http://azure.microsoft.com/blog/2014/11/18/docker-cli-for-windows-clients/
You can use boot2docker http://boot2docker.io/
On boot2docker installation, it will install virtualbox behind the scenes.
You only have to start the boot2docker shortcut and the virtual box management and vms are hidden.