I have a system that has no connection to the internet but does have local mirrors of the OS package manager repos, PyPI, Conda, etc. I'd like to build Singularity images on this system. It looks like the localimage bootstrap agent was designed specifically for this use case, but how would I download the base image from which to build my images? From an internet-connected machine, can I download an "official" base image from SyLabs, Singularity Hub, or somewhere else in my browser without installing Singularity?
The easiest would probably be to use singularity to build just the base image on the internet connected computer. For example, if you would normally bootstrap off ubuntu:latest from Docker hub:
sudo singularity build ubuntu_latest.sif docker://ubuntu:latest
Then copy it to the offline server and use that with Bootstrap: localimage.
Related
I have installed Docker Desktop 3.0 on my mac - on an external drive.
I'm trying to uprade it to latest. I don't get prompted for them and I can't see any option to check for updates in the GUI.
I'd rather not uninstall incase it removes the containers and the data I have set up. But maybe I won't lose them?
can someone point me in the right direction?
thanks.
The location of the files dosn't play role in the upgrade. What plays the major role role is the source version and target version. There are two ways to upgrade the Docker for Mac, and there will be described in detail below:
Incremental upgrade
Upgrade with rebuild (Export / Import)
In the specific case of the upgrade from 3.03 to 3.3, the upgrade to 3.1 and 3.2 can be done as incremental, without need to export / import data, however the upgrade to 3.2.1 will require rebuild of the VM, so the export / import method will help to move the data to the new VM.
Location of data
The application dosn't contain the data for the containers. Docker for macOS is running on a virtual machine, and the data is stored on the virtual disk of the virtual machine.
The raw data (the virtual disk) of the virtual machine is located in
~/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw
Where the 0 is for the 0th VM. By default it's only one VM, but it's better to check. You can find this information from the GUI from Preferences > Resources > Disk Image Location. Simple copy of the content of the data folder will preserve the virtual machine.
Incremental Upgrade
It's always recommended to do incremental upgrade by following the official release versions between your current version and the target version. The releass can be found in the release notes.
The steps to upgrade are:
Stop the docker application (from the menu, find the docker icon and select Quit Docker Desktop
Make full backup of the ~/Library/Containers/com.docker.docker/Data
Rename the application (you might need it to go back, in case the upgrade is not successful)
Copy the new application in the destination and start it.
This steps will be repeated for each minor version between the source version and target version. Some versions can be skipped, however it's not recommended.
In case of problems you can use the troubleshooting guid.
Upgrades with rebuild of the VM
Be aware that upgrading the docker from certain older version to new version might require a rebuild of the virtual machine.
In addition, sometimes the upgrade is not working properly and you will need to uninstall the application as described in the troubleshooting page.
So before you proceed with the uninstall and rebuild of the VM, if you have data that it's important for you, it is highly recommendable to export the containers, volumes and images before the upgrade.
You will need to make a list of all of the containers, images and volumes you want to move across to the new VM. Following docker commands will help:
~# docker ps -a
~# docker volume ls
~# docker image ls
To export the information, you can use the docker export and docker image save. The following example will save the image or the container/volume to file called file.raw:
~# docker export <<volume name or contianer name>> > file.raw
~# docker image save <<image name>> > file.raw
Once the Docker is installed (the latest version) and the VM is rebuild, to import the data back from the files, you will need to use:
~# docker import file.raw
~# docker image load file.raw
I am looking to play with apache-superset on a cloud-based ide. I have it on my local. I tried unsuccessfully to set it up on gitpod. I wanted suggestions on where can I set it up, opensource preferably not necessarily. I believe cloud9 is 1 such place, but I am looking for other options before I settle. If you've ever set it up on any such platform, even if it is on gitpod and can help me, kindly do so.
[Disclaimer: Gitpod staff]
You can indeed use Gitpod to work on apache-superset, and for that you'll just need a working configuration.
From what I can see in apache-superset's requirements, you'll need to get:
PostgreSQL (e.g. by using Gitpod's official gitpod/workspace-full-postgres Docker base image)
Redis (e.g. by installing it in a Dockerfile via sudo apt-get install)
Various Python dependencies (e.g. by running pip install . after cloning)
Various Node.js dependencies for the front-end (e.g. by running npm install)
Here is a basic configuration I wrote to achieve this:
https://github.com/jankeromnes/incubator-superset/commit/0d345a76ec8126fd1f8b9bc7b6ce4961bf3b593d
What it does is:
Create a Docker image with PostgreSQL and Redis
Once the repository is cloned, open 4 separate Terminals ("tasks"):
Redis server
Superset backend
Superset worker
Superset front-end
All dependencies will be installed automatically, and once the front-end is ready, it will automatically open in a web preview IDE side panel.
You can try it out by opening my personal fork of the apache-superset repository in Gitpod, e.g. by following this link:
https://gitpod.io/#https://github.com/jankeromnes/incubator-superset
Docker can work decently remotely by defining DOCKER_HOST variable but now I do want to avoid installing the fat Docker for MacOS which also installs and starts the docker engine on a VM, one that seems to consume resources.
As docker work remotely it should continue to be able to build images, list images, start and stop them without having a docker servic/vm mac.
How can I do this? (docker cli seems to be come only with the entire cow).
I guess you are looking for install-client-binaries-on-macos.
Docker company afford some prebuilt binary, just download here, unpackage it, then you will find a standalone docker client binary there, copy it to your mac, out of the box for use.
There seems to be no install of the client only; but after installing the fat cow, you can tell it to stay off your grass by unchecking Start Docker Desktop when you log in in the preferences, and then shut down the Docker server.
That's what I do, and when I need docker locally I just start and then after using it, I shut it down again.
I'm using Windows 7 Pro and have existing shared Docker engine running on a Linux. I would like to use my workstation (with development environment) to access shared Docker engine.
Does someone know how to retrieve Docker client only for Windows 7 ? I have no admin privilege, so I can't install Docker Toolbox.
Older Clients can be found here https://download.docker.com/win/static/stable/x86_64/
And here a some newer Builds by Stefan Scherer (Docker employee) https://github.com/StefanScherer/docker-cli-builder.
Just download docker.exe and add it to your windows path variable.
Then set your DOCKER_HOST variable to define against which Docker daemon you want to speak.
The official Downloads have finally been published:
Mac CLI binaries are available at
https://download.docker.com/mac/static/stable/
Windows CLI (and daemon) binaries at https://download.docker.com/win/
Linux CLI packages are available for each distro as docker-cli (deb and rpm) packages: https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/amd64/
From: https://github.com/docker/cli/issues/2281#issuecomment-947699400
Although docker provides a REST-like API, there aren't many clients for it. A quick google turned up one on github, but ymmv. Even if you did find one, you're likely to run into the same problems involved in running docker-ce locally anyway.
There are a handful of gui clients that you could run on that engine and access with a browser, but if you are specifically after a cli you're SOL with this.
If you have an ssh client (git bash, or putty, or something), and you can arrange to run a bastion container on the engine, then you could run a container to ssh into and use that as if it's your local machine. You'd still have to scp resources onto it, but you'll eventually have to solve that problem anyway. Something like:
sudo docker container run --interactive --tty -v /var/run/docker.sock:/var/run/docker.sock image
Where 'image' is a linux image with an ssh server and appropriate keys, git, and docker installed. You could mount a local volume for persistence, or you could just keep everything in git.
How can I "mirror" my whole system to a Docker image on an automated way? I wouldn't like to create only a base-image, but all my installed applications to image.
You can find some example in "Create a base image".
You have two options:
use a distro command like debootstrap, which can install a distro base system into a subdirectory of another, already installed system.
But that is not exactly what you want since it wouldn't copy your own host distro content.
use the SCRATCH image, and copy the files you want in it.