How to enable Docker API access from Windows running Docker Toolbox (docker machine) - windows

I am running the latest Docker Toolbox, using latest Oracle VirtualBox, with Windows 7 as a host OS.
I am trying to enable non-TLS access to Docker remote API, so I could use Postman REST client running on Windows and hit docker API running on docker-machine in the VirtualBox. I found that if Docker configuration included -H tcp://0.0.0.0:2375, that would do the trick exposing the API on port 2375 of the docker machine, but for the life of me I can't find where this configuration is stored and can be changed.
I did docker-machine ssh from the Toolbox CLI, and then went and pocked around the /etc/init.d/docker file, but no changes to the file survive docker-machine restart.
I was able to find answer to this question for Ubuntu and OSX, but not for Windows.

#CarlosRafaelRamirez mentioned the right place, but I will add a few details and provide more detailed, step-by-step instructions, because Windows devs are often not fluent in Linux ecosystem.
Disclaimer: following steps make it possible to hit Docker Remote API from Windows host, but please keep in mind two things:
This should not be done in production as it makes Docker machine very not secure.
Current solution disables most of the docker-machine and all docker CLI functionality. docker-machine ssh remains operational, forcing one to SSH into docker machine to access docker commands.
Solution
Now, here are the steps necessary to switch Docker API to non-TLS port. (Docker machine name is assumed to be "default". If your machine name has a different name, you will need to specify it in the commands below.)
Start "Docker Quickstart Terminal". It starts Bash shell and is the place where all following commands will be run. Run docker-machine ip command and note the IP address of the docker host machine. Then do
docker-machine ssh
cd /var/lib/boot2docker
sudo vi profile This starts "vi" editor in elevated privileges mode required for editing "profile" file, where Docker host settings are. (If as a Windows user you are not familiar with vi, here's is super-basic crash course on it. When file is open in the vi, vi is not in editing mode. Press "i" to start edit mode. Now you can make changes. After you made all the changes, hit Esc and then ZZ to save changes and exit vi. If you need to exit vi without saving changes, after Esc please type :q! and hit Enter. ":" turns on vi's command mode, and "q!" command means exit without saving. Detailed vi command info is here.)
Using vi, change DOCKER_HOST to be DOCKER_HOST='-H tcp://0.0.0.0:2375', and set DOCKER_TLS=no. Save changes as described above.
exit to leave SSH session.
docker-machine restart
After doocker machine has restarted, your sould be able to hit docker API URL, like http://dokerMachineIp:2375/containers/json?all=1, and get valid JSON back.
This is the end of steps required to achieve the main goal.
However, if at this point you try to run docker-machine config or docker images, you will see an error message indicating that docker CLI client is trying to get to the Docker through the old port/TLS settings, which is understandable. What was not expected to me though, is that even after I followed all the Getting Started directions, and ran export DOCKER_HOST=tcp://192.168.99.101:2375 and export DOCKER_TLS_VERIFY=0, resulting in
$ env | grep DOCKER
DOCKER_HOST=tcp://192.168.99.101:2375
DOCKER_MACHINE_NAME=default
DOCKER_TLS_VERIFY=0
DOCKER_TOOLBOX_INSTALL_PATH=C:\Program Files\Docker Toolbox
DOCKER_CERT_PATH=C:\Users\USERNAME\.docker\machine\machines\default
the result was the same:
$ docker-machine env
Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates for host
"192.168.99.101:2376"
If you see a problem with how I changed environment variables to point Docker CLI to the new Docker host address, please comment.
To work around this problem, use docker-machine ssh command and run your docker commands after that.

I encountered the same problem and thanks to #VladH made it working not changing any internal Docker profile properties. All you have to do is correctly define Windows local env variables (or configure maven plugin properties, if you use io.fabric8 docker-maven-plugin).
Note that 2375 port is used for non-TLS connections, and 2376 only for TLS connections.
DOCKER_HOST=tcp://192.168.99.100:2376
DOCKER_TLS_VERIFY=0
DOCKER_TOOLBOX_INSTALL_PATH=C:\Program Files\Docker Toolbox
DOCKER_CERT_PATH=C:\Users\USERNAME\.docker\machine\machines\default

Related

How are Docker Desktop proxy settings on Windows propagated to Docker?

I am on a corporate Windows laptop and I want to start experimenting with Docker. Being a corporate machine, everything needs to go through the corporate proxy.
I installed Debian on WSL and then the Docker Desktop, which installed its components on the Debian WSL VM. My first priority however was to test docker on WSL directly and not through Docker Desktop. So I set to read the Docker docs and download the docker/getting-started image through the Debian terminal. That, however, failed due to not using the network proxy.
Desktop Docker docs state that setting the proxy settings on Docker Desktop will propagate the proxy settings to Docker itself. Indeed, I set the proxy settings on Docker Desktop, and I was now able to properly download my image from inside Debian.
Since I want to have full control of Docker through the Debian terminal and not Docker Desktop, I want to understand in which way the proxy settings propagate to Docker inside WSL. I imagined that Docker Desktop altered some configuration file inside Debian, but a grep on the whole system of the proxy ip got me nothing. So my question is, in what way does the Docker Desktop let Docker know which proxy to use?
As much as I know, And am not 100% sure as I have not worked with docker in a while.
When you start docker service in WSL, this will trigger the init.d/docker script, And when you set the Company proxy manually in docker desktop, The loading time is :
Stopping Docker service
Updating configuration Script at /etc/init.d/docker
Starting the service again, and with it the new script
And to make sure that this is valid, You can try to check the /etc/init.d/docker script contents.
and as an alternative way of not adding the scripts manually. you can export the proxy configuration in WSL, and check if it will work without adding the proxy configuration to Docker Desktop.

Can I develop with VS Code in containers on a remote host running Windows/WSL2?

Original Post
I have a Windows workstation with WSL2 and Docker installed that I am able to use for container based development in VS Code. I would like to be able to develop inside the containers on this system remotely. I am able to SSH directly into the WSL2 environment on the workstation and am able to start the docker daemon without logging directly into Windows by creating a Task to start the daemon automatically as described here: https://stackoverflow.com/a/59467740/10692741
However when I try to access Docker on the remote machine by following this guide: https://code.visualstudio.com/docs/remote/containers-advanced#_developing-inside-a-container-on-a-remote-docker-host, I get the following error:
error during connect: Get http://docker/v1.24/version: net/http: HTTP/1.x transport connection broken: malformed HTTP status code "\x00c\x00o\x00m\x00m\x00a\x00n\x00d\x00"
I have also tried connecting via a SSH tunnel as outlined here: https://code.visualstudio.com/docs/remote/troubleshooting#_using-an-ssh-tunnel-to-connect-to-a-remote-docker-host and am unable to connect to Docker as well.
Has anyone had success with a setup like this? Or is this not supported due to limitations with Docker on Windows, WSL2, and/or Windows OpenSSH implementation?
Update: 2021-01-21
When I SSH into the Windows machine remotely, I am able to see the docker containers in the VS Code extension. I am able to start them, stop them, and enter into them with the shell. However, when I try to attach VS Code I get same error shown above.
Things that may have possibly affected this over the past couple days:
Adding SSH keys on my local machine to the ssh-agent via ssh-add /my/key
Exposing Docker daemon on tcp://localhost:2375 without TLS on the remote Windows machine
Also I want to note that the I've tried using Windows, Mac, and Linux as the local machine. With Mac and Linux I am able to open a remote session into the Windows machine, but from the Windows local machine I am able to SSH into the remote Windows machine but cannot open a remote connection in VS Code for some reason.
Ok, I was able to get this working using the port/socket forwarding technique. For sake of clarity, I'll use:
local development workstation, local workstation, or just workstation to indicate the computer from which we wish to use VSCode to access Docker containers on ...
the remote Docker host, remote, or just Docker host
Sanity check -- Do you have Docker Desktop installed on both systems? On the local development workstation, you can skip the WSL2 integration, but you'll at least need the client tools, since the VSCode extension uses them.
Steps I took:
I already had Docker with WSL2 integration set up on my main system (which for the purposes of this exercise, became my remote Docker host), along with VSCode, so I knew everything was working there. It sounds like that was your starting point as well.
On another system on the same network (accessed with RDP to make it simple), I already had VSCode installed as well, with the Remote Development Extension Pack. I also have WSL on that system, but only a v1 instance there. Not that WSL on the workstation should be a factor at all for the purposes of this exercise.
I installed Docker Desktop for Windows on that local development workstation.
I also installed the Docker extension for VSCode, since I didn't yet have it on the local development workstation.
On the workstation, I was not yet set up to SSH from PowerShell into my WSL Ubuntu distro on the remote. From PowerShell on the workstation, I generated an ECDSA key (per this and other documents) and added the public key to my authorized_keys on the the remote.
On the workstation, I started the OpenSSH Authentication Service and added the newly created key to the agent (in PowerShell) with ssh-agent add ~\.ssh\id_ecdsa.
I logged out of the workstation and back in so that the path changes were picked up for the Docker desktop install.
I was then able to ssh from Powershell on the local to Ubuntu/WSL on the remote with the port forwarding. Since I'm using the Windows 10 OpenSSH server as a jumphost to my WSL SSH servers, my command looked slightly different (with a -o "ProxyCommand ... mainly), but overall the structure is the same as the one listed in the "SSH Tunnel" doc you linked in your question.
On the remote (manually, not through any integration from the local), I did a basic docker run -it --rm Ubuntu and left it open.
On the local, from PowerShell, I set the DOCKER_HOST environment variable via [System.Environment]::SetEnvironmentVariable("DOCKER_HOST","tcp://localhost:23750").
I was then able to see the remote container using docker ps on the local. I could also docker exec -it containername bash into it remotely.
Of course, the above two steps aren't needed in the long term for VSCode, they were just part of my process to make sure everything was up and running (since, as you might expect, I did have several points at which I failed during this process).
So with that working, it was a simple matter in VSCode to change the Docker extension's DOCKER_HOST setting to tcp://localhost:23750. And voila, I could see all images on the remote as well as attach to them from VSCode.
Other thing(s) to check
I'll add to this list if we find additional reasons why it might not be working, but for now:
You mention that you are starting the Docker Desktop daemon automatically at startup via Task Manager, but you don't mention anything about the WSL2 instance. However, since you are able to ssh into it, I assume you have a way to bring it up as well? My experience has been that, unless the owning user is logged in, WSL terminates any instances after a few seconds, even if a service is running. There's a workaround, I believe, that I can dust off if this is a problem.

docker deamon is not work in windows

I try to run docker in bash ubuntu on windows. But every time I get this message
"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?". If i run it in powershell - it work. Can somebody help?
Connecting to the docker deamon requires some privilidges that you don't have when starting the bash terminal.
You can however use the docker command terminal which will allow you to interact with the docker deamon.
Found the solution on this post: https://blog.jayway.com/2017/04/19/running-docker-on-bash-on-windows/
Connect Docker on WSL to Docker on Windows
Running docker against an engine on a different machine is actually quite easy, as Docker can expose a TCP endpoint which the CLI can attach to.
This TCP endpoint is turned off by default; to activate it, right-click the Docker icon in your taskbar and choose Settings, and tick the box next to “Expose daemon on tcp://localhost:2375 without TLS”.
With that done, all we need to do is instruct the CLI under Bash to connect to the engine running under Windows instead of to the non-existing engine running under Bash, like this:
$ docker -H tcp://0.0.0.0:2375 images
REPOSITORY TAG IMAGE ID CREATED SIZE
There are two ways to make this permanent – either add an alias for the above command, or better yet, export an environment variable which instructs Docker where to find the host engine:
$ echo "export DOCKER_HOST='tcp://0.0.0.0:2375'" >> ~/.bashrc
$ source ~/.bashrc
Now, running docker commands from Bash works just like they’re supposed to.
$ docker run hello-world
Hello from Docker!This message shows that your installation appears to be working correctly.

Docker stuck on "Waiting for SSH to be available..."

I'm using a docker with Windows and Hyper-v to create containers. I've added a docker machine vmachine to my docker configuration. First time the machine is created, it gets an IP (although I cannot manage nginx to access it - ERR_CONNECTION_REFUSED) and finishes the bootup.
When I turn off the machine and then try to boot it, i get stuck in this message
Waiting for SSH to be available...
And it doesn't evolve from there. The machine is booted, however, I get an IPv6 when I input the command docker-machine ip vmachine like - fe80::215:5dff:fe21:10b insted of a IPv4
What am I doing wrong?
Problem here is by default docker uses DockerNAT network switch. You should create a new external network switch instead. This issue is covered here and here. You can create an external network switch using the below command
docker-machine create -d hyperv --hyperv-virtual-switch external-switch tempbox1
or you can create one through the UI
Be sure to reboot the device after creating the external switch.
I had a similar issue and non of the solutions worked. Turns out that according to this answer, docker launches SSH with Unix specific elements. This is said to have been fixed in the releases that followed, but I still encountered the 'Waiting for SSH' issue. I resolved this by simply using GIT bash to run all docker related SSH commands.
Use the switch --native-ssh
for example docker-machine --native-ssh .... Get more details from here
docker-machine.exe -debug create --driver hyperv --hyperv-virtual-switch "External Virtual Switch" --hyperv-cpu-count "1" --hyperv-memory "1024" --hyperv-disk-size "20000" mydockervm
make sure to have additional VirtualSwitch configure , with external network driver selected , Uninstall virtualbox
Use the debug switch to see the exact error , for me it was not able to allocate memory.
Here's what's solved it for me.
Turns out Windows 10 starting version 1709 has a built in SSH client at C:\Windows\System32\OpenSSH. Here's an article discussing it.
Looks like docker is using that SSH implementation and it's not compatible. I didn't look for a proper way to remove the built-in SSH implementatino in Windows 10, and simply renamed the folder. That was enough to fix it for me.
After doing what is mentioned in the above suggestions and if you are running docker on a windows machine try to login using cli. This has worked for me.
If you are using Command Promt Docker will stuck at Waiting for SSH to be available..., So change to use GIT BASH as #Dave Howson said it will work.
If you're using oracle VM you must ensure first that your new cloud vm is running.
Before:
After:

How do I get Docker to run on a Windows system behind a corporate firewall?

I'm trying to get a working Docker installation following this tutorial:
http://docs.docker.io/en/latest/installation/windows/
So far, I got the VM running with a manually downloaded repository (followed the GitHub link and downloaded as a ZIP file, because "git clone" didn't work behind my corporate proxy, even after setting up the proxy with "git conf --global http.proxy ..." - it kept asking me for authentification 407, although I entered my user name and password).
Now I am in the state in which I should use "docker run busybox echo hello world" (Section "Running Docker").
When I do this, I first get told that Docker is not installed (as shown at the bottom of the tutorial), and then, after I got it with apt-get install docker, I get "Segmentation Fault or critical error encountered. Dumping core and aborting."
What can I do now? Is this because I didn't use git clone or is something wrong with the Docker installation? I read somewhere, that apt-get install docker doesn't install the Docker I want, but some GNOME tool. Can I maybe specify my apt-request to get the right tool?
Windows Boot2Docker behind corporate proxy
(Context: March 2015, Windows 7, behind corporate proxy)
TLDR; see GitHub project VonC/b2d:
Clone it and:
configure ..\env.bat following the env.bat.template,
add the alias you want in the 'profile' file,
execute senv.bat then b2d.bat.
You then are in a properly customized boot2docker environment with:
an ssh session able to access internet behind corporate proxy when you type docker search/pull.
Dockerfiles able to access internet behind corporate proxy when they do an apt-get update/install and you type a docker build.
Installation and first steps
If you are admin of your workstation, you can run boot2docker install on your Windows.
It currently comes with:
Boot2Docker 1.5.0 (Docker v1.5.0, Linux v3.18.5)
Boot2Docker Management Tool v1.5.0
VirtualBox v4.3.20-r96997
msysGit v1.9.5-preview20141217
Then, once installed:
add c:\path\to\Boot2Docker For Windows\ in your %PATH%
(one time): boot2docker init
boot2docker start
boot2docker ssh
type exit to exit the ssh session, and boot2docker ssh to go back in: the history of commands you just typed is preserved.
if you want to close the VM, boot2docker stop
You actually can see the VM start or stop if you open the Virtual Box GUI, and type in a DOS cmd session boot2docker start or stop.
Hosts & Proxy: Windows => Boot2Docker => Docker Containers
The main point to understand is that you will need to manage 2 HOSTS:
your Windows workstation is the host to the Linux Tiny Core run by VirtualBox in order for you to define and run containers
(%HOME%\.boot2docker\boot2docker.iso =>
.%USERPROFILE%\VirtualBox VMs\boot2docker-vm\boot2docker-vm.vmdk),
Your boot2docker Linux Tiny Core is host to your containers that you will run.
In term of proxy, that means:
Your Windows Host must have set its HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variable (you probably have them already, and they can be used for instance by the Virtual Box to detect new versions of Virtual Box)
Your Tiny Core Host must have set http_proxy, https_proxy and no_proxy (note the case, lowercase in the Linux environment) for:
the docker service to be able to query/load images (for example: docker search nginx).
If not set, the next docker pull will get you a dial tcp: lookup index.docker.io: no such host.
This is set in a new file /var/lib/boot2docker/profile: it is profile, not .profile.
the docker account (to be set in /home/docker/.ashrc), if you need to execute any other command (other than docker) which would require internet access)
any Dockerfile that you would create (or the next RUN apt-get update will get you a, for example, Could not resolve 'http.debian.net').
That means you must add the lines ENV http_proxy http://... first, before any RUN command requiring internet access.
A good no_proxy to set is:
.company,.sock,localhost,127.0.0.1,::1,192.168.59.103
(with '.company' the domain name of your company, for the internal sites)
Data persistence? Use folder sharing
The other point to understand is that boot2docker uses Tiny Core, a... tiny Linux distribution (the .iso file is only 26 MB).
And Tiny Core offers no persistence (except for a few technical folders): if you modify your ~/.ashrc with all your preferred settings and alias... the next boot2docker stop / boot2docker start will restore a pristine Linux environment, with your modification gone.
You need to make sure the VirtualBox has the Oracle_VM_VirtualBox_Extension_Pack downloaded and added in the Virtual Box / File / Settings / Extension / add the Oracle_VM_VirtualBox_Extension_Pack-4.x.yy-zzzzz.vbox-extpack file).
As documented in boot2docker, you will have access (from your Tiny Core ssh session) to /c/Users/<yourLogin> (ie the %USERPROFILE% is shared by Virtual Box)
Port redirection? For container and for VirtualBox VM
The final point to understand is that no port is exported by default:
your container ports are not visible from your Tiny Core host (you must use -p 80:80 for example in order to expose the 80 port of the container to the 80 port of the Linux session)
your Tiny Cort ports are not exported from your Virtual Box VM by default: even if your container is visible from within Tiny Core, your Windows browser won't see it: http://127.0.0.1 won't work "The connection was reset".
For the first point, docker run -it --rm --name my-apache-app -v "$PWD":/usr/local/apache2/htdocs/ httpd:2.4 won't work without a -p 80:80 in it.
For the second point, define an alias doskey vbm="c:\Program Files\Oracle\VirtualBox\VBoxManage.exe" $*, and then:
- if the Virtual Box 'boot2docker-vm' is not yet started, uses vbm modifyvm
- if the Virtual Box 'boot2docker-vm' is already started, uses vbm controlvm
Typically, if I realize, during a boot2docker session, that the port 80 is not accessible from Windows:
vbm controlvm "boot2docker-vm" natpf1 "tcp-port80,tcp,,80,,80";
vbm controlvm "boot2docker-vm" natpf1 "udp-port80,udp,,80,,80";
Then, and only then, I can access http://127.0.0.1
Persistent settings: copied to docker service and docker account
In order to use boot2docker easily:
create on Windows a folder %USERPROFILE%\prog\b2d
add a .profile in it (directly in Windows, in%USERPROFILE%\prog\b2d), with your settings and alias.
For example (I modified the original /home/docker/.ashrc):
# ~/.ashrc: Executed by SHells.
#
. /etc/init.d/tc-functions
if [ -n "$DISPLAY" ]
then
`which editor >/dev/null` && EDITOR=editor || EDITOR=vi
else
EDITOR=vi
fi
export EDITOR
# Alias definitions.
#
alias df='df -h'
alias du='du -h'
alias ls='ls -p'
alias ll='ls -l'
alias la='ls -la'
alias d='dmenu_run &'
alias ce='cd /etc/sysconfig/tcedir'
export HTTP_PROXY=http://<user>:<pwd>#proxy.company:80
export HTTPS_PROXY=http://<user>:<pwd>#proxy.company:80
export NO_PROXY=.company,.sock,localhost,127.0.0.1,::1,192.168.59.103
export http_proxy=http://<user>:<password>#proxy.company:80
export https_proxy=http://<user>:<password>#proxy.company:80
export no_proxy=.company,.sock,localhost,127.0.0.1,::1,192.168.59.103
alias l='ls -alrt'
alias h=history
alias cdd='cd /c/Users/<user>/prog/b2d'
ln -fs /c/Users/<user>/prog/b2d /home/docker
(192.168.59.103 is usually the ip returned by boot2docker ip)
Putting everything together to start a boot2docker session: b2d.bat
create and add a b2d.bat script in your %PATH% which will:
start boot2docker
copy the right profile, both for the docker service (which is restarted) and for the /home/docker user account.
initiate an interactive ssh session
That is:
doskey vbm="c:\Program Files\Oracle\VirtualBox\VBoxManage.exe" $*
boot2docker start
boot2docker ssh sudo cp -f /c/Users/<user>/prog/b2d/.profile /var/lib/boot2docker/profile
boot2docker ssh sudo /etc/init.d/docker restart
boot2docker ssh cp -f /c/Users/<user>/prog/b2d/.profile .ashrc
boot2docker ssh
In order to enter a new boot2docker session, with your settings defined exactly as you want, simply type:
b2d
And you are good to go:
End result:
a docker search xxx will work (it will access internet)
any docker build will work (it will access internet if the ENV http_proxy directives are there)
any Windows file from %USERPROFILE%\prog\b2d can be modified right from ~/b2d.
Or you actually can write and modify those same files (like some Dockerfile) right from your Windows session, using your favorite editor (instead of vi)
And all this, behind a corporate firewall.
Bonus: http only
Tuan adds in the comments:
Maybe my company's proxy doesn't allow https. Here's my workaround:
boot2docker ssh,
kill the docker process and
set the proxy export http_proxy=http://proxy.com, then
start docker with docker -d --insercure-registry docker.io

Resources