I have installed the cloudera quickstart using docker based on the instructions given here.
https://blog.cloudera.com/blog/2015/12/docker-is-the-new-quickstart-option-for-apache-hadoop-and-cloudera/
docker run --privileged=true --hostname=quickstart.cloudera -p 7180 -p 8888 -t -i 9f3ab06c7554 /usr/bin/docker-quickstart
You can see that I am doing -p 7180 and -p 8888 for port mapping.
when the container booted successfully. I saw that the hue service startup failed. but i ran it manually using sudo service hue restart and it showed OK.
Now I ran
/home/cloudera/cloudera-manager --express --force
this command was successful I got a message to connect to the CM using http://cloudera.quickstart:7180
Now on my host machine I did docker-machine env default and I could see the output
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/Users/abhishek.srivastava/.docker/machine/machines/default"
export DOCKER_MACHINE_NAME="default"
Now in my browser on host machine I did
http://192.168.99.100:7180
http://192.168.99.100:8888
http://quickstart.cloudera:7180
http://quickstart.cloudera:8888
but everything fails to connect to any page. So even after doing port forwarding... I am not able to access either cloudera manager or HUE UI from the host machine.
I am using OSX.
I also went into virtualbox manager UI and selected the default VM. I went into settings -> network -> port forwarding. and made the following entries
but still I cannot access the cloudera manager and HUE....
When you run docker using -p 7180 and -p 8888, it will allocate a random port on your windows host. However, if you use -p 7180:7180 and -p 8888:8888, assuming those ports are free on the host, it will map them directly.
Otherwise you can execute docker ps and it will show you which ports it mapped the 7180 and 8888 to. Then in your host browser you can enter
http://192.168.99.100:<docker-allocated-port>
instead of
http://192.168.99.100:7180
If its all on your local machine, you shouldn't need the port forwarding.
Since you're running the docker machine inside a VM, you need to open the port on VirtualBox.
You can do this from the Port Forwarding button in the network adapter panel in VirtualBox.
Settings > Network > Advanced > Port Forwarding
You should see an SSH port already being forwarded for docker. Just add any additional ports like that one.
And here are lists of all the ports used by CDH. Of course you don't need all of them. I would suggest at least Cloudera Manager (7180), namenode and datanode UI (50070 & 50075), and the job servers like mapreduce (8088,8042 & 10020) or spark (18080 & 18081). And I personally don't use it, but Hue is 8888.
The same issue happened to me. I was able start hue successfully after increasing the number of CPUs in VirtualBox.
I also increased the amount of RAM earlier. The original CPU I had was 1, changed to 3
I have encountered the same issue here, and resolved now based on the comments and posts above. There are two issues mentioned above:
Failed to start Hue.
In my case, this is caused by limited resources allocated with default docker VM settings. According to #Ronald Teo's answer, going to
VirtualBox -> 'default'[your docker-machine name] -> Settings ->
System
, increase base memory to 8192MB, and processors to at least 3, have fixed my problem.
Can not access Hue from my host machine. Based on the original post, Try docker run --privileged=true --hostname=quickstart.cloudera -p 7180:7180 -p 8888:8888 -t -i 9f3ab06c7554 /usr/bin/docker-quickstart should solve this problem.
Restart Hue after container is up
Increase the memory of docker to 8GB if you can. Otherwise, set it at least 4GB.
Let hue fail while starting the container.
After that, attach to the docker container and access its shell to run the following command,
To stop the Hue Server:
$ sudo service hue stop
To start the Hue Server:
$ sudo service hue start
I was just trying to spin up the Cloudera quickstart docker myself, and it turns out this seems to do the trick:
http://127.0.0.1:8888
Note the http, not https, and that I use 127.0.0.1 (or localhost)
Note that this assumes that the internal 8888 port is mapped to your 8888 port.
Suppose docker inspect yields something like
"8888/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "32769"
}
Then you would want
http://127.0.0.1:32769
Related
Hi what I am actually trying is to connect remotly from a MySQL Client in Windows Subsystem for Linux mysql -h 172.18.0.2 -P 3306 -u root -p and before that I started the Docker Container as follows: docker container run --name testdb --network testnetwork -p 3306:3306 -e MYSQL_ROOT_PASSWORD=mysqlRootPassword -e MYSQL_DATABASE=localtestdb -d mariadb/server.
The purpose why I put the container in a own network, is because I also have a dockerized Spring Boot Application (GraphQL-Server) which shall communicated with this db. But always when I try to connect from my built-in mysql client, in my Windows Subsystem for Linux, with the above shown command. I got the error message: ERROR 2002 (HY000): Can't connect to MySQL server on '172.18.0.2' (115).
What I already tried, to solve the problem on my own is, look up whether the configuration file line (bind-address) is commented out. But it wont work. Interestingly it already worked to set up a docker container with MariaDB and connect from the outside, but now when I try exactly the same, only with the difference that I now put the container in a own existing network, it wont work.
Hopefully there some one out there which is able to help me with this annonying problem.
Thanks!
So far,
Daniel
//edit:
Now I tried the solution advice from a guy from this topic: How to configure containers in one network to connect to each other (server -> mysql)?. Futhermore I linked my Spring Boot (server) application with the "--link databaseContainerName" parameter to the MariaDB container.
Now I am able to start both containers without any error, but I am still not able to connect remotly to the MariaDB container. Which is now running in a virtual docker network with his own subnet.
I explored this recently - this is by design - container isolation. Usually only main (service httpd) host is accessible externally, hiding internal connections (hosts it communicates to deliver response).
Container created in own network is not accessible from external adresses, even from containers in the same bridge but other network (172.19.0.0/16).
Your container should be accessible on docker host address (127.0.0.1 if run locally) and mapped ("-p 3306:3306") port - 3306. But of course it won't work if many running db containers have the same mapping to the same host port.
Isolation is done using firewall - iptables. You can list rules (iptables -L) to see that - from docker host level.
You can modify firewall to allow external access to internal networks. I used this rule:
iptables -A DOCKER -d 172.16.0.0/12 -j ACCEPT
After that your MySQL containerized engine should be accessible using internal address 172.18.0.2 and source (not mapped) port 3306.
Warnings
it disables all isolation, dont't use it on production;
you have to run this after every docker start - rules created/modified by docker on the fly
not every docker container will respond on ping, check it from docker host (linux subsystem in this case) first, from windows cmd later
I used this option (in docker.service) to make rule permanent:
ExecStartPost=/bin/sh -c '/etc/iptables/accept172_16.sh'
For docker on external(shared in lan) host you should use route add (or hosts file on your machine or router) to forward 172.x.x.x addresses into lan docker host.
Hint: use portainer project (with restart policy - always) to manage docker containers. It's easier to see config errors, too.
So I'm using Docker Toolbox because I don't have Hyper-V on my machine since it's not Windows 10 pro. Everything seems to work fine, but when I try to go on my browser 0.0.0.0:80 it always returns me: This site can’t be reached
But when I run the command: docker container ps I get the following: 0.0.0.0:80->80/tcp meaning that this address should work. I searched across stackoverflow and github issues. Now I'm stuck.
Am I missing something?
Thanks,
Mark
EDIT:
Using docker-machine ip default returns me 192.168.99.100. I run that on port 80. I still get the same result except that the address becomes the container id: https://fd677edg12
I run that command on cmd to find my ipv4: cmd /k ipconfig /all. Put the result with the port and it returns the same thing: https://fd677edg12
Docker Toolbox doesn't get as many conveniences as Docker for Windows, but you're correct in using it since you're on Home edition.
In Toolbox, nothing will be localhost, and will be 192.168.99.100 by default, since it's running a Linux VM in VirtualBox.
So if you run docker run -p 80:80 nginx
(notice I had to publish a port for 192.168.99.100 to listen on that port)
Then going to http://192.168.99.100 should work.
I initially had a few issues with accessing my Applications at localhost:8080 while using DockerToolBox and OracleVM VirtualBox.
In VirtualBox:
Click the appropriate machine (probably the one labeled "default")
Settings
Network > Adapter 1 > Advanced > Port Forwarding
Click "+" to add a new Rule
Set Host Port 8080 & Guest Port 8080; be sure to leave Host IP and Guest IP empty
Run the command:
docker run -p 8080:8080 ${image_id}
I was following docker for windows tutorial in https://docs.docker.com/docker-for-windows/#set-up-tab-completion-in-powershell and got stuck in step #6 when test nginx in the web browser. Seems I faced a similar problem since I also use Windows Home and don't have Hyper-V. My workaround is quite simple:
check your docker IP default
$ docker-machine ip default
192.168.99.100
Go to Oracle Virtual Machine to set for port forwarding. Make sure the network setting is NAT, and add port forwarding.
Host IP: 127.0.0.1, Guest IP: 192.168.99.100, port all set to 80
like this
Try again to your browser and run http://localhost or http://127.0.0.1 (can add the port 80 also). It should run.
The thing is that the nginx IP is meant to be accessible within the docker Virtual Machine, so that we need that port forwading setting in order to access it directly in the host machine's browser
You can use localhost instead of '192.168.99.100' by following the instructions:
Step #01:
docker-machine ip default
You will see the default IP
Step #02:
docker-machine stop default
Step #03:
Open VirtualBox Manager (from the start programs in windows search for VirtualBox Manager)
Select your Docker Machine VirtualBox image (e.g.: default)
Open Settings -> Network -> Advanced -> Port Forwarding
Add your app name, the desired host port and your guest port
i.e, app name : nginx, host: 127.0.0.1, host port: 80, guest port: 80
Step #04:
Now you’re ready to start your Docker Machine by executing the following:
docker-machine start default
Then just start your Docker container and you will be able to access it via localhost.
Have a look here for details.
To map the ports expected to localhost instead of hitting the docker-machine IP directly, you can use the VirtualBox CLI.
If the docker-machine VM (here called default) is running, add and delete rules like this:
> VBoxManage.exe controlvm "default" natpf1 "nginx,tcp,,8888,,8888"
> VBoxManage.exe controlvm "default" natpf1 delete nginx
If the VM is not running, or you want to stop before altering it:
> docker-machine stop
> VBoxManage.exe modifyvm "default" --natpf1 "nginx,tcp,,8888,,8888"
> VBoxManage.exe modifyvm "default" --natpf1 delete "nginx"
> docker-machine start
Where the format of the port forwarding rule is [<name>],tcp|udp,[<hostip>],<hostport>,[<guestip>], <guestport>.
Note that in VirtualBox, you want to map to the host port of Docker map, not the internal container port. You're mapping host -> VM, then Docker maps VM -> container.
See the VirtualBox docs.
This is another easy way to avoid typing the ip 192.168.99.100.
Go to C:\Windows\System32\drivers\etc\hosts and add at the end of the file:
192.168.99.100 docker.awesome or any name of your liking.
Save the file (You need to have admin rights so make sure you right click on the file and run as administrator to be able to save it when you edit it).
Go to your chosen domain name, docker.awesome:8080 in this case and there you have it.
After lot of trials, I was able to get this bulletin board.
The docker run command I used - docker run -p 4680:8080 --name bb bulletinboard:1.0
Here, 4680 is localhost port number.
8080 is container port number, the port at which the container will be listening. This port number is mentioned in the EXPOSE command in the Dockerfile.
Then, go to web-browser and type 192.168.99.100:4680
Here, 192.168.99.100 is the docker machine IP address (use command -> docker-machine ip)
After this, your browser page should open to -
Hope this helps you all!!
I am using Docker Quickstart Terminal to run a docker container. The container should work on port 8088 of localhost:
docker run -it --name myContainer -p 8088:8088
However, when I go to localhost:8088 or 127.0.0.1:8088 I can't find any process running.
This works on OSX.
Why is this not working on Windows?
I'm assuming you're using VirtualBox, since that's what is integrated with the Quickstart terminal.
The reason it doesn't work is that Windows isn't running your (Linux) containers natively, it's running them in a separate Linux-based VM. This VM is available under a different ip address than your "physical" machine, usually printed when you start the quickstart terminal:
This is the ip address you need to use in order to connect to published container ports.
One possibility is the kind of VM you are using : HyperV (Docker For Windows) or VirtualBox (Docker Toolbox).
If it is the later (which seems probable since you are using the Docker Quickstart Terminal), you need to port forward 8088 in order for your PC (localhost) to see it.
See "How do I configure docker compose to expose ports correctly?" as an example when using VirtualBox.
If localhost does not work, a docker-machine ip will show you the ip of the VM being executed.
In my job I working with docker and the option --net=host working like a charm forwarding the docker container ports to the machine. This allows me to adding grunt tasks that use certain ports by example:
A taks for serving my coverage report in a port 9001
A local deployed version of my app served in the port 9000
A watch live reload the port 35729
For Unit testing runner use the 9876 port
When I begin to use Docker in Mac, the first problem that i had was: The option --net=host don't work anymore.
I researched and I understand why this is not possible (Docker in Mac runs in a own virtual machine) and my momentary solution it's use the -p option for expose the ports, but this limit to me to add more and more task that use ports because i need run the explicit -p command for each port that i need expose.
Anyone with this same problem? How to dealing with this ?
Your issue is most probably that you are using dockertoolbox or dhingy/dlite or anything else providing a full-fledged linux VM, which then hosts docker to run your container inside this VM. This VM has, of course, its own network stack and own IP on the host, and thats were your tools will have issues with. The exposed ports of the container are not exposed to OSX host localhost, but rather OSX Docker-VM-ip.
To solve those issues elegantly
Expose ports to OSX localhost from the container
First, use/install docker-for-mac https://docs.docker.com/engine/installation/mac/ instead of dockertoolbox or others. Its based on a special xhyve stack which reuses your hosts network stack
when you now do docker run -p 3306:3306 percona it will bind 3306 on the osx-host-localhost, thus every other osx-tool trying to attach to localhost:3306 will work ( very useful ) just as you have been used to it when you installed mysql using brew install mysql or likewise
If you experience performance issues with code shares on OSX with docker containers, check http://docker-sync.io - it is compatible with docker-for-mac ( hint: i am biased on this one )
Export ports from the OSX-host to a containter
You do not really export anything in particular, you rather make them accessable as a whole from all containers ( all ports of the OSX-host-localhost)
If you want to attach to a port you offered on the OSX host, from within a container, e.g. during a xdebug session were your IDE listens on port 9000 on the OSX-host-localhost and the container running FPM/PHP should attach to this osx-localhost:9000 on the mac, you need to do this: https://gist.github.com/EugenMayer/3019516e5a3b3a01b6eac88190327e7c
So you create a dummy loopback ip, so you can access your OSX-host ports from without containers using 10.254.254.254:9000 - this is portable and basically gives you all you need to develop like you have used to
So one gives you the connectivity to container-exposed ports to apps running on the mac and trying to connect to localhost:port
And the second the inverse, if something in the container wants to attach to a port on the host.
One workaround, mentioned in "Bind container ports to the host" would be to use -P:
(or --publish-all=true|false) to docker run which is a blanket operation that identifies every port with an EXPOSE line in the image’s Dockerfile or --expose <port> commandline flag and maps it to a host port somewhere within an ephemeral port range.
The docker port command then needs to be used to inspect created mapping.
So if your app can use docker port <CONTAINER> to retrieve the mapped port, you can add as many containers as you want and get the mapped ports that way (without needed an "explicit -p command for each port").
Not sure if docker for mac can support bi-directional connection later https://forums.docker.com/t/will-docker-for-mac-support-bi-directional-connection-between-host-and-container-in-the-future/19871
I have two solution:
you can write a simple wrapper script and pass the port you want to expose to the script
use vagrant to start a virtual machine with network under control.
I'm new to Docker, and I can't seem to connect to any containers.
I installed Docker Toolbox. Now I'm trying to get Shipyard to work. I followed the steps inside of a Docker Quickstart Terminal. The instructions say:
Once deployed, the script will output the URL to connect along with credential information.
The Shipyard installer ended with:
Shipyard available at http://10.0.2.15:8080
Username: [elided] Password: [elided]
However, I went to http://10.0.2.15:8080 on my browser and it didn't connect.
In another Docker Quickstart Terminal, I did a docker ps to see what the container was and to get its IP Address and I got:
$ docker inspect a4755 | grep IPAddress
"SecondaryIPAddresses": null,
"IPAddress": "172.17.0.8",
"IPAddress": "172.17.0.8",
I'm not sure why the IP was different, but I tried to connect to http://172.17.0.8:8080 and this didn't work either. http://localhost:8080 also failed.
This also happened when I tried to run docker-gunicorn-nginx - everything started, but I couldn't connect to the machine.
What gives?
If you read through Docker's Installation on Mac OS X you'll see that on OSX, Docker containers don't run on the host machine itself:
In a Docker installation on Linux, your physical machine is both the localhost and the Docker host. In networking, localhost means your computer. The Docker host is the computer on which the containers run.
On a typical Linux installation, the Docker client, the Docker daemon, and any containers run directly on your localhost. This means you can address ports on a Docker container using standard localhost addressing such as localhost:8000 or 0.0.0.0:8376.
[...]
In an OS X installation, the docker daemon is running inside a Linux VM called default. The default is a lightweight Linux VM made specifically to run the Docker daemon on Mac OS X. The VM runs completely from RAM, is a small ~24MB download, and boots in approximately 5s.
In OS X, the Docker host address is the address of the Linux VM. When you start the VM with docker-machine it is assigned an IP address. When you start a container, the ports on a container map to ports on the VM. To see this in practice, work through the exercises on this page.
Indeed, opening a new Docker Quickstart Terminal, I see:
docker is configured to use the default machine with IP 192.168.99.100
And, opening http://192.168.99.100:8080 takes me to Shipyard. Success!
You can try and execute this command:
docker-machine ip default
it will return some thing like:
192.168.99.100
To get port number:
docker ps
Example output (scroll right to see port mapping):
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
113346425f20 springio/spring1 "sh -c 'java $JAVA_OP" 34 minutes ago Up 34 minutes 0.0.0.0:8080->8080/tcp pensive_kirch
To verify if it is working do:
curl 192.168.99.100:8080