I have an ICp installation on some bare metal to educate myself with. So I don't need to keep it running all the time. What is the proper way to shut it down while I am not using it? I have two physical nodes; master and worker. Currently I just ssh into each and issue a sudo shutdown now command.
When I bring the cluster back on line later, the I can't get to the admin UI. It responds with a 502 bad gateway error. When I load https://master:9443 I get the Welcome to Liberty page (indicating that at least the web server is running).
If you stop docker containers or the docker runtime, then the kubelet will attempt to restart them.
If you want to shutdown the system, you must stop the kubelet on each node. On Ubuntu, you would use systemctl:
sudo systemctl stop kubelet
sudo systemctl stop docker
Confirm that all processes are shutdown:
top
And that all related network ports are no longer in use:
netstat -antp
(Note that netstat's "-p" option requires root privileges to inspect the pid holding onto the port).
To restart the cluster, start docker and then the kubelet. Again for Ubuntu:
sudo start docker
sudo start kubelet
And of course you can follow the logs for the kubelet:
sudo journalctl -e -u kubelet
Stop Docker to shut it down, I hope this helped.
systemctl stop docker
I'm trying to mount a local hadoop cluster using docker and ambari, the problem im having is that ambari install check shows NTP is not running, and it is needed to know if the services installed with ambari are working. I checked ntpd in the containers and tried to launch them but it failed
[root#97ea7075ca78 ~]# service ntpd start
Starting ntpd: [ OK ]
[root#97ea7075ca78 ~]# service ntpd status
ntpd dead but pid file exists
Is there a way to start ntp daemon in those containers?
In docker you don't use the service command as there is no init system. Just run the ntpd command and it should work
ntpd by default goes to background. If that was not the case you would need to use ntpd &
iam using hadoop apache 2.7.1 on centos 7
and my cluster is ha cluster and iam using zookeeper quorum for automatic failover
but i want to automate zookeeper start process and ofcourse in the shell script we have to stop firewall first in order to let other quorum elements able to contact current zookeeper element
iam writing the following script in /etc/rc.d/rc.local
hostname jn1
systemctl stop firewalld
ZOOKEEPER='/usr/local/zookeeper-3.4.9/'
source /etc/rc.d/init.d/functions
source $ZOOKEEPER/bin/zkEnv.sh
daemon --user root $ZOOKEEPER/bin/zkServer.sh start
but iam facing the problem that when iam issuing the command
systemctl stop firewalld
in rc.local
and issuing zkServer status after host boots iam getting the error
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.9/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
but if i execute the same commands with out a script i mean after my host boots as normal process
systemctl status firewalld
zkServer start
there is no problem and zkstatus shows its mode
i have noticed the difference in zookeeper.out log between executing rc.local script and normal commands after the host boots
and the difference is reading server environments in normal commands execute
what could be the effect of stopping firewall at rc.local script to server environment and how to handle it
?
i have abig headache about stopping and restarting firewall scenarios
and i discovered that stopping firewall at rc.local is a fake stopping
so because idon't want fire wall to work at all i ended up with the following solution
systemctl disable firewalld
https://www.rootusers.com/how-to-disable-the-firewall-in-centos-7-linux/
so firewall is not going to work again at any boot
I have installed the cloudera quickstart using docker based on the instructions given here.
https://blog.cloudera.com/blog/2015/12/docker-is-the-new-quickstart-option-for-apache-hadoop-and-cloudera/
docker run --privileged=true --hostname=quickstart.cloudera -p 7180 -p 8888 -t -i 9f3ab06c7554 /usr/bin/docker-quickstart
You can see that I am doing -p 7180 and -p 8888 for port mapping.
when the container booted successfully. I saw that the hue service startup failed. but i ran it manually using sudo service hue restart and it showed OK.
Now I ran
/home/cloudera/cloudera-manager --express --force
this command was successful I got a message to connect to the CM using http://cloudera.quickstart:7180
Now on my host machine I did docker-machine env default and I could see the output
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/Users/abhishek.srivastava/.docker/machine/machines/default"
export DOCKER_MACHINE_NAME="default"
Now in my browser on host machine I did
http://192.168.99.100:7180
http://192.168.99.100:8888
http://quickstart.cloudera:7180
http://quickstart.cloudera:8888
but everything fails to connect to any page. So even after doing port forwarding... I am not able to access either cloudera manager or HUE UI from the host machine.
I am using OSX.
I also went into virtualbox manager UI and selected the default VM. I went into settings -> network -> port forwarding. and made the following entries
but still I cannot access the cloudera manager and HUE....
When you run docker using -p 7180 and -p 8888, it will allocate a random port on your windows host. However, if you use -p 7180:7180 and -p 8888:8888, assuming those ports are free on the host, it will map them directly.
Otherwise you can execute docker ps and it will show you which ports it mapped the 7180 and 8888 to. Then in your host browser you can enter
http://192.168.99.100:<docker-allocated-port>
instead of
http://192.168.99.100:7180
If its all on your local machine, you shouldn't need the port forwarding.
Since you're running the docker machine inside a VM, you need to open the port on VirtualBox.
You can do this from the Port Forwarding button in the network adapter panel in VirtualBox.
Settings > Network > Advanced > Port Forwarding
You should see an SSH port already being forwarded for docker. Just add any additional ports like that one.
And here are lists of all the ports used by CDH. Of course you don't need all of them. I would suggest at least Cloudera Manager (7180), namenode and datanode UI (50070 & 50075), and the job servers like mapreduce (8088,8042 & 10020) or spark (18080 & 18081). And I personally don't use it, but Hue is 8888.
The same issue happened to me. I was able start hue successfully after increasing the number of CPUs in VirtualBox.
I also increased the amount of RAM earlier. The original CPU I had was 1, changed to 3
I have encountered the same issue here, and resolved now based on the comments and posts above. There are two issues mentioned above:
Failed to start Hue.
In my case, this is caused by limited resources allocated with default docker VM settings. According to #Ronald Teo's answer, going to
VirtualBox -> 'default'[your docker-machine name] -> Settings ->
System
, increase base memory to 8192MB, and processors to at least 3, have fixed my problem.
Can not access Hue from my host machine. Based on the original post, Try docker run --privileged=true --hostname=quickstart.cloudera -p 7180:7180 -p 8888:8888 -t -i 9f3ab06c7554 /usr/bin/docker-quickstart should solve this problem.
Restart Hue after container is up
Increase the memory of docker to 8GB if you can. Otherwise, set it at least 4GB.
Let hue fail while starting the container.
After that, attach to the docker container and access its shell to run the following command,
To stop the Hue Server:
$ sudo service hue stop
To start the Hue Server:
$ sudo service hue start
I was just trying to spin up the Cloudera quickstart docker myself, and it turns out this seems to do the trick:
http://127.0.0.1:8888
Note the http, not https, and that I use 127.0.0.1 (or localhost)
Note that this assumes that the internal 8888 port is mapped to your 8888 port.
Suppose docker inspect yields something like
"8888/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "32769"
}
Then you would want
http://127.0.0.1:32769
I have stopped the ntpd and restarted it again. Have done a ntpdate pool.ntp.org. the error went once and the hosts were healthy but after sometime again got a clock offset error.
Also I observed that after doing a ntpdate the web interface of cloudera stopped working. It says potential mismatch configuration fix and restart hue.
I have the cloudera quick start vm with centos setup on VMware.
Check if /etc/ntp.conf file is the same across all nodes/masters
restart ntp
add deamon with chkconfig and set it to on
You can fix it by restarting the NTP service which syncronizes the time with a central source.
You can do this by logging in as root from the commandline and running service ntpd restart.
After about a minute the error in CM shoud go away.
Host Terminal
sudo su
service ntpd restart
Clock offset Error occur on Cloudera Manager if host\node's NTP service could not located or did not respond to a request for the clock offset.
Solution:
1)Identify NTP Server IP or Get details of NTP Server IP for your hadoop Cluster
2)On your Hadoop Cluster Nodes Edit-> /etc/ntp.conf
3)Add entries in ntp.conf
server [NTP Server IP]
server xxx.xx.xx.x
4)Restart Services.Execute
Service ntpd restart
5) Restart Cluster From Cloudera Manager
Note: If Problem Still Persist .Reboot you Hadoop Nodes & Check Process.
Check $ cat /etc/ntp.conf make sure configuration file is same as others (nodes)
$ systemctl restart ntpd
$ ntpdc -np
$ ntpdate -u 0.centos.pool.ntp.org
$ hwclock --systohc
$ systemctl restart cloudera-scm-agent
After that wait a few seconds to let it auto configure.