I am trying to understand the various ports at which various daemons / processes listens at in a Hadoop cluster.
core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://master.hadoop.cluster:54310</value>
</property>
yarn-site.xml
<property>
<name>yarn.resourcemanager.address</name>
<value>master.hadoop.cluster:8032</value>
</property>
I see we have three other ports, which are:
1) 50070 --> To see hdfs GUI
2) 8088 --> To see RM GUI
3) 8042 --> Not sure which GUI we can see at this port
As there are so many ports, I am not clear which port is meant for which thing. If I make HTTP request to port, say at 8032, it says this is Hadoop IPC port.
Can anyone help me understand this, what are the main port numbers which we should be aware of, and what processes listens at those ports.
The port defined in fs.defaultFS is for file system metadata operations. You cannot use it for accessing the Web UI.
8042 is for NodeManager Web UI and 8032 is for ResourceManager IPC.
Refer
hdfs-default.xml - for HDFS related ports
yarn-default.xml - for YARN related ports
mapred-default.xml - for JHS related ports.
Related
The Hadoop Version is 2.7.1
Modify the hdfs.xml , add two properties：
<property>
<name>dfs.namenode.http-address</name>
<value>HADOOP-MASTER:50070</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>HADOOP-SLAVE-1:50090</value>
</property>
And restart the hadoop cluster,but I can't access http://hadoop-master:50070.
the namenode process is alive.
and
[hadoop#HADOOP-MASTER ~]$ lsof -i:50070
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 26541 hadoop 184u IPv4 1261606 0t0 TCP HADOOP-MASTER:50070 (LISTEN)
But when I remove the dfs.namenode.http-address property,the 50070 worked on.
So,The Problem is what dfs.namenode.http-address property mean,I guess it define the node who can access?
That value is defined as "The address and the base port where the dfs namenode web ui will listen on" and defaults to 0.0.0.0:50070, which means it is publicly accessible to all machines that can reach it.
Notice that is says address, not hostname. If you need to change this value from the default, use an IP address, not a physical machine name.
Source: https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
I have been trying to start a MapReduce job on my cluster with the following command:
bin/hadoop jar myjar.jar MainClass /user/hduser/input /user/hduser/output
But I get the following error over and over again, until connection is refused:
13/08/08 00:37:16 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
I then checked with netstat to see if the service was listening to the correct port:
~> sudo netstat -plten | grep java
tcp 0 0 10.1.1.4:54310 0.0.0.0:* LISTEN 10022 38365 11366/java
tcp 0 0 10.1.1.4:54311 0.0.0.0:* LISTEN 10022 32164 11829/java
Now I notice that my service is listening to port 10.1.1.4:54310, which is the IP of my master, but it seems that the 'hadoop jar' command is connecting to 127.0.0.1 (the localhost, which is the same machine) but therefore doesn't find the service. Is there anyway to force 'hadoop jar' to look at 10.1.1.4 instead of 127.0.0.1?
My NameNode, DataNode, JobTracker, TaskTracker, ... are all running. I even checked for DataNode and TaskTracker on the slaves and it all seems to be working. I can check the WebUI on the master and it shows my cluster is online.
I expect the problem to be DNS related since it seems that the 'hadoop jar' command finds the correct port, but always uses the 127.0.0.1 address instead of the 10.1.1.4
UPDATE
Configuration in core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://master:54310</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
</configuration>
Configuration in mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>master:54311</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
</configuration>
Configuration in hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>
</configuration>
Although it seemed to be a DNS issue, it was actually Hadoop trying to resolve a reference to localhost in the code. I was deploying the jar of someone else and assumed it was correct. Upon further inspection I found the reference to localhost and changed it to master, solving my issue.
I am trying to start hbase master but getting the below error:
Could not start ZK at requested port of 2181. ZK was started at port: 2182. Aborting as clients (e.g. shell) will not be able to find this ZK quorum.
13/07/14 06:33:23 ERROR master.HMasterCommandLine: Failed to start master
java.io.IOException: Could not start ZK at requested port of 2181. ZK was started at port: 2182. Aborting as clients (e.g. shell) will not be able to find this ZK quorum.
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:134)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1684)
13/07/14 06:33:23 INFO server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:46283 (no session established for client)
hbase-site.xml
<configuration>
<!-- Changing the default port for REST since it conflicts with yarn nodemanager -->
<property>
<name>hbase.rest.port</name>
<value>8070</value>
<description>The port for the HBase REST server.</description>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:8020/hbase</value>
</property>
</configuration>
It seems like something else is already using port 2181, or perhaps you had started another ZK instance earlier on this port. Either stop that processe or change its port. If that's is not possible then set hbase.zookeeper.property.clientPort to 2182 in hbase-site.xml.
Please note that HBase needs ZK's services, even in standalone mode, so you should make sure that it's running OK.
HTH
Surprisingly, it can be an issue with privileges to connect to the port 2181, but not to 2182. Instead of ./start-hbase.sh try:
sudo ./start-hbase.sh
In my case it helped.
When starting Hbase in standalone mode, a single JVM hosts the HBase Master, an HBase RegionServer, and a ZooKeeper quorum peer. So, you don't need to start a ZK instance separately.
In your case, hbase is not able to start the ZK because another instance is probably already running on port 2181. So, just close that ZooKeeper instance and restart hbase. Also, do ensure proper permissions for the hbase rootdir.
I have installed Hadoop in my system, Jobtracker : localhost:50030/jobtracker.jsp is working fine but localhost:50075/ host is not resolved. Can anybody help my what is the problem in my Ubuntu system. Below check my code-site.xml configuration :
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
never seen 50075 before, but 50070 is the local NameNode, I suggest you format the NameNode to have a try:
rm -r /tmp/hadoop-*;
bin/hadoop namenode -format;
./bin/start-all.sh
Check your port configurations. Here is a list of hadoop daemon configurable parameters
dfs.http.address: The address and the base port where the dfs namenode web ui will listen on. If the port is 0 then the server will start on a free port.
The default for the name node is usually set to port 50070, so try localhost:50070.
http://hadoop.apache.org/common/docs/r0.20.2/hdfs-default.html
I think you mean core-site.xml (not code-site.xml). The fs.default.name configuration here does not determine the location of the hadoop web ui/dashboard. This is the port used by data nodes to communicate with the name node.
We have an 8 node cluster using CDH3u2 configured using Cloudera Manager. We have a dedicated master node running our only instance of zookeeper. When I configure hive to run local hadoop, executed from the master node, I have no problem retreiving the data from HBase. When I run distributed map/reduce via hive, I am getting the following error when the slave nodes connect to zookeeper.
HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default).
We have tried setting max connections higher (we even tried removing the limit). This is a development cluster that has very few users, I know that the problem is not that there are too many connections (I am able to connect to zookeeper from the slave nodes using ./zkCli).
Server side logs indicate that the session was terminated by the client.
Client side hadoop log says:
'Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
Any idea why I am unable to maintian a connection to zookeeper via Hive Map/Reduce?
Configs for hbase and zookeeper are:
# Autogenerated by Cloudera SCM on Wed Dec 28 08:42:23 CST 2011
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=1000
minSessionTimeout=4000
maxSessionTimeout=40000
HBase Site-XML is:
<property>
<name>hbase.rootdir</name>
<value>hdfs://alnnimb01:8020/hbase</value>
<description>The directory shared by region servers. Should be fully-qualified to include the filesystem to use. E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR</description>
</property>
<property>
<name>hbase.master.port</name>
<value>60000</value>
<description>The port master should bind to.</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)</description>
</property>
<property>
<name>hbase.master.info.port</name>
<value>60010</value>
<description>The port for the hbase master web UI Set to -1 if you do not want the info server to run.</description>
</property>
<property>
<name>zookeeper.znode.parent</name>
<value>/hbase</value>
<description>Root ZNode for HBase in ZooKeeper. All of HBase's ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase's ZooKeeper file path are configured with a relative path, so they will all go under this directory unless changed.</description>
</property>
<property>
<name>zookeeper.znode.rootserver</name>
<value>root-region-server</value>
<description>Path to ZNode holding root region location. This is written by the master and read by clients and region servers. If a relative path is given, the parent folder will be ${zookeeper.znode.parent}. By default, this means the root location is stored at /hbase/root-region-server.</description>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
<description>The ZooKeeper client port to which HBase clients will connect</description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>alnnimb01.aln.experian.com</value>
<description>Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".</description>
Turns out that the Map/Reduce submitted by Hive is trying to connect to zookeeper at 'localhost', regardless of how the zookeeper.quorom is setup in the config file. I changed /etc/hosts to have to the alias 'localhost' point to the IP of my master node and the connection to zookeeper is maintained. Still looking for a better resolution, but this will work for now.
I figured it out. It was a configuration issue (as I suspected all along). The solution was to:
-set ‘hbase.zookeeper.quorum’ within the ‘hive-site.xml’ and place it in the ‘hadoop-conf’ directory
What threw me off was that there is no 'hbase.zookeeper.quorum' in hive-default.xml. I had been playing with 'hive.zookeeper.quorum' which was not the correct configuration to change.
I'm sorry for posting a new answer. I wanted to comment on the previous answer but the commenting UI seems to have disappeared >.< ...
Anyway, I wanted to say that I am experiencing the same problem, and it is solved by doing the /etc/hosts hack, but that seems like a very dirty solution...
Did anyone figure out a way of fixing this cleanly...??
Thanks :) !
I meet exactly the same problem. What I did is to use the following conf to start hive cli and it works fine.
hive --hiveconf hbase.zookeeper.quorum={zk-host}
You should config HBase to use the external zookeeper and replace {zk-host} with the host of zookeeper.
I'm still looking for how to resolve this when using jdbc to access hive.