I have a Cloudera (the version is cdh6.2.0) cluster and every components(HDFS, HIVE etc.) worked well. However, recently I want to connect to WebHDFS, I found the port(14000) was not running at all, by executing command netstat -antpl|grep 14000 on the NameNode.
I have confirmed the WebHDFS was enabled in Cloudera Manager and it was used 14000 port by default.
Besides, I also tried 50070 port, it didn't listen either. I also tried curl:
curl -i "http://localhost:14000/webhdfs/v1/user/user.name=cloudera&op=GETFILESTATUS"
curl: (7) Failed to connect to localhost port 14000: Connection refused
I appreciate for any help. Thanks.
I solved it by using 9870 port instead.
I found the version of my Hadoop is 3.0, and it is listening with 9870 but not the 50070 for dfs.namenode.http-address.
As for 14000, it is may used for HTTPS REST.
Reference:
https://community.cloudera.com/t5/Support-Questions/Cannot-connect-to-webhdfs/td-p/34830
I started following an online tutorial to configure multi ndoes on my single local VM. here is the hosts on master node:
127.0.0.1 localhost
192.168.96.132 hadoop
192.168.96.135 hadoop1
192.168.96.136 hadoop2
ssh:ALL:allow
sshd:ALL:allow
Here is the command that used to work:hdfs dfs -ls
Now I am seeing error message below:
ls: Call From hadoop/192.168.96.132 to hadoop:9000 failed on connection exception:
java.net.ConnectException: Connection refused;
For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
What is wrong with my configuration? where should I check and correct it?
Thank you very much.
First try to
ping hadoop,
ping hadoop1 and
ping hadoop2.
Ex: ping hadoop
Then just try to connect via ssh
The syntax is
ssh username#hadoop
ssh username#hadoop1
ssh username#hadoop2
Then see the results to find out whether the systems are connecting or not.
I am using VM with Ambari 2.2 and HDP 2.3 and installing services using Ambari user interface. Issue is NameNode not starting and log indicates error saying port is in use 50070. I tried netstat and other tools to find out if anything is running on port 50070, it is not. I also tried changing 50070 to 50071 but error remains the same except it now says port is in use 50071.Below is the error I get in ambari error file:
resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode'' returned 1. starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-hdp-m.out
2016-02-07 11:52:47,058 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode.
java.net.BindException: Port in use: hdp-m.samitsolutions.com:50070
When using Ambari, I came across the port is in use 50070 problem. I found it's actually caused by the mismatch of NameNode's host, not port. In sometimes, Ambari will start namenode on HostB and HostC, while your configure are HostA and HostC.
Such a situation could by caused by: Update wrong namenode config when moving namenode
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this question
When I setup the hadoop cluster, I read the namenode runs on 50070 and I set up accordingly and it's running fine.
But in some books I have come across name node address :
hdfs://localhost:9000/
or
hdfs://localhost:8020
What exactly is the proper number to set the port of namenode?
The default Hadoop ports are as follows: (HTTP ports, they have WEB UI):
Daemon Default Port Configuration Parameter
----------------------- ------------ ----------------------------------
Namenode 50070 dfs.http.address
Datanodes 50075 dfs.datanode.http.address
Secondarynamenode 50090 dfs.secondary.http.address
Backup/Checkpoint node? 50105 dfs.backup.http.address
Jobracker 50030 mapred.job.tracker.http.address
Tasktrackers 50060 mapred.task.tracker.http.address
Internally, Hadoop mostly uses Hadoop IPC, which stands for Inter Process Communicator, to communicate amongst servers. The following table presents the ports and protocols that Hadoop uses. This table does not include the HTTP ports mentioned above.
Daemon Default Port Configuration Parameter
------------------------------------------------------------
Namenode 8020 fs.default.name
Datanode 50010 dfs.datanode.address
Datanode 50020 dfs.datanode.ipc.address
Backupnode 50100 dfs.backup.address
check out this link For more info: http://blog.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/
The default address of namenode web UI is http://localhost:50070/. You can open this address in your browser and check the namenode information.
The default address of namenode server is hdfs://localhost:8020/. You can connect to it to access HDFS by HDFS api. The is the real service address.
Default port for namenode is 9870 on hadoop 3.x. Please refer to https://hadoop.apache.org/docs/r3.0.0/ for details.
9000 is the default HDFS service port.This does not have a web UI.50070 is the default NameNode web UI port (Although, in hadoop 3.0 onwards 50070 is updated to 9870)
That is because default is different for different hadoop configurations and distributions.
We can always configure port by changing fs.default.name or fs.defaultFS properties as below in core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
For Hadoop 1.0.4 if I dont mention port number like below
<value>hdfs://localhost</value>
then default port taken is 8020. But for some of the version like .20 i read it is 9000. So it is dependent on the version of hadoop you are using.
But all the configuration and distributation are using 50070 as standard port number for HDFS ui.
To access Hadoop WEB UI , you need to type http://localhost:50075/
though your core-site.xml is having http://localhost:9000 because it is for hdfs requests and 50075 is the default port for WEB UI.
There are other HTTP ports that would run in server for monitoring. Example: 50070, 8088, 9870, 9864, 9868, 16010, 16030
Hadoop IPC(Inter Process Communicator) ports (Eg. 9000) cannot be accessed through your web browser.
You can find the ports that can be accessed in browser by the following command:
lsof -i -P -n | grep LISTEN
For example, the ports in my server were:
Hadoop Cluster - http://server-name:8088/cluster
Hadoop NameNode/DFS Health - http://server-name:9870/dfshealth.html#tab-overview
Hadoop DataNode - http://server-name:9864/datanode.html
Hadoop Status - http://server-name:9868/status.html
HBase Master Status - http://server-name:16010/master-status
HBase Region server - http://server-name:16030/rs-status
50070 is the default UI port for namenode . while 8020/9000 is the Inter Process Communicator port (IPC) for namenode.
Reference to IPC port : https://en.wikipedia.org/wiki/Inter-process_communication
50070 is default UI port of Namenode for http. for https its 50470.
9000 is the IPC port(Inter Process Communicator). If you click on localhost:50070, you can see namenode configurations with an overview 9000 (active) and on localhost:9000 you will get message:
"It looks like you are making an HTTP request to a Hadoop IPC port. This is not the correct port for the web interface on this daemon." required for file system metadata operations.
You can check what ports each daemon is listening on if you’re having trouble finding the web interface. For example, to check ports the NameNode is listening on:
lsof -Pan -iTCP -sTCP:LISTEN -p `jps | grep "\sNameNode" | cut -d " " -f1`
This will give you output similar to
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 4053 pi 275u IPv6 45474 0t0 TCP *:9870 (LISTEN)
java 4053 pi 288u IPv6 43476 0t0 TCP 127.0.0.1:9000 (LISTEN)
As you can see *:9870 is listed, which is the web interface.
Explanation:
lsof -Pan -iTCP -sTCP:LISTEN -p <pid> lists all network files with TCP state LISTEN. -p filters the list by process id. So by plugging in a process id after this command, you can see all the ports a process is listening on.
jps | grep "\sNameNode" | cut -d " " -f1 gets the process id of the NameNode.
I am getting below error when I am trying to configure hadoop plugin in eclipse.
Error:call to localhost:54310 failed on connection exception:java.net.connectException:Connection refused:no further informaion
Hadoop version is 1.0.4
I have installed hadoop in Linux and I am running my Eclipse using Windows.
In the hadoop location window, I have tried with host as localhost and linux server.
MR Master: Host: localhost and port 54311
DFS Master: Host: localhost and port 54310
MR Master: Host: <Linux server name> and port 54311
DFS Master: Host: <Linux server name> and port 54310
In my mapred-site.xml I see this entry entry localhost:54311.
ConnectionRefused error is, you are trying to connect a directory which you dont have permission to read/write.
This may be caused by, a directory created another user(e.g. root) and your master machine is trying to read from/write to that directory.
It is more likely that you are trying to read an input from wrong place. Check your input directory if there is no problem with it, check your output directory