browse file system link - hadoop - localhost link - hadoop

I am using Hadoop 2.2 on Ubuntu.
I am able to load this link in my browser.
http://[my_ip]:50070/dfshealth.jsp
From there, when I click the "Browse the filesystem" link, I am sent to
http://localhost:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/&nnaddr=127.0.0.1:9000
while here I think I want my_ip instead of localhost and 127.0.0.1
Also, if I type manually
http://my_ip:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/&nnaddr=my_ip:9000
it still does not work.
The my_ip is an external/global IP throughout my whole question text.
How can I get this working? All I want is to be able to browse my HDFS filesystem from the browser.
core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
<!-- <value>hdfs://my_ip:9000</value> -->
</property>
<!--
fs.default.name
hdfs://localhost:9000
-->
</configuration>
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/var/lib/hadoop/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/var/lib/hadoop/hdfs/datanode</value>
</property>
<!--
dfs.replication
1
dfs.namenode.name.dir
file:/var/lib/hadoop/hdfs/namenode
dfs.datanode.data.dir
file:/var/lib/hadoop/hdfs/datanode
-->
<property>
<name>dfs.http.address</name>
<value>my_ip:50070</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>my_ip:50075</value>
</property>
</configuration>
/etc/hosts
127.0.0.1 localhost test02
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
EDIT ERROR:
HTTP ERROR 500
Problem accessing /nn_browsedfscontent.jsp. Reason:
Cannot issue delegation token. Name node is in safe mode.
The reported blocks 21 has reached the threshold 0.9990 of total blocks 21. The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically in 2 seconds.
Caused by:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot issue delegation token. Name node is in safe mode.
The reported blocks 21 has reached the threshold 0.9990 of total blocks 21. The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically in 2 seconds.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5887)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:447)
at org.apache.hadoop.hdfs.server.namenode.NamenodeJspHelper$1.run(NamenodeJspHelper.java:623)
at org.apache.hadoop.hdfs.server.namenode.NamenodeJspHelper$1.run(NamenodeJspHelper.java:620)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)

In your hdfs-site.xml, replace
<property>
<name>dfs.http.address</name>
<value>my_ip:50070</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>my_ip:50075</value>
</property>
by
<property>
<name>dfs.namenode.http-address</name>
<value>localhost:50070</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>localhost:50075</value>
</property>
But usually, in pseudo-ditributed mode it's not necessary to specify those properties.
Reboot your cluster after changing the properties.

Related

unable to connect to hdfs on localhost

I am unable to connect to hdfs on port 9000, I keep getting this error:
localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused
hdfs-site.xml file is this:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/localhdfs/datanode</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>dfs.namenode.rpc-bind-host</name>
<value>0.0.0.0</value>
</property>
</configuration>
and core-site.xml file is this:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
I have restarted the cluster multiple times, I keep getting connection errors:
this is my /etc/hosts file look like:
127.0.0.1 localhost
what am I missing?
Why are you using port 9000 in your config? fs.defaultFS should contain something like: hdfs://nameofcluster
Is this a single node instance? Sandbox? Are you running the command hdfs dfs -ls /?
I would first check:
Remove port from fs.default.name
iptables or Firewalls
hadoop.proxyuser.hdfs.hosts
hadoop.proxyuser.hdfs.groups
Ranger
Logs are exceeding 80% of disk
/etc/hosts file contains the FQDN and your machine's public IP.
Get the IP with ip a command and set is like: 192.166.6.6 abc.xxx.com
Delete fs.default.name property from hdfs-site.xml
Configure a Single node cluster

successive errors on hadoop :failed on connection exception then com.google.protobuf.InvalidProtocolBufferException

i am trying to execute the command hadoop dfs -ls, and i got this error
Call From localhost/127.0.0.1 to yass-SATELLITE-C855-2CF:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
once i resolved i got another which is
ls: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type.; Host Details : local host is: "localhost/127.0.0.1"; destination host is: "yass-SATELLITE-C855-2CF":9000;
and i keep in loop between this two errors
my core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://yass-SATELLITE-C855-2CF:9000</value>
</property>
</configuration>
hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.safemode.threshold.pct</name>
<value>0</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/hadoop/data/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/hadoop/data/datanode</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/home/yass/Téléchargements/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
etc/hosts
127.0.0.1 localhost
127.0.0.1 yass-SATELLITE-C855-2CF
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
i got the first exception once it dispair i got the second one and i keep always in loop with this two exceptions
Any suggestions please ?
in reality i find somehow a solution in order to not staying with same errors
so i installed Hadoop verion 2.6.2 ,configure the version using the XML then keep working ,is not good solution for everyone but i hope that will show light for others

namenode can not started in Multi-node cluster in ec2

so i follow this guide to setup mu multi-node cluster:http://disi.unitn.it/~lissandrini/notes/installing-hadoop-on-ubuntu-14.html
when i finish all setup and run start-dfs.sh, after that, when i run jps, only have SecondaryNameNode started.
here is my core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://mynode1/</value>
<description>NameNode URI</description>
</property>
</configuration>
and my hdfs.xml
<configuration>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///usr/local/hadoop/data/datanode</value>
<description>DataNode directory</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///usr/local/hadoop/data/namenode</value>
<description>NameNode directory for namespace and transaction logs storage.</description>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
</configuration>
and my /etc/hosts
127.0.0.1 localhost
54.225.196.4 mynode1
54.80.40.198 mynode2
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
This is my namenode log
2014-10-26 01:16:57,756 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ip-10-169-41-62.ec2.internal/10.169.41.62
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.4.1
The thing i notice is if i change everything in the setting from mynode1 to localhost, then seem namenode can get started, but datanode in node2 will not response to master, which i cant upload a file to hdfs
I recommend you to use this tutorial here. I used it and everything was working fine, I had just changing the port number from 8020 t0 9000, and 8021 to 9001.
The core and hdfs files are not correct. I cannot understand how many nodes you are deploying. So, change your tutorial as I refereed you in the above link, and if you have any issue just let me know.

Setting up Hadoop Cluster - NameNode smoke test "Unable to connect"

I am having trouble connection to my NameNode server from another server on the cluster. the namenode starts fine and i can get to the namenode dashboard browsing to http://localhost:50070, but trying to browse to http://hadoop-cluster-1:50070 or even using the IP address doesn't work. I am able to ping hadoop-cluster-1 and the IP address. I am also able to traceroute the port and host all from the server that i am getting an "Unable to connect" in Firefox. See below for values files.
core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop-cluster-1.com:8020</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>8192</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop-cluster-1:8020</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop-cluster-1:8020</value>
<description>The port where the NameNode runs the HDFS protocol.
Combined with the NameNode's hostname to build its address.
</description>
</property>
<property>
<name>dfs.namenode.rpc-address</name>
<value>hadoop-cluster-1:8020</value>
<description>
RPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist, the name ser
vice id is added to the name e.g. dfs.namenode.rpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this propert
y will take the form of nn-host1:rpc-port.
</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data/nn1,file:///data/nn2</value>
</property>
<property>
<name>dfs.blocksize</name>
<value>131072</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data/data1</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

Can't connect to remote HBase Cluster

I have a small HBase/Hadoop cluster. I can login to the master server and run commands against hbase via the hbase shell without a problem. However, when I try to do the same thing from a remote machine I get the below error:
ERROR: 20000 millis timeout while waiting for channel to be ready for connect. h : java.nio.channels.SocketChannel[connection-pending remote=/10.175.224.75:6000]
I ran a netstat on the master server to confirm it was listening on port 60000 and got the following (x.x.x.x is the ip address of the master server).
tcp 0 0 x.x.x.x:60000 : LISTEN
I believe this means that the master server is only listening for connections on port 60000 that originate from itself. Is there a way to configure HBase to accept remote connections? Here is my hbase-site.xml.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.master</name>
<value>master:60000</value>
<description>The host and port that the HBase master runs at.
TODO: Support 'local' (All running in single context).
</description>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:9000/hbase</value>
<description>The directory shared by RegionServers.
</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
</description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>
<property>
<name>zookeeper.znode.parent</name>
<value>/master</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/hduser/hbase/zookeeper</value>
</property>
</configuration>

Resources