so i follow this guide to setup mu multi-node cluster:http://disi.unitn.it/~lissandrini/notes/installing-hadoop-on-ubuntu-14.html
when i finish all setup and run start-dfs.sh, after that, when i run jps, only have SecondaryNameNode started.
here is my core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://mynode1/</value>
<description>NameNode URI</description>
</property>
</configuration>
and my hdfs.xml
<configuration>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///usr/local/hadoop/data/datanode</value>
<description>DataNode directory</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///usr/local/hadoop/data/namenode</value>
<description>NameNode directory for namespace and transaction logs storage.</description>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
</configuration>
and my /etc/hosts
127.0.0.1 localhost
54.225.196.4 mynode1
54.80.40.198 mynode2
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
This is my namenode log
2014-10-26 01:16:57,756 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ip-10-169-41-62.ec2.internal/10.169.41.62
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.4.1
The thing i notice is if i change everything in the setting from mynode1 to localhost, then seem namenode can get started, but datanode in node2 will not response to master, which i cant upload a file to hdfs
I recommend you to use this tutorial here. I used it and everything was working fine, I had just changing the port number from 8020 t0 9000, and 8021 to 9001.
The core and hdfs files are not correct. I cannot understand how many nodes you are deploying. So, change your tutorial as I refereed you in the above link, and if you have any issue just let me know.
Related
I have Hadoop 3.2.1 installed on Ubuntu 16.04lts and my cluster has 18 datanodes and 1 master.
After running:
$ start-dfs.sh
$ start-yarn.sh
$ jps
On master I get the following:
ResourceManager
NameNode
SecondaryNameNodecode
jps
And on datanodes:
DataNode
jps
All the nodes seems to be live:
NameNode Overview Web Page
But when I reach the Cluster overview, none of my datanodes seems to be active:
Cluster Overview
My configurations files:
core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop-3.2.1/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop-master:9000</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/hadoop-3.2.1/data/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/hadoop-3.2.1/data/datanode</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
The namenode and datanode directories exists on every host (master and datanodes)
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop-master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services </name>
<value> mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
</configuration>
Also I have configured hadoop-env.sh for JAVA_HOME Path and all the other variables are in .bashrc file (also in every host).
I have modified the /etc/hosts file to include all the hosts with their IPs and hostnames and finally I have also modified the workers file to include all the IPs of the datanodes.
The first time I have formatted the NameNode, the directories for the hdfs-site.xml was wrong (I had the datanode dir twice), so hdfs make its own directories under /tmp/hdfs/ (if I remember correctly). But I fixed this with formating again the NameNode with the corect directories.
I installed Hadoop 2.9.0. I checked jps in slave nodes, are running datanode and nodemanager in slaves and namenode and resourcemanager in master. Please help that Why is this error? According to presented solutions, I disabled firewall, SELinux and IPv6. Problem is that Live nodes in web UI is 0.
/etc/hosts:
1 localhost
127.0.1.1 hadoopmaster
#The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.111.154 hadoopmaster
192.168.111... hadoopslave1
192.168.111... hadoopslave2
192.168.111... hadoopslave3
Configuration in all of the nodes:
core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoopmaster:9000</value>
</property>
</configuration>
yarn-site.xml:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoopmaster:8025</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoopmaster:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoopmaster:8050</value>
</property>
</configuration>
Output of netstat -nap | grep 9000:
tcp 0 0 127.0.1.1:9000 0.0.0.0:* LISTEN 18205/java
tcp 0 0 127.0.0.1:59048 127.0.1.1:9000 TIME_WAIT -
I restarted hadoop and formatted namenode again and agin But still I get this error: (hadoop-hadoop-datanode-hadoopslave1.log)
2018-06-11 02:42:33,593 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: hadoopmaster/192.168.111.154:9000. Already tried 4
time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)
I got the solution. edit /etc/hosts
Define FQDN
Remove 127.0.1.1
i am trying to execute the command hadoop dfs -ls, and i got this error
Call From localhost/127.0.0.1 to yass-SATELLITE-C855-2CF:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
once i resolved i got another which is
ls: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type.; Host Details : local host is: "localhost/127.0.0.1"; destination host is: "yass-SATELLITE-C855-2CF":9000;
and i keep in loop between this two errors
my core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://yass-SATELLITE-C855-2CF:9000</value>
</property>
</configuration>
hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.safemode.threshold.pct</name>
<value>0</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/hadoop/data/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/hadoop/data/datanode</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/home/yass/Téléchargements/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
etc/hosts
127.0.0.1 localhost
127.0.0.1 yass-SATELLITE-C855-2CF
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
i got the first exception once it dispair i got the second one and i keep always in loop with this two exceptions
Any suggestions please ?
in reality i find somehow a solution in order to not staying with same errors
so i installed Hadoop verion 2.6.2 ,configure the version using the XML then keep working ,is not good solution for everyone but i hope that will show light for others
I am using hadoop 1.2.1 and hbase 0.94.20.I created a cluster of 5 slave when i started hbase by using ./start-hbase.sh it start the services on one slave but other slaves unable to start...
my Hbase-site.xml file is
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:54310/opt/hbase-0.94.20</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2222</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master</value>
<description>
</description>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/opt/hbase-0.94.20/tmp</value>
</property>
</configuration>
and /etc/hosts file is
127.0.0.1 localhost ubuntu
10.10.73.42 master sitmaster-HP-Compaq-6200-Pro-SFF-PC
10.10.73.5 slave1 sit2-HP-Pro-2110
10.10.73.25 slave2 sit-HP-Pro-2110
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
I am using Hadoop 2.2 on Ubuntu.
I am able to load this link in my browser.
http://[my_ip]:50070/dfshealth.jsp
From there, when I click the "Browse the filesystem" link, I am sent to
http://localhost:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/&nnaddr=127.0.0.1:9000
while here I think I want my_ip instead of localhost and 127.0.0.1
Also, if I type manually
http://my_ip:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/&nnaddr=my_ip:9000
it still does not work.
The my_ip is an external/global IP throughout my whole question text.
How can I get this working? All I want is to be able to browse my HDFS filesystem from the browser.
core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
<!-- <value>hdfs://my_ip:9000</value> -->
</property>
<!--
fs.default.name
hdfs://localhost:9000
-->
</configuration>
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/var/lib/hadoop/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/var/lib/hadoop/hdfs/datanode</value>
</property>
<!--
dfs.replication
1
dfs.namenode.name.dir
file:/var/lib/hadoop/hdfs/namenode
dfs.datanode.data.dir
file:/var/lib/hadoop/hdfs/datanode
-->
<property>
<name>dfs.http.address</name>
<value>my_ip:50070</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>my_ip:50075</value>
</property>
</configuration>
/etc/hosts
127.0.0.1 localhost test02
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
EDIT ERROR:
HTTP ERROR 500
Problem accessing /nn_browsedfscontent.jsp. Reason:
Cannot issue delegation token. Name node is in safe mode.
The reported blocks 21 has reached the threshold 0.9990 of total blocks 21. The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically in 2 seconds.
Caused by:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot issue delegation token. Name node is in safe mode.
The reported blocks 21 has reached the threshold 0.9990 of total blocks 21. The number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically in 2 seconds.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5887)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:447)
at org.apache.hadoop.hdfs.server.namenode.NamenodeJspHelper$1.run(NamenodeJspHelper.java:623)
at org.apache.hadoop.hdfs.server.namenode.NamenodeJspHelper$1.run(NamenodeJspHelper.java:620)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
In your hdfs-site.xml, replace
<property>
<name>dfs.http.address</name>
<value>my_ip:50070</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>my_ip:50075</value>
</property>
by
<property>
<name>dfs.namenode.http-address</name>
<value>localhost:50070</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>localhost:50075</value>
</property>
But usually, in pseudo-ditributed mode it's not necessary to specify those properties.
Reboot your cluster after changing the properties.