I installed hadoop-2.2.0 version on vmware and when it starts, it shows all the processes working, but after some time datanode gets killed. So, I checked the logs and I found this:
2014-01-21 04:36:00,325 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-01-21 04:36:21,034 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.net.UnknownHostException: ubuntu: ubuntu
at java.net.InetAddress.getLocalHost(InetAddress.java:1402)
at org.apache.hadoop.security.SecurityUtil.getLocalHostName(SecurityUtil.java:227)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:247)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1640)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1665)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1837)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1858)
2014-01-21 04:36:21,107 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2014-01-21 04:36:21,114 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at java.net.UnknownHostException: ubuntu: ubuntu
************************************************************/
I think the problem is in your /etc/hosts file.
Make sure it is configured correctly. My /etc/hosts file looks like this:
127.0.0.1 localhost
192.168.0.1 master mypcname
192.168.0.18 slave1
192.168.0.12 slave2
192.168.0.17 slave3
for master and
127.0.0.1 localhost
192.168.0.1 master
192.168.0.18 slave1 slave1pcname
192.168.0.12 slave2
192.168.0.17 slave3
for slave1.
Related
I'm trying to upgrade my 5 node hadoop cluster from 1.0 to 2.2.0. When I try to upgrade the namenode using hadoop-daemon.sh start namenode -upgrade command and check the log files I get the following error message.
015-03-13 10:02:24,549 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:335)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getRpcServerAddress(NameNode.java:388)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:471)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:483)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:684)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:669)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1254)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
2015-03-13 10:02:24,586 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2015-03-13 10:02:24,593 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at nn.cluster.com/192.168.1.75
************************************************************/
Please check your core-site.xml and it should contain valid namenode address.
everyone, I have a little problem when I build the Hadoop Cluster
My node install CentOS 6.5,java1.7.60 and hadoop 2.2.0
I want to build a master and three slaves
I try to build it like this
But in the end of this, I try to start up my namenode and datanode
My /etc/hosts like this:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.10.10.10 master
10.10.10.11 slave1
10.10.10.12 slave2
10.10.10.13 slave3
Just like this when I type:
$ hadoop namenode -fromat:
java.net.UnknownHostException: hadoop01.hadoopcluster: hadoop01.hadoopcluster
at java.net.InetAddress.getLocalHost(InetAddress.java:1473)
at org.apache.hadoop.net.DNS.resolveLocalHostname(DNS.java:264)
at org.apache.hadoop.net.DNS.<clinit>(DNS.java:57)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:914)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
Caused by: java.net.UnknownHostException: hadoop01.hadoopcluster
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
... 8 more
14/07/01 16:50:46 WARN net.DNS: Unable to determine address of the host-falling back to "localhost" address
java.net.UnknownHostException: hadoop01.hadoopcluster: hadoop01.hadoopcluster
at java.net.InetAddress.getLocalHost(InetAddress.java:1473)
at org.apache.hadoop.net.DNS.resolveLocalHostIPAddress(DNS.java:287)
at org.apache.hadoop.net.DNS.<clinit>(DNS.java:58)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:914)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:550)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:144)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
Caused by: java.net.UnknownHostException: hadoop01.hadoopcluster
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
... 8 more
and try to issue start-dfs.sh and start-yarn.sh:
$ start-dfs.sh
14/07/01 16:55:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master]
master: namenode running as process 2395. Stop it first.
slave2: starting datanode, logging to /opt/hadoop/logs/hadoop-hadoop-datanode-hadoop03.hadopcluster.out
slave1: starting datanode, logging to /opt/hadoop/logs/hadoop-hadoop-datanode-hadoop02.hadopcluster.out
slave3: starting datanode, logging to /opt/hadoop/logs/hadoop-hadoop-datanode-hadoop04.hadopcluster.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 2564. Stop it first.
14/07/01 16:55:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/logs/yarn-hadoop-resourcemanager-hadoop01.hadoopcluster.out
slave1: starting nodemanager, logging to /opt/hadoop/logs/yarn-hadoop-nodemanager-hadoop02.hadopcluster.out
slave3: starting nodemanager, logging to /opt/hadoop/logs/yarn-hadoop-nodemanager-hadoop04.hadopcluster.out
slave2: starting nodemanager, logging to /opt/hadoop/logs/yarn-hadoop-nodemanager-hadoop03.hadopcluster.out
and type jps:
$ jps
2564 SecondaryNameNode
5591 Jps
2395 NameNode
I only look like this, didn't have DataNode, NodeManager, ResourceManger... etc,
Is it any where wrong when I setting it?
Could anyone can suggest me something, thanks!
DHCP (Dynamic Host Configuration Protocol)
it is a server deployed on an IP network and it is used to allocate IP addresses to its clients. Therefore you had to configure the DHCP on both the server and clients sides.
On the server side:
getting the package: isc-dhcp-server
editing /etc/network interfaces to configure interfaces, choose a static IP address for the DHCP server
specifying the local network interface DHCP will listen to (/etc/default/isc-dhcp-server)
editing configuration file /etc/dhcp/dhcpd.conf to choose the IP addresses that will be used by the local network, define the DNS server
On the client side:
editing /etc/network interfaces to configure interfaces
You can make sure that it is well installed and configured you can use ifconfig and ping commands.
DNS (Domain Name System)
intalling bind9
editing /etc/bind/named.conf to add hosts
Good luck.
I am running a Red Hat Enterprise Linux Server release 6.4 (Santiago) distribution with Hadoop 1.1.2 installed on it. I have made the required configurations to enable the pseudo distributed mode. But on trying to run hadoop, the datanode and tasktracker don't start.
I am not able to copy any files to hdfs.
[hduser#is-joshbloom-hadoop hadoop]$ hadoop dfs -put README.txt /input
Warning: $HADOOP_HOME is deprecated.
13/05/23 16:42:00 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /input could only be replicated to 0 nodes, instead of 1
Also after trying hadoop-daemon.sh start datanode I get the message:
starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-datanode-is-joshbloom-hadoop.out
same goes for tasktracker. But when I try the same command for namenode, secondarynamenode, jobtracker they are seem to be running.
namenode running as process 32933. Stop it first.
I tried the following solutions:
Reformatting namenode
Reinstalling hadoop
Installing different version of hadoop (1.0.4)
None seem to work. I have followed the same installation steps on my Mac and on amazon ubuntu VM and it works perfectly.
How can I get hadoop working?
Thanks!
*UPDATE**
Here is the log entry of namenode
2013-05-23 16:27:44,087 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = java.net.UnknownHostException: is-joshbloom-hadoop: is-joshbloom-hadoop
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.1.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
************************************************************/
2013-05-23 16:27:44,382 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-05-23 16:27:44,432 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2013-05-23 16:27:44,446 ERROR org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Error getting localhost name. Using 'localhost'...
java.net.UnknownHostException: is-joshbloom-hadoop: is-joshbloom-hadoop
at java.net.InetAddress.getLocalHost(InetAddress.java:1438)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.getHostname(MetricsSystemImpl.java:463)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSystem(MetricsSystemImpl.java:394)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:390)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:152)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:133)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1589)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1608)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1734)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1751)
Caused by: java.net.UnknownHostException: is-joshbloom-hadoop
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:866)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1258)
at java.net.InetAddress.getLocalHost(InetAddress.java:1434)
... 11 more
2013-05-23 16:27:44,453 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-05-23 16:27:44,453 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2013-05-23 16:27:44,768 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-05-23 16:27:44,914 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2013-05-23 16:27:45,212 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.net.UnknownHostException: is-joshbloom-hadoop: is-joshbloom-hadoop
at java.net.InetAddress.getLocalHost(InetAddress.java:1438)
at org.apache.hadoop.security.SecurityUtil.getLocalHostName(SecurityUtil.java:271)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:289)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:301)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1651)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1590)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1608)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1734)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1751)
Caused by: java.net.UnknownHostException: is-joshbloom-hadoop
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:866)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1258)
at java.net.InetAddress.getLocalHost(InetAddress.java:1434)
... 8 more
2013-05-23 16:27:45,228 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at java.net.UnknownHostException: is-joshbloom-hadoop: is-joshbloom-hadoop
************************************************************/
*UPDATE***
content of /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
Amend your /etc/hosts to include a hostname loopback mapping:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
127.0.1.1 is-joshbloom-hadoop
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
Your problem is your machine doesn't know how to resolve the hostname is-joshbloom-hadoop to a specific IP address. There are typically two places/methods resolution occurs - either via a DNS server or using a local hosts file (hosts file takes precedence).
The above amendment to your hosts file allows you machine to resolve the machine name is-joshbloom-hadoop to the IP address 127.0.1.1. The OS has an internal loopback address for the range 127.0.0.0/8, so you could name any address in here. On my Ubuntu laptop, it uses the 127.0.1.1 and i'm sure it changes between OS's, but my guess is by not using 127.0.0.1 you don't have to search for it in the localhost line if you change your machine name in future.
Check your core-site.xml in HADOOP_HOME/conf.It will have fs.default.name property.It should have a hostname specified in your /etc/hosts."is-joshbloom-hadoop" hostname is not there in /etc/hosts.Use localhost instead
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
</property>
</configuration>
Problem seems to be like that u have nothing in slaves file under conf/slaves.
Check your slaves file in conf/slaves.Remove everything and add localhost in that file.Remove the name and data directory mentioned in dfs.name.dir and dfs.data.dir property under hdfs-site.xml.
Format ur HDFS file system and then start ur daemons again.
I am trying to configure a pseudo hadoop 1 node cluster on my ubuntu machine. However when i give the following command
bin/start-all.sh
it start all the daemons but when i do jps,it does not give me namnode port and when i go to the namenode logs
The following message is displayed.
2013-04-26 11:59:09,927 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: Incomplete HDFS URI, no host: hdfs://vikasXXX.XX.XX.XX:X000
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:85)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
at org.apache.hadoop.fs.Trash.<init>(Trash.java:62)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier(NameNode.java:314)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:310)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
2013-04-26 11:59:09,929 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at vikas/XXX.XX.XX.XX
************************************************************/
What could be the reason?
Thanks
I assume that you have masked and shared your URL
hdfs://vikasXXX.XX.XX.XX:X000
I think it is not recognizing your machine by name. Try using localhost and check if it works.
hdfs://localhost:8020
I caught "Temporary failure in name resolution" while run Hadoop/bin/start-all.sh on my SUSE Linux.I have searched many website to look for the problem,but can not find the effective answer. I look forward your help,thank you!!
It is deployed on single same maching,so in both master/slaves files only one line:localhost
solom#linux87:~/hadoop> bin/hadoop namenode -format
11/07/12 17:43:10 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = linux87/10.18.6.87
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707;
compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
Re-format filesystem in /home/solom/tmp/hadoop/dfs/name ? (Y or N)
Y
11/07/12 17:43:13 INFO namenode.FSNamesystem: fsOwner=solom,solom,dialout,video
11/07/12 17:43:13 INFO namenode.FSNamesystem: supergroup=supergroup
11/07/12 17:43:13 INFO namenode.FSNamesystem: isPermissionEnabled=true
11/07/12 17:43:13 INFO common.Storage: Image file of size 95 saved in 0 seconds.
11/07/12 17:43:14 INFO common.Storage: Storage directory
/home/solom/tmp/hadoop/dfs/name has been successfully formatted.
11/07/12 17:43:14 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at linux87/10.18.6.87
************************************************************/
solom#linux87:~/hadoop> bin/start-all.sh
starting namenode, logging to /home/solom/hadoop/bin/../logs/hadoop-solom-namenode-
linux87.out
: Temporary failure in name resolution
: Temporary failure in name resolution
starting jobtracker, logging to
/home/solom/hadoop/bin/../logs/hadoop-solom-jobtracker-linux87.out
: Temporary failure in name resolution
additional:
solom#linux87:~> cat /etc/hosts
#
# hosts This file describes a number of hostname-to-address
# mappings for the TCP/IP subsystem. It is mostly
# used at boot time, when no name servers are running.
# On small systems, this file can be used instead of a
# "named" name server.
# Syntax:
#
# IP-Address Full-Qualified-Hostname Short-Hostname
#
127.0.0.1 localhost
# special IPv6 addresses
::1 localhost ipv6-localhost ipv6-loopback
fe00::0 ipv6-localnet
ff00::0 ipv6-mcastprefix
ff02::1 ipv6-allnodes
ff02::2 ipv6-allrouters
ff02::3 ipv6-allhosts
10.18.6.87 linux87.site linux87
solom#linux87:~>
$ vi -b conf/slaves
Maybe you can know what happened!
I had the same problem and I've resolved it adding to /etc/hosts file the next line:
192.168.56.101 localhost hadoop
where you must change the ip and change hadoop and put your own hostname.