I've tried to install hadoop in my system and first i was getting permissions that i was able to resolve by just a simple chmod and chown but after solving this issue now there is a new error that arises whenever I use start-dfs.sh
kishan#RoCk ~ $ start-dfs.sh
17/04/08 12:22:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
localhost: starting namenode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-kishan-namenode-RoCk.out
localhost: starting datanode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-kishan-datanode-RoCk.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-kishan-secondarynamenode-RoCk.out
17/04/08 12:22:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
kishan#RoCk ~ $ jps
10303 Jps
it's a warning but all the nodes are not running.
UPDATE:
Namenode ERROR log:
2017-04-09 21:32:40,002 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/kishan/hdfs/namenode is in an inconsistent state: storage directory does not exist or is not accessible.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:327)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:215)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:812)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:796)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
2017-04-09 21:32:40,003 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/kishan/hdfs/namenode is in an inconsistent state: storage directory does not exist or is not accessible
Your Namenode needs a format.
hdfs namenode -format
If the Datanode data directories were already created, manually remove them before re-starting the cluster.
Related
I am new in learning Hadoop. And I meet some problems in its configuration. Before it, I finished the configuration of JAVA, SSH, core-site.xml, hdfs-site.xml and hadoop-env.sh. Please tell me how can I solve it. Thank you very much.
root#host1-virtual-machine:/home/host1/usr/hadoop-3.2.1# sbin/start-all.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [host1-virtual-machine]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2020-02-12 13:19:37,322 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
After I add 5 lines into hadoop-env.sh, it warns the following:
Starting namenodes on [localhost]
localhost: root#localhost: Permission denied (publickey,password).
Starting datanodes
localhost: root#localhost: Permission denied (publickey,password).
Starting secondary namenodes [host1-virtual-machine]
host1-virtual-machine: root#host1-virtual-machine: Permission denied (publickey,password).
2020-02-12 20:26:27,702 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Starting nodemanagers
localhost: root#localhost: Permission denied (publickey,password).
The reason for this issue is use of different user in installation and for starting the service.
You can define the users as root in hadoop-env.sh as below:
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
Hope this helps
I'm using mac and java version:
$java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
followed this link: https://dtflaneur.wordpress.com/2015/10/02/installing-hadoop-on-mac-osx-el-capitan/
I first brew install hadoop, config ssh connection and xml files as required, and
start-dfs.sh
start-yarn.sh
The screen output is like this:
$start-dfs.sh
17/05/06 09:58:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 74213. Stop it first.
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.7.3/libexec/logs/hadoop-x-datanode-xdeMacBook-Pro.local.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 74417. Stop it first.
17/05/06 09:58:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
$start-dfs.sh
17/05/06 09:58:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 74213. Stop it first.
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.7.3/libexec/logs/hadoop-x-datanode-xdeMacBook-Pro.local.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 74417. Stop it first.
17/05/06 09:58:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Then using jps I cannot see "DataNode" and "ResourceManager". I suppose DataNode is hdfs module and ResourceManager is yarn module:
$jps
74417 SecondaryNameNode
75120 Jps
74213 NameNode
74539 ResourceManager
74637 NodeManager
I can list hdfs files:
$hdfs dfs -ls /
17/05/06 09:58:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x - x supergroup 0 2017-05-05 23:50 /user
But running the pi examples throws exception:
$hadoop jar /usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar pi 2 5
Number of Maps = 2
Samples per Map = 5
17/05/06 10:19:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/06 10:19:49 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/x/QuasiMonteCarlo_1494037188550_135794067/in/part0 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
I wonder if I missed any configuation, how can I make sure that they run successfully, and how to check or trouble shoot possible failure reasons?
Thanks.
I am too in learning phase yet. This error comes when there is no datanode available to read/write.
You can check Resource Manager using this URL: http://localhost:50070
Is there any datanode running or not.
For trouble shooting you can check logs generated under installation directory of hadoop . If you can share that logs i can try to help.
I installed Hadoop 2.7.2 on Ubuntu 16.04 in single mode. But neither NameNode nor DataNode Services run after starting the Hadoop.
hduser#saber-Studio-1435:/usr/local/hadoop$ start-all.sh
This script is Deprecated.
Instead use start-dfs.sh and start-yarn.sh
16/06/20 15:34:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-saber-Studio-1435.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-saber-Studio-1435.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 7214. Stop it first.
16/06/20 15:35:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
resourcemanager running as process 7374. Stop it first.
localhost: nodemanager running as process 7502. Stop it first.
Status:
hduser#saber-Studio-1435:/usr/local/hadoop$ jps
8747 Jps
7502 NodeManager
7374 ResourceManager
7214 SecondaryNameNode
First stop the hadoop $HADOOP_HOME ./sbin/stop-all.sh
Then format the hadoop ecosytem
./bin/hadoop namenode -format
./bin/hadoop datanode -format
./bin/hdfs namenode -format
./bin/hdfs datanode -format
Then start agian using ./sbin/start-all.sh
Then try jps on cli and if still does'nt works then remove the directory created for hdfs and recreate it using mkdir -p
I installed hadoop single node, and now Im starting the cluster with start-dfs.sh command.
But jobotracker and tasktracker are not appearing with jps command, so it seems that they are not starting.
Do you see why? Im installing the version 2.6.2...
After execute the command start-dfs.sh, this appears:
[hadoopadmin#hadoop ~]$ start-dfs.sh
16/03/23 12:17:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop-2.6.2/logs/hadoop-hadoopadmin-namenode-hadoop.out
localhost: starting datanode, logging to /usr/local/hadoop-2.6.2/logs/hadoop-hadoopadmin-datanode-hadoop.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.6.2/logs/hadoop-hadoopadmin-secondarynamenode-hadoop.out
16/03/23 12:17:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoopadmin#hadoop ~]$ jps
2881 DataNode
2758 NameNode
3142 Jps
3039 SecondaryNameNode
[hadoopadmin#hadoop ~]$
There is no JobTracker and TaskTracker anymore. We have NodeManager and resourceManager. Here you just started dfs services not started yarn services, to start yarn services run start-yarn.sh then only yarn related services will start.
If you want to start all services run start-all.sh (not a good practice)
Curently, I have installed Hadoop in my Ubuntu system. And I started it. Here are the details:
krish#krish-VirtualBox:~$ start-dfs.sh
14/10/20 13:16:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-krish-namenode-krish-VirtualBox.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-krish-datanode-krish-VirtualBox.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-krish-secondarynamenode-krish-VirtualBox.out
14/10/20 13:16:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
krish#krish-VirtualBox:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-krish-resourcemanager-krish-VirtualBox.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-krish-nodemanager-krish-VirtualBox.out
krish#krish-VirtualBox:~$ jps
3065 NodeManager
2800 SecondaryNameNode
2941 ResourceManager
3307 Jps
2497 NameNode
krish#krish-VirtualBox:~$
I just want to know if all things are perfect in it. I do not see Datanode in the checklist.
stop the cluster .
if you have specifically defined tmp directory location in core-site.xml then remove all files under those directory .
if you have specifically defined data node and namenode directory in hdfs-site.xml then delete all the files under those directories .
if you have not defined anything in core-site or hdfs-site then please remove all the files under /tmp/hadoop-*nameofyourhadoop user
format the namenode
it should work