I have a hdoop cluster of 6 nodes.
I node is a master nodes. All the nodes, including master node, are running TaskTracker & DataNode.
The command
hadoop dfsadmin -report
Return only 3 data nodes including the DataNode running at master node. There are no errors on the console. Master node can SSH to these nodes and DataNode is running on the unreported nodes. I checked SSH loging to the un-reported nodes and by executing he JPS command. All the slave nodes have the following output:
ubuntu#slave2:¬/hadoop/logs$ jps
5007 TaskTracker
4868 DataNode
5261 Jps
How to look for unreported DataNodes in the dfsadmin command?
Related
my question might be duplicated, i had searched similar question here but it didn't solve my problem.
so i'm new in hadoop, now i'm setting up multinode cluster which is 1 master and 2 slaves. when i run jps command on master node, my terminal shows this
3250 - DataNode
3090 - NameNode
4099 - jps
3498 - SecondaryNameNode
and when i run jps command on slaves node, my terminal shows this
3896 - DataNode
4684 - jps
4111 - SecondaryNameNode
according to this tutorial link, my master node would have the this output
jps
namenode
secondary namenode
resource manager
and my slaves node would have this
jps
NodeManager
DataNode
so on master node there is no resource manager and on slaves node there is no node manager
edit
when i start start-dfs.sh command, the output of my terminal shows this
Starting namenodes on [HadoopMaster]
starting datanodes
starting secondary namenodes [farhan-master]
and when i start start-yarn.sh command, the output of my terminal shows this
starting resourcemanager
starting nodemanager
how do i solve this problem? thanks in advance
iam using hadoop apache 2.7.1
on centos 7 environment
and i have an HA Cluster which consists of two name nodes(mn1 and mn2)
and 6 data nodes
issuing jps on mn1 shows
34734 DFSZKFailoverController
34245 NameNode
31529 DFSAdmin
34551 JournalNode
34822 Jps
3857 QuorumPeerMain
and issuing jps on mn2 shows
26272 JournalNode
26483 Jps
26110 NameNode
26388 DFSZKFailoverController
2259 QuorumPeerMain
what does DFSAdmin Process in mn1 jps output refers to ?
i noticed that this dfsadmin process appeared in the following scenario:
when number of failed journal nodes exceeds possible numbers of journal nodes for cluster to continue working
Which is defined in formula
(N-1)/ 2 where N is number of journal nodes
So the cluster will not continue to work and active name node shut downs
so if i started accepted number of journal nodes and name node again
jps on active name node shows dfsadmin process
and this problem was solved by restarting all cluster services a gian
I have Hadoop-2.7.1 and I have configured a cluster consists of three nodes.
when I call jps command on name node i am getting the following output
3234 SecondaryNameNode
3039 NameNode
9019 Jps
3382 ResourceManager
calling jps command on secondary name node output is
4720 DataNode
4826 NodeManager
4949 Jps
calling jps command on data node output is
4824 Jps
4587 DataNode
4701 NodeManager
Is this output right? why jps shows secondarynamenode on name node and showing data node on secondary name node
isn't there any conflict!
It looks like you have used start-all.sh or start-dfs.sh to start the daemons and have not set the property dfs.namenode.secondary.http-address in hdfs-site.xml.
In that case, secondarynamenode will be started in the same node from where the start-dfs(all).sh script is executed. To start it in a different node, add this property to hdfs-site.xml
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>secondary_namenode_hostname:50090</value>
</property>
Datanodes are started based on the hostname(s) listed in the slaves file.
Alternatively, use hadoop-daemon.sh and yarn-daemon.sh scripts to start the specific HDFS and YARN services respectively on each node.
I followed this tutorial and tried to setup a multinode hadoop cluster on centOS. After doing all the configurations and running start-dfs.sh and start-yarn.sh, this is what jps outputs:
Master
26121 ResourceManager
25964 SecondaryNameNode
25759 NameNode
25738 Jps
Slave 1
19082 Jps
17826 NodeManager
Slave 2
17857 Jps
16650 NodeManager
Data node is not started on slaves.
Can anyone suggest what is wrong with this setup?
We have a Hadoop setup with 2 Master nodes and 1 Slave node.
We have configured the Hadoop cluster. After configuring, when we executed "jps" command, we are getting following output on my Master Node:
13405 NameNode
14614 Jps
13860 ResourceManager
13650 DataNode
14083 NodeManage
On my second Master Node, output is:
9698 Jps
9234 DataNode
9022 NameNode
9450 NodeManager
On my Data Node, output is:
21681 NodeManager
21461 DataNode
21878 Jps
I feel my secondary node is not running. Please tell me this is right or wrong. If its wrong, what should be the status of my node? Please answer me as soon as possible.
You can check status of node by running below command
hdfs haadmin -getServiceState