I have Hadoop-2.7.1 and I have configured a cluster consists of three nodes.
when I call jps command on name node i am getting the following output
3234 SecondaryNameNode
3039 NameNode
9019 Jps
3382 ResourceManager
calling jps command on secondary name node output is
4720 DataNode
4826 NodeManager
4949 Jps
calling jps command on data node output is
4824 Jps
4587 DataNode
4701 NodeManager
Is this output right? why jps shows secondarynamenode on name node and showing data node on secondary name node
isn't there any conflict!
It looks like you have used start-all.sh or start-dfs.sh to start the daemons and have not set the property dfs.namenode.secondary.http-address in hdfs-site.xml.
In that case, secondarynamenode will be started in the same node from where the start-dfs(all).sh script is executed. To start it in a different node, add this property to hdfs-site.xml
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>secondary_namenode_hostname:50090</value>
</property>
Datanodes are started based on the hostname(s) listed in the slaves file.
Alternatively, use hadoop-daemon.sh and yarn-daemon.sh scripts to start the specific HDFS and YARN services respectively on each node.
Related
my question might be duplicated, i had searched similar question here but it didn't solve my problem.
so i'm new in hadoop, now i'm setting up multinode cluster which is 1 master and 2 slaves. when i run jps command on master node, my terminal shows this
3250 - DataNode
3090 - NameNode
4099 - jps
3498 - SecondaryNameNode
and when i run jps command on slaves node, my terminal shows this
3896 - DataNode
4684 - jps
4111 - SecondaryNameNode
according to this tutorial link, my master node would have the this output
jps
namenode
secondary namenode
resource manager
and my slaves node would have this
jps
NodeManager
DataNode
so on master node there is no resource manager and on slaves node there is no node manager
edit
when i start start-dfs.sh command, the output of my terminal shows this
Starting namenodes on [HadoopMaster]
starting datanodes
starting secondary namenodes [farhan-master]
and when i start start-yarn.sh command, the output of my terminal shows this
starting resourcemanager
starting nodemanager
how do i solve this problem? thanks in advance
iam using hadoop apache 2.7.1
on centos 7 environment
and i have an HA Cluster which consists of two name nodes(mn1 and mn2)
and 6 data nodes
issuing jps on mn1 shows
34734 DFSZKFailoverController
34245 NameNode
31529 DFSAdmin
34551 JournalNode
34822 Jps
3857 QuorumPeerMain
and issuing jps on mn2 shows
26272 JournalNode
26483 Jps
26110 NameNode
26388 DFSZKFailoverController
2259 QuorumPeerMain
what does DFSAdmin Process in mn1 jps output refers to ?
i noticed that this dfsadmin process appeared in the following scenario:
when number of failed journal nodes exceeds possible numbers of journal nodes for cluster to continue working
Which is defined in formula
(N-1)/ 2 where N is number of journal nodes
So the cluster will not continue to work and active name node shut downs
so if i started accepted number of journal nodes and name node again
jps on active name node shows dfsadmin process
and this problem was solved by restarting all cluster services a gian
I followed this tutorial and tried to setup a multinode hadoop cluster on centOS. After doing all the configurations and running start-dfs.sh and start-yarn.sh, this is what jps outputs:
Master
26121 ResourceManager
25964 SecondaryNameNode
25759 NameNode
25738 Jps
Slave 1
19082 Jps
17826 NodeManager
Slave 2
17857 Jps
16650 NodeManager
Data node is not started on slaves.
Can anyone suggest what is wrong with this setup?
We have a Hadoop setup with 2 Master nodes and 1 Slave node.
We have configured the Hadoop cluster. After configuring, when we executed "jps" command, we are getting following output on my Master Node:
13405 NameNode
14614 Jps
13860 ResourceManager
13650 DataNode
14083 NodeManage
On my second Master Node, output is:
9698 Jps
9234 DataNode
9022 NameNode
9450 NodeManager
On my Data Node, output is:
21681 NodeManager
21461 DataNode
21878 Jps
I feel my secondary node is not running. Please tell me this is right or wrong. If its wrong, what should be the status of my node? Please answer me as soon as possible.
You can check status of node by running below command
hdfs haadmin -getServiceState
when i give the command
for service in /etc/init.d/hadoop*
>do
>sudo $service stop
>done
its stops all the service
and when i give
for service in /etc/init.d/hadoop-hdfs-*
>do
>sudo $service stop
>done
its stops all the service
it sometimes start datanode and sometimes namenode
eg:
21270 NameNode
21422 Jps
21374 SecondaryNameNode
2624 HMaster
or
11070 DataNode
11422 Jps
11554 SecondaryNameNode
2554 HMaster
same thing happens for jobtracker and tasktracker
I tried formating the namenode but it didnt help
I also changing the path of localhost in
core-site.xml from 8020 to 50020
and also in mapred-site.xml from 8021 to 50020
this time it shows NameNode, DataNode, JobTracker,Tasktracker using jps
but when i check the browser localhost:50070 and localhost:50030
it refers to 8020 instead of 50020.
why is this happening ?
please help
Run the following script from terminal to stop the running hadoop daemons.
> $HADOOP_INSTALL/hadoop/bin/stop-all.sh
Run the following script from terminal to start the hadoop daemons.
$HADOOP_INSTALL/hadoop/bin/start-all.sh