I have a Hadoop cluster running Cloudera's CDH3, Apache Hadoop's 0.20.2 equivalent. I want to restart the job-tracker as there are some jobs which are not getting killed. I tried killing them from the command line, the command executes successfully, but the jobs are still in Job Cleanup: Pending status. Anyways I want to restart the job-tracker and see if that cleanup the jobs. I know the command to restart the job-tracker, but I am not sure if I need to put the name-node in safe-mode before I restart the job-tracker.
You can try to kill the unwanted jobs using hadoop job -kill <Job-ID> and check for command status echo "$?". If that doesn't work, Restart is the only option.
Hadoop Jobtracker and namenodes are independent components, No need to execute namenode safenode before Jobtracker restart. You can restart Jobtracker process alone.(tasktracker if required)
Related
I am getting an error while implementing Hadoop NameNode HA using supervisord
when I run the command without using supervisord, it works fine,
but supervisord seems to be having problems especially with zkfc
The namenode supervisord configuration currently used in NameNode is as follows:
[program:NameNode]
command=/home/deploy/hadoop/bin/hdfs namenode
user=deploy
[program:ResourceManager]
command=/home/deploy/hadoop/bin/yarn resourcemanager
user=deploy
[program:DFSZKFailoverController]
command=/home/deploy/hadoop/bin/hdfs zkfc
user=deploy
What I'm wondering is,
What is the difference between 'hdfs --daemon start {something}' and 'hdfs {something}'?
When I checked the hadoop/bin/hdfs code, I don't know what the difference is.
When the above two commands are executed with supervisord, when the daemon dies supervisord does not automatically failover because supervisord does not recognize the first command, but NameNode HA works when the daemon is alive
The second command is recognized by supervisord and when the daemon dies, supervisord automatically performs failover, but NameNode HA does not work.
It means when an Active NameNode dies, another NameNode does not automatically change to active. By the way, if I restart zkfc of all NameNodes by 'supervisorctl restart', Active NameNode is elected, but it is one-time
If NameNode uses 'hdfs namenode' and zkfc uses 'hdfs --deamon start zkfc', NameNode HA works, but fails over automatically when zkfc dies.
In conclusion, I'm curious as to the difference between the two commands and why zkfc doesn't work normally in the second command.
I have a hadoop cluster 2.7.4 version. Due to some reason, I have to restart my cluster. I need job IDs of those jobs that were executed on cluster before cluster reboot. Command mapred -list provide currently running of waiting jobs details only
You can see a list of all jobs on the Yarn Resource Manager Web UI.
In your browser go to http://ResourceManagerIPAdress:8088/
This is how the history looks on the Yarn cluster I am currently testing on (and I restarted the services several times):
See more info here
Job tracker and Task tracker don't sow up when ran the start-all.sh command in ububtu for hadoop
I do get the rest of the processes while i run the "JPS" command in unix.
Not sure why i am not being shown with the job tracker and task tracker.Have been following couple of links and couldn't get my prob sorted.
Steps done :
-Multiple times formatted the namenode
-Multiple time deleted and recreated the tmp folder with appropriate permissions.
What could be the issue ?
Any suggestions would really help me as i am struggling in setting up hadoop on my laptop.I am new to it though.
Try starting jobtracker and tasktracker separately.
From your hadoop HOME directory run
. bin/../libexec/hadoop-config.sh
Then from hadoop BIN directory run
hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker
hadoop-daemon.sh --config $HADOOP_CONF_DIR start tasktracker
You must have been using hadoop 2.x version where jobtracker is replaced with YARN resource manager. Using jps(jdk is needed) you can check whether resouce manager is running. If it is running then the default url for it is (host-name):8088. You can check your nodes,jobs also configuration there.If not running then start them with sbin/start-yarn.sh.
We have setup a cluster with 6 slave nodes. I am trying to see how replication happens when one of the DataNode dies.
I logged into one of the slave and killed the DataNode using the kill -9 command. After sometime the DataNode is restarted automatically and HDFS gets back into healthy status. I am verify this because the PID of the DataNode has changed.
I don't see any documentation on the above behavior of DataNode. Is this the Apache Hadoop or Cloudera CDH feature? Any reference to the documentation is appreciated.
As the pid of datanode has been changed, I don't think it is a behavior of datanode. If you are managing your cluster using Cloudera Manager, there is an option for restarting datanode daemon if it fails(Automatically Restart Process). This option will be set by default. When the datanode process gets failed or killed, As Automatic restart option is set Cloudera Scm agent will start the the datanode daemon.
For Automatic restart option : Choose HDFS services -> go to Configuration section -> Search for automatic restart.
This feature is available in CM 4.X release as well.
How do I restart the task trackers and job tracker using CDH4 from the command line?
I tried following given script but got error
[root#adc1210765 bin]# ./stop-mapred.sh
/usr/lib/hadoop-0.20-mapreduce/bin/../conf no jobtracker to stop cat:
/usr/lib/hadoop-0.20-mapreduce/bin/../conf/slaves: No such file or
directory
I want to restart all instance of tasktracker running at my cluster nodes
You must do this on each of the task trackers
sudo service hadoop-0.20-mapreduce-tasktracker restart
And on the job tracker
sudo service hadoop-0.20-mapreduce-jobtracker restart
You can also use stop and start in place of restart. Might have to change your hadoop version number.
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_3.html?scroll=topic_11_3
You can also try starting the JobTracker Daemon by
/etc/init.d/hadoop-0.20-mapreduce-jobtracker start
and Task Tracker by
/etc/init.d/hadoop-0.20-mapreduce-tasktracker start
Ensure the version number is appropiate.