I have installed the Hadoop Cluster which is the hadoop 0.23.9 version. I install the HDFS-1943.patch and now I can start all the namenode and datanode. (start-dfs.sh is working for me)
However, when I want to start the yarn daemons (running start-yarn.sh) , it shows the following error as the same as the previous happening:
[root#dbnode1 sbin]# ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hchen/hadoop-0.23.9/logs/yarn-root- resourcemanager-dbnode1.out
datanode: starting nodemanager, logging to /home/hchen/hadoop-0.23.9/logs/yarn-root-nodemanager-dbnode2.out
datanode: Unrecognized option: -jvm
datanode: Error: Could not create the Java Virtual Machine.
datanode: Error: A fatal exception has occurred. Program will exit.
I have installed the patch already and start-dfs.sh is working for me. Why start-yarn.sh does not work??
Run HDFS as a non-root user with the appropriate permissions. Here is a JIRA with more details.
Related
It is my first time in installing Hadoop on my Linux (Fedora distro) running on VM (using Parallel on my Mac). And I followed every step on this video and including the textual version of it.And then when I run it on localhost (or the equivalent value from hostname) in port 50070, I got the following message.
...can't establish a connection to the server at localhost:50070
When I run the jps by the way command I don't have the datanode and namenode unlike at the end of the textual version tutorial which has the following:
While mine has only the following processes running:
6021 NodeManager
3947 SecondaryNameNode
5788 ResourceManager
8941 Jps
When I run the hadoop namenode command I have some of the following [redacted] error:
Cannot access storage directory /usr/local/hadoop_store/hdfs/namenode
16/10/11 21:52:45 WARN namenode.FSNamesystem: Encountered exception loading fsimage
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /usr/local/hadoop_store/hdfs/namenode is in an inconsistent state: storage directory does not exist or is not accessible.
I tried to access by the way the above mentioned directories and it existed.
Any hint for this newbie? ;-)
You would need to give read and write permission to user with which you are running the services on directory /usr/local/hadoop_store/hdfs/namenode.
Once done, you should run format command using hadoop namenode -format
Then try to start your services.
delete files /app/hadoop/tmp/*
and try again formatting the namenode and then start-dfs.sh & start-yarn.sh
I'm trying to install Hadoop on Mac, following this post.
Everything looks fine, but in the end, I cannot start the ResourceManager.
i.e. after running $jps.
It is supposed to show something like below.
2507 ResourceManager ---------this is missing (not started)
1712 SecondaryNameNode
1412 NameNode
1540 DataNode
2045 NodeManager
2858 Jps
Here is the error message I got:
starting resourcemanager, logging to /usr/local/Cellar/hadoop/2.6.0/libexec/logs/yarn-myUserName-resourcemanager-Mac.out
nohup: can't detach from console: No such file or directory
FYI， I've also changed the ownership of this hadoop folder. (as follows)
$mkdir -p /usr/local/Cellar/hadoop/2.6.0/hadoop_data/hdfs/namenode
$mkdir -p /usr/local/Cellar/hadoop/2.6.0/hadoop_data/hdfs/datanode
$sudo chown -R Tiger /usr/local/Cellar/hadoop
Any suggestions?
that's wired, I just retried to start the resourcemanager, it works !!!! thought the nodemanager stoped. sorry for bothering, Omigo
I would like to see if the hdfs file system for Hadoop is working properly. I know that jps lists the daemons that are running, but I don't actually know which daemons to look for.
I ran the following commands:
$HADOOP_PREFIX/sbin/hadoop-daemon.sh start namenode
$HADOOP_PREFIX/sbin/hadoop-daemon.sh start datanode
$HADOOP_PREFIX/sbin/yarn-daemon.sh start resourcemanager
$HADOOP_PREFIX/sbin/yarn-daemon.sh start nodemanager
Only namenode, resourcemanager, and nodemanager appeared when I entered jps.
Which daemons are supposed to be running in order for hdfs/Hadoop to function? Also, what could you do to fix hdfs if it is not running?
Use any of the following approaches for to check your deamons status
JPS command would list all active deamons
the below is the most appropriate
hadoop dfsadmin -report
This would list down details of datanodes which is basically in a sense your HDFS
cat any file available in hdfs path.
So, I spent two weeks validating my setup (it was fine) , finally found this command:
sudo -u hdfs jps
Initially my simple JPS command was showing only one process, but Hadoop 2.6 under Ubuntu LTS 14.04 was up. I was using 'Sudo' to run the startup scripts.
Here is the startup that work with JPS listing multiple processes:
sudo su hduser
/usr/local/hadoop/sbin/start-dfs.sh
/usr/local/hadoop/sbin/start-yarn.sh
I am using hadoop-0.20.203, after did required changes when i start the hdfs it throws following warning while start up
root#master:/usr/local/hadoop-0.20.203# bin/start-all.sh
starting namenode, logging to /usr/local/hadoop-0.20.203/bin/../logs/hadoop-root-namenode-master.out
master: starting datanode, logging to /usr/local/hadoop-0.20.203/bin/../logs/hadoop-root-datanode-master.out
master: Unrecognized option: -jvm
master: Error: Could not create the Java Virtual Machine.
master: Error: A fatal exception has occurred. Program will exit.
master: starting secondarynamenode, logging to /usr/local/hadoop-0.20.203/bin/../logs/hadoop-root-secondarynamenode-master.out
starting jobtracker, logging to /usr/local/hadoop-0.20.203/bin/../logs/hadoop-root-jobtracker-master.out
master: starting tasktracker, logging to /usr/local/hadoop-0.20.203/bin/../logs/hadoop-root-tasktracker-master.out
Run the script normally as a non-root user. Ensure your non-root user has appropriate permissions. Refer to this bug report for more information
https://issues.apache.org/jira/browse/HDFS-1943
Use sudo bin/start-all.sh and see if it helps. Ideally you should avoid running Hadoop as root user.
I am trying with installing CDH4 using tarball version , but facing issues as in steps taken by me are as below :
i downloaded tarball from link https://ccp.cloudera.com/display/SUPPORT/CDH4+Downloadable+Tarballs
i first untar the hadoop-0.20-mapreduce-0.20.2+1341 tar file
i did with configuration changes in
hadoop-0.20-mapreduce-0.20.2+1341 since i wanted mrv1 not yarn .
the first thing as per mentioned in cdh4 installation was to configure HDFS
i made the relevant changes in
core-site.xml
hdfs-site.xml
mapred-site.xml
masters --- which is my namenode
slaves ---- my datanodes
copied the hadoop configurations on all the nodes in the cluster
did a namenode format .
after format i had to start the cluster , but in the bin folder could not
find start-all.sh script . so in that case i started with command
bin/start-mapred.sh
in the logs it shows jobtracker started and tasktracker started on slave nodes
but when i do a jps
i can see only
jobtracker
jps
further going did a datanode start on the datanode with below command
bin/hadoop-daemon.sh start datanode .
it shows datanode started .
Namenode not getting started , tasktracker not getting started .
when i checked with my logs i could see
ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.FileNotFoundException: webapps/hdfs not found in CLASSPATH
not sure what is stopping my cluster to work .
earlier i had a cdh3 running . so i stopped the cdh3 cluster . Then i started with installing cdh4 . Also i changed all the directories hdfs-site.xml i.e. pointed it new empty directories for namenode and datanode and not the used the ones defined in cdh3.
but still nothing seems to help .
Also i turned off firewall since i do have a root access but same thing it did not work for me .
Any help on above will be great help.
thank you for kind reply but
I do not have
start-dfs.sh file in bin folder
only files in /home/hadoop-2.0.0-mr1-cdh4.2.0/bin folder are as
start-mapred.sh
stop-mapred.sh
hadoop-daemon.sh
hadoop-daemons.sh
hadoop-config.sh
rcc
slaves.sh
hadoop
command now i am using are as below
for starting datanode :
for x in /home/hadoop-2.0.0-mr1-cdh4.2.0/bin/hadoop-* ; do $x start datanode ; done ;
for starting namenode :
bin/start-mapred.sh
still i am working on the same issue .
Hi sorry for the above misunderstanding the following commands can be run to start your datanodes and namenode
To start namenode:
hadoop-daemon.sh start namenode
To start datanode:
hadoop-daemons.sh start datanode
To start secondarynamenode:
hadoop-daemons.sh --hosts masters start secondarynamenode
The jobtracker demon will get started in your master node and tasktraker demons will get started in each of your datanodes after you run the command
bin/start-mapred.sh
In Hadoop Cluster Setup only jobtacker demon will be show by JPS command in masternode and in each of your datanodes you can see Tasktracker demons runnig by using JPS command.
Then you have to start HDFS by running the following command in your masternode
bin/start-dfs.sh
This command will start namenode demon in you namenode machine (in this configuration your masternode itself I believe) and Datanode demons are started in each of your slave nodes.
Now you can run JPS on each of your datanodes and it will give output
tasktracker
datanode
jps
I think this link will be usefull
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/