I am trying to run the simple-yarn-app at https://github.com/hortonworks/simple-yarn-app in the Managed Mode.
I have copied the jar from local to hdfs like
$ hadoop fs -copyFromLocal simple-yarn-app-1.0-SNAPSHOT.jar /simple-yarn-app-1.0-SNAPSHOT.jar
When I try to run it like
hadoop jar simple-yarn-app-1.0-SNAPSHOT.jar com.hortonworks.simpleyarnapp.Client /bin/date 2 /simple-yarn-app-1.0-SNAPSHOT.jar
I get FAILED as the state. I looked for the log at $HADOOP_HOME/logs, but I do not find an associated log.
Will there be a log present somewhere in the HDFS?
Below is the error I get:
`[[2~14/07/10 12:23:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/10 12:23:32 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
Submitting application application_1405002979164_0004
14/07/10 12:23:32 INFO impl.YarnClientImpl: Submitted application application_1405002979164_0004 to ResourceManager at /0.0.0.0:8032
Application application_1405002979164_0004 finished with state FAILED at 1405013016326
I get the following Diagnostic Report,
----------------Failure Diagnostics----------------
Application application_1405062676632_0002 failed 2 times due to AM Container for appattempt_1405062676632_0002_000002 exited with exitCode: -1000 due to: .
Failing this attempt..
Failing the application.
How should I debug or fix this?
Just in case if anybody else is going through the same problem, when running in managed mode for this application, upload jar to hdfs and give hdfs path.
bin/hadoop jar simple-yarn-app-1.0-SNAPSHOT.jar com.hortonworks.simpleyarnapp.Client /bin/date 2 hdfs://XXX/user/XYZ/apps/simple/simple-yarn-app-1.0-SNAPSHOT.jar
Related
I installed Hadoop and Spark via Homebrew
$ brew list --versions | grep spark
apache-spark 2.2.0
$ brew list --versions | grep hadoop
hadoop 2.8.1 2.8.2 hdfs
where Hadoop 2.8.2 is what I am using.
I followed this post to configure Hadoop. Also, followed this post to configure spark.yarn.archive as:
spark.yarn.archive hdfs://localhost:9000/user/panc25/spark-jars.zip
The following are my Hadoop/Spark related environment setting in my .bash_profile :
# ---------------------
# Hadoop
# ---------------------
export HADOOP_HOME=/usr/local/Cellar/hadoop/2.8.2
export YARN_CONF_DIR=$HADOOP_HOME/libexec/etc/hadoop/
alias hadoop-start="$HADOOP_HOME/sbin/start-dfs.sh;$HADOOP_HOME/sbin/start-yarn.sh"
alias hadoop-stop="$HADOOP_HOME/sbin/stop-yarn.sh;$HADOOP_HOME/sbin/stop-dfs.sh"
# ---------------------
# Apache Spark
# ---------------------
export SPARK_HOME=/usr/local/Cellar/apache-spark/2.2.0/libexec
export PATH=$SPARK_HOME/../bin:$SPARK_HOME/sbin:$PATH
I can successfully start hadoop (hdfa + yarn):
$ hadoop-start
17/11/12 17:08:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-panc25-namenode-mbp13mid2017.local.out
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-panc25-datanode-mbp13mid2017.local.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-panc25-secondarynamenode-mbp13mid2017.local.out
17/11/12 17:08:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/yarn-panc25-resourcemanager-mbp13mid2017.local.out
localhost: starting nodemanager, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/yarn-panc25-nodemanager-mbp13mid2017.local.out
$ jps
92723 NameNode
93188 Jps
93051 ResourceManager
93149 NodeManager
92814 DataNode
92926 SecondaryNameNode
However, when I start spark-shell --master yarn it seems to freeze and I don't know what is going on:
What is wrong?
BTW, I could visit the SparkUI http://localhost:4040/, but all pages are blank.
I experienced a similar issue an was caused by the fact that I forgot to append /conf to HADOOP_CONF_DIR env variable (/etc/hadoop/conf).
In my case I was running spark 2.1 cloudera distribution and specified HADOOP_CONF_DIR=/etc/hadoop/conf/:/etc/hive/conf/ . Due to some reason it was getting stuck so I modified it to HADOOP_CONF_DIR=/etc/hadoop/conf/ and it worked. Still looking for the root cause !
I'm using mac and java version:
$java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
followed this link: https://dtflaneur.wordpress.com/2015/10/02/installing-hadoop-on-mac-osx-el-capitan/
I first brew install hadoop, config ssh connection and xml files as required, and
start-dfs.sh
start-yarn.sh
The screen output is like this:
$start-dfs.sh
17/05/06 09:58:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 74213. Stop it first.
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.7.3/libexec/logs/hadoop-x-datanode-xdeMacBook-Pro.local.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 74417. Stop it first.
17/05/06 09:58:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
$start-dfs.sh
17/05/06 09:58:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 74213. Stop it first.
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.7.3/libexec/logs/hadoop-x-datanode-xdeMacBook-Pro.local.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 74417. Stop it first.
17/05/06 09:58:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Then using jps I cannot see "DataNode" and "ResourceManager". I suppose DataNode is hdfs module and ResourceManager is yarn module:
$jps
74417 SecondaryNameNode
75120 Jps
74213 NameNode
74539 ResourceManager
74637 NodeManager
I can list hdfs files:
$hdfs dfs -ls /
17/05/06 09:58:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x - x supergroup 0 2017-05-05 23:50 /user
But running the pi examples throws exception:
$hadoop jar /usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar pi 2 5
Number of Maps = 2
Samples per Map = 5
17/05/06 10:19:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/06 10:19:49 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/x/QuasiMonteCarlo_1494037188550_135794067/in/part0 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
I wonder if I missed any configuation, how can I make sure that they run successfully, and how to check or trouble shoot possible failure reasons?
Thanks.
I am too in learning phase yet. This error comes when there is no datanode available to read/write.
You can check Resource Manager using this URL: http://localhost:50070
Is there any datanode running or not.
For trouble shooting you can check logs generated under installation directory of hadoop . If you can share that logs i can try to help.
I've tried to install hadoop in my system and first i was getting permissions that i was able to resolve by just a simple chmod and chown but after solving this issue now there is a new error that arises whenever I use start-dfs.sh
kishan#RoCk ~ $ start-dfs.sh
17/04/08 12:22:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
localhost: starting namenode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-kishan-namenode-RoCk.out
localhost: starting datanode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-kishan-datanode-RoCk.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-kishan-secondarynamenode-RoCk.out
17/04/08 12:22:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
kishan#RoCk ~ $ jps
10303 Jps
it's a warning but all the nodes are not running.
UPDATE:
Namenode ERROR log:
2017-04-09 21:32:40,002 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/kishan/hdfs/namenode is in an inconsistent state: storage directory does not exist or is not accessible.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:327)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:215)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:812)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:796)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
2017-04-09 21:32:40,003 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/kishan/hdfs/namenode is in an inconsistent state: storage directory does not exist or is not accessible
Your Namenode needs a format.
hdfs namenode -format
If the Datanode data directories were already created, manually remove them before re-starting the cluster.
I have installed Hadoop-2.4.0 on sigle node cluster. After starting the dfs and yarn and executing the jps I get the following services running..
6584 ResourceManager
5976 NameNode
6706 NodeManager
6407 SecondaryNameNode
6148 DataNode
7471 Jps
When I try to execute the following command I get the error
hduser#dhruv-VirtualBox:/usr/local/hadoop$ bin/hdfs dfs -mkdir /hello
OpenJDK 64-Bit Server VMwarning: You have loaded llibrary
/usr/local/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack' -c
', or link it with '-z noexecstack. 14/10/22 12:21:36 WARN
util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin java classes where applicable.
Can please somebody suggest me, what is wrong and how to rectify this ?
Thanks
Dhruv
You can ignore that if you want. It means that you are running a 32-bit native libraries on a 64-bit runtime.
If the log still annoys you, then you need to build those native libraries on a 64-bit environment.
Am trying to install Hortonworks Hadoop single node cluster. I am able to start namenode and secondary namenode, but datanode failed with the following error. How do I solve this issue?
2014-04-04 18:22:49,975 FATAL datanode.DataNode (DataNode.java:secureMain(1841)) - Exception in secureMain
java.lang.RuntimeException: Although a UNIX domain socket path is configured as /var/lib/hadoop-hdfs/dn_socket, we cannot start a localDataXceiverServer because libhadoop cannot be loaded."
See Native Libraries Guide. Make sure libhadoop.so is available in $HADOOP_HOME\bin. Look into the logs for this message:
INFO util.NativeCodeLoader - Loaded the native-hadoop library
If instead you find
INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
then it means the libhadoop.so is not available, and you'll have to investigate why. Alternatively you can turn off HDFS shortcircuit if you wish, or enable the legacy short-circuit instead using dfs.client.use.legacy.blockreader.local, to remove the libhadoop dependency. But I reckon would be better to find out what's the problem with your library.
Make sure you read and understand the articles linked before asking further questions.