Im using Hadoop 2.7.3 version and hive 1.2.1 version.
I face problem with hive using tez engine. Is there any setup error or other kind of error ??
Logging initialized using configuration in jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hduser/tez/tez/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type
[org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.RuntimeException:org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown.
Application application_1568628322588_0002 failed 2 times due to AM Container for appattempt_1568628322588_0002_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://rohan-VirtualBox:8088/cluster/app/application_1568628322588_0002Then, click on links to logs of each attempt.
Diagnostics: Exception from
container-launch.
Container id: container_1568628322588_0002_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: at
org.apache.hadoop.util.Shell.runCommand(Shell.java:582) at
org.apache.hadoop.util.Shell.run(Shell.java:479) at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
This is likely similar to Application failed 2 times due to AM Container: exited with exitCode: 1.
Essentially the code you pasted does not contain the actual error code - so we cannot help much with it. You need to find the exact error message, by going to your Resource Manager and looking at the logs there.
My spark version is 1.6.2 and runs on yarn.
The log of drive container reports SUCCEEDED as follows:
16/11/17 17:25:56 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
16/11/17 17:25:56 INFO ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
However, The log of one executor container records an ERROR:
16/11/17 17:25:56 WARN CoarseGrainedExecutorBackend: An unknown (xxx-xxx-xxx-xxx:xxxx) driver disconnected.
16/11/17 17:25:56 ERROR CoarseGrainedExecutorBackend: Driver xxx-xxx-xxx-xxx:xxxx disassociated! Shutting down.
I think the job is succeeded because the output results are as expected.
But I want to know why the error was thrown and whether the job was really succeeded.
I find more info from the log of yarn NodeManager:
WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_xxxxxxxxx and exit code: 1
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Someone encounter the same question?
Thanks.
On executing jar in hadoop, I get the following error:
16/11/04 18:32:59 INFO mapreduce.Job: Task Id : attempt_1478261728730_0005_m_000000_2, Status : FAILED
Exception from container-launch.
Container id: container_1478261728730_0005_01_000004
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/11/04 18:33:09 INFO mapreduce.Job: map 100% reduce 0%
This is application log:
Native code library failed to load.
java.lang.UnsatisfiedLinkError: no opencv_java2411 in java.library.pathopencv_java2411
I don't know what it mean, can anybody help with this please?
You are missing opencv on your cluster nodes.
See here for all the details on how to handle this.
Long story short though, you need to install opencv on your executors. You cannot really compile it into your job's .jar in a portable way since it's C and not Java code.
Update:
Note that the environment on your Hadoop executors is set by your hadoop-env.sh. So it needs to contain a line like:
JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/etc/opencv/lib
I'm new to Hadoop. I am trying to check if my Yarn/Map Reduce is working correctly.(http://hadooptutorial.info/run-example-mapreduce-program/) I get this error message
http://localhost:8088/cluster/app/application_1468349436383_0001
Application application_1468349436383_0001 failed 2 times due to AM Container for appattempt_1468349436383_0001_000002 exited with exitCode: 127
For more detailed output, check application tracking page:http://localhost:8088/cluster/app/application_1468349436383_0001Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1468349436383_0001_02_000001
Exit code: 127
Stack trace: ExitCodeException exitCode=127:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 127
Failing this attempt. Failing the application.
I'm guessing my configurations aren't correct because MapReduce dosen't even start up.
I'm really sorry if this is a trivial question. I've very new to hadoop and I couldn't find anything online to fix this.
Here are my hadoop configurations
yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
Core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
hdfs.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>8</value>
</property>
</configuration>
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
Thank You so much
nodemanager log error snapshot
2016-07-13 10:37:42,145 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1468349436383_0001_01_000001 is : 127
2016-07-13 10:37:42,147 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1468349436383_0001_01_000001 and exit code: 127
ExitCodeException exitCode=127:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-07-13 10:37:42,149 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2016-07-13 10:37:42,149 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1468349436383_0001_01_000001
2016-07-13 10:37:42,149 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 127
2016-07-13 10:37:42,150 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=127:
2016-07-13 10:37:42,150 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
2016-07-13 10:37:42,150 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:456)
2016-07-13 10:37:42,150 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
2016-07-13 10:37:42,150 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
2016-07-13 10:37:42,150 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
2016-07-13 10:37:42,150 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
2016-07-13 10:37:42,150 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2016-07-13 10:37:42,150 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2016-07-13 10:37:42,150 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2016-07-13 10:37:42,150 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745)
2016-07-13 10:37:42,150 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 127
2016-07-13 10:37:42,151 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1468349436383_0001_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE
2016-07-13 10:37:42,151 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1468349436383_0001_01_000001
2016-07-13 10:37:42,183 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-aditghosh/nm-local-dir/usercache/aditghosh/appcache/application_1468349436383_0001/container_1468349436383_0001_01_000001
2016-07-13 10:37:42,184 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=aditghosh OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1468349436383_0001 CONTAINERID=container_1468349436383_0001_01_000001
2016-07-13 10:37:42,188 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1468349436383_0001_01_000001 transitioned from EXITED_WITH_FAILURE to DONE
2016-07-13 10:37:42,188 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1468349436383_0001_01_000001 from application application_1468349436383_0001
2016-07-13 10:37:42,188 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: ResourceCalculatorPlugin is unavailable on this system. org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is disabled.
2016-07-13 10:37:42,188 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1468349436383_0001
2016-07-13 10:37:43,970 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1468349436383_0001_01_000001]
2016-07-13 10:37:43,981 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1468349436383_0001_000002 (auth:SIMPLE)
2016-07-13 10:37:43,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1468349436383_0001_02_000001 by user aditghosh
2016-07-13 10:37:43,987 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=aditghosh IP=127.0.0.1 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1468349436383_0001 CONTAINERID=container_1468349436383_0001_02_000001
2016-07-13 10:37:43,987 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1468349436383_0001_02_000001 to application application_1468349436383_0001
2016-07-13 10:37:43,988 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1468349436383_0001_02_000001 transitioned from NEW to LOCALIZING
2016-07-13 10:37:43,988 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1468349436383_0001
2016-07-13 10:37:43,988 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1468349436383_0001_02_000001 transitioned from LOCALIZING to LOCALIZED
2016-07-13 10:37:44,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1468349436383_0001_02_000001 transitioned from LOCALIZED to RUNNING
2016-07-13 10:37:44,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: ResourceCalculatorPlugin is unavailable on this system. org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is disabled.
2016-07-13 10:37:44,100 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-aditghosh/nm-local-dir/usercache/aditghosh/appcache/application_1468349436383_0001/container_1468349436383_0001_02_000001/default_container_executor.sh]
2016-07-13 10:37:44,132 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1468349436383_0001_02_000001 is : 127
2016-07-13 10:37:44,132 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1468349436383_0001_02_000001 and exit code: 127
ExitCodeException exitCode=127:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-07-13 10:37:44,132 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2016-07-13 10:37:44,132 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1468349436383_0001_02_000001
2016-07-13 10:37:44,132 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 127
2016-07-13 10:37:44,132 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=127:
2016-07-13 10:37:44,132 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
2016-07-13 10:37:44,132 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:456)
2016-07-13 10:37:44,133 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
2016-07-13 10:37:44,133 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
2016-07-13 10:37:44,133 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
2016-07-13 10:37:44,133 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
2016-07-13 10:37:44,133 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2016-07-13 10:37:44,133 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2016-07-13 10:37:44,133 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2016-07-13 10:37:44,133 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745)
2016-07-13 10:37:44,133 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 127
2016-07-13 10:37:44,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1468349436383_0001_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURE
2016-07-13 10:37:44,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1468349436383_0001_02_000001
2016-07-13 10:37:44,160 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-aditghosh/nm-local-dir/usercache/aditghosh/appcache/application_1468349436383_0001/container_1468349436383_0001_02_000001
2016-07-13 10:37:44,161 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=aditghosh OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1468349436383_0001 CONTAINERID=container_1468349436383_0001_02_000001
2016-07-13 10:37:44,161 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1468349436383_0001_02_000001 transitioned from EXITED_WITH_FAILURE to DONE
2016-07-13 10:37:44,161 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1468349436383_0001_02_000001 from application application_1468349436383_0001
2016-07-13 10:37:44,161 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: ResourceCalculatorPlugin is unavailable on this system. org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is disabled.
2016-07-13 10:37:44,161 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1468349436383_0001
2016-07-13 10:37:45,983 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1468349436383_0001_02_000001]
2016-07-13 10:37:45,987 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1468349436383_0001 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2016-07-13 10:37:45,988 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-aditghosh/nm-local-dir/usercache/aditghosh/appcache/application_1468349436383_0001
2016-07-13 10:37:45,988 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1468349436383_0001
2016-07-13 10:37:45,992 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1468349436383_0001 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2016-07-13 10:37:45,992 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1468349436383_0001, with delay of 10800 seconds
2016-07-13 13:37:46,480 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /usr/local/hadoop/logs/userlogs/application_1468349436383_0001
2016-07-14 08:57:21,802 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Caught exception in status-updater
java.io.IOException: Failed on local exception: java.io.IOException: Operation timed out; Host Details : local host is: "Adits-MacBook-Pro.local/127.0.0.1"; destination host is: "0.0.0.0":8031;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy72.nodeHeartbeat(Unknown Source)
at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.nodeHeartbeat(ResourceTrackerPBClientImpl.java:80)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy73.nodeHeartbeat(Unknown Source)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:596)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Operation timed out
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:520)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1084)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:979)
Resource Manager log error snapshot
2016-07-13 10:37:44,978 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1468349436383_0001_000002
2016-07-13 10:37:44,978 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1468349436383_0001_000002
2016-07-13 10:37:44,978 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1468349436383_0001_000002 State change from FINAL_SAVING to FAILED
2016-07-13 10:37:44,978 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of failed attempts is 2. The max attempts is 2
2016-07-13 10:37:44,979 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1468349436383_0001 with final state: FAILED
2016-07-13 10:37:44,987 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1468349436383_0001 State change from ACCEPTED to FINAL_SAVING
2016-07-13 10:37:44,987 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating info for app: application_1468349436383_0001
2016-07-13 10:37:44,987 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Attempt appattempt_1468349436383_0001_000002 is done. finalState=FAILED
2016-07-13 10:37:44,987 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1468349436383_0001 requests cleared
2016-07-13 10:37:44,988 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1468349436383_0001 failed 2 times due to AM Container for appattempt_1468349436383_0001_000002 exited with exitCode: 127
For more detailed output, check application tracking page:http://localhost:8088/cluster/app/application_1468349436383_0001Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1468349436383_0001_02_000001
Exit code: 127
Stack trace: ExitCodeException exitCode=127:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 127
Failing this attempt. Failing the application.
2016-07-13 10:37:44,988 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application removed - appId: application_1468349436383_0001 user: aditghosh queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2016-07-13 10:37:44,990 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1468349436383_0001 State change from FINAL_SAVING to FAILED
2016-07-13 10:37:44,991 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application removed - appId: application_1468349436383_0001 user: aditghosh leaf-queue of parent: root #applications: 0
2016-07-13 10:37:44,992 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=aditghosh OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1468349436383_0001 failed 2 times due to AM Container for appattempt_1468349436383_0001_000002 exited with exitCode: 127
For more detailed output, check application tracking page:http://localhost:8088/cluster/app/application_1468349436383_0001Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1468349436383_0001_02_000001
Exit code: 127
Stack trace: ExitCodeException exitCode=127:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 127
Failing this attempt. Failing the application. APPID=application_1468349436383_0001
2016-07-13 10:37:44,994 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1468349436383_0001,name=QuasiMonteCarlo,user=aditghosh,queue=default,state=FAILED,trackingUrl=http://localhost:8088/cluster/app/application_1468349436383_0001,appMasterHost=N/A,startTime=1468431458818,finishTime=1468431464979,finalStatus=FAILED,memorySeconds=8214,vcoreSeconds=4,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources=<memory:0\, vCores:0>,applicationType=MAPREDUCE
2016-07-13 10:47:24,771 INFO org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is accessing unchecked http://localhost:8088/cluster/app/application_1468349436383_0001 which is the app master GUI of application_1468349436383_0001 owned by aditghosh
2016-07-13 11:50:38,782 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2016-07-13 11:50:38,782 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2016-07-13 11:50:38,792 INFO org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager: storing master key with keyID 3
2016-07-13 11:50:38,793 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing RMDTMasterKey.
2016-07-14 08:56:27,045 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8031: readAndProcess from client 10.0.0.19 threw exception [java.io.IOException: Operation timed out]
java.io.IOException: Operation timed out
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.hadoop.ipc.Server.channelRead(Server.java:2603)
at org.apache.hadoop.ipc.Server.access$2800(Server.java:136)
at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1481)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:771)
at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:637)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:608)
2016-07-14 08:57:22,834 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Received duplicate heartbeat from node localhost:53667 responseId=65089
2016-07-14 11:50:41,081 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2016-07-14 11:50:41,081 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2016-07-14 11:50:41,094 INFO org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager: storing master key with keyID 4
2016-07-14 11:50:41,096 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing RMDTMasterKey.
Hi I have installed Cloudera Manager.While turning my benchmarking tests map-reduce task is failing saying following error message :
16/04/01 12:42:40 INFO mapreduce.Job: Job job_1459494626924_0001 running in uber mode : false
16/04/01 12:42:40 INFO mapreduce.Job: map 0% reduce 0%
16/04/01 12:42:54 INFO mapreduce.Job: map 16% reduce 0%
16/04/01 12:42:55 INFO mapreduce.Job: map 29% reduce 0%
16/04/01 12:42:56 INFO mapreduce.Job: map 75% reduce 0%
16/04/01 12:42:57 INFO mapreduce.Job: map 83% reduce 0%
16/04/01 12:42:59 INFO mapreduce.Job: map 100% reduce 0%
16/04/01 12:43:01 INFO mapreduce.Job: Task Id : attempt_1459494626924_0001_r_000000_0, Status : FAILED
Exception from container-launch.
Container id: container_1459494626924_0001_01_000010
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
at org.apache.hadoop.util.Shell.run(Shell.java:478)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:210)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/04/01 12:43:01 INFO mapreduce.Job: Task Id : attempt_1459494626924_0001_r_000005_0, Status : FAILED
Exception from container-launch.
Container id: container_1459494626924_0001_01_000015
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
at org.apache.hadoop.util.Shell.run(Shell.java:478)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:210)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/04/01 12:43:01 INFO mapreduce.Job: Task Id : attempt_1459494626924_0001_r_000001_0, Status : FAILED
Exception from container-launch.
Container id: container_1459494626924_0001_01_000011
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
at org.apache.hadoop.util.Shell.run(Shell.java:478)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:210)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/04/01 12:43:01 INFO mapreduce.Job: Task Id : attempt_1459494626924_0001_r_000006_0, Status : FAILED
Exception from container-launch.
Container id: container_1459494626924_0001_01_000016
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
at org.apache.hadoop.util.Shell.run(Shell.java:478)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:210)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
16/04/01 12:43:01 INFO mapreduce.Job: Task Id : attempt_1459494626924_0001_r_000004_0, Status : FAILED
Exception from container-launch.
Container id: container_1459494626924_0001_01_000014
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
at org.apache.hadoop.util.Shell.run(Shell.java:478)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:210)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
what could be the issue that need to be check.I have done YARN tuning using excel sheet provided by cloudera.I also tried to adjust Namenode's Vcores and memory.
In log file I can see following error message
2016-03-30 22:04:27,404 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=yarn IP=10.195.48.127 OPERATION=refreshNodes TARGET=AdminService RESULT=SUCCESS
2016-03-30 17:38:24,133 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: RECEIVED SIGNAL 15: SIGTERM
2016-03-30 17:38:24,148 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2016-03-30 17:38:24,151 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup#slave5:8088
2016-03-30 17:38:24,151 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2016-03-30 17:38:24,152 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2016-03-30 17:38:24,254 INFO org.apache.hadoop.ipc.Server: Stopping server on 8032
2016-03-30 17:38:24,257 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8032
2016-03-30 17:38:24,257 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2016-03-30 17:38:24,257 INFO org.apache.hadoop.ipc.Server: Stopping server on 8033
2016-03-30 17:38:24,260 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8033
2016-03-30 17:38:24,260 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2016-03-30 17:38:24,263 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type STATUS_UPDATE for node slave2:8041
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException
at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:247)
at org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl$StatusUpdateWhenHealthyTransition.transition(RMNodeImpl.java:778)
at org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl$StatusUpdateWhenHealthyTransition.transition(RMNodeImpl.java:736)
at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl.handle(RMNodeImpl.java:418)
at org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl.handle(RMNodeImpl.java:79)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$NodeEventDispatcher.handle(ResourceManager.java:866)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$NodeEventDispatcher.handle(ResourceManager.java:850)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:174)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:242)
... 13 more
2016-03-30 17:38:24,283 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning to standby state
2016-03-30 17:38:24,283 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ResourceManager metrics system...
2016-03-30 17:38:24,284 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics system stopped.
2016-03-30 17:38:24,285 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics system shutdown complete.
2016-03-30 17:38:24,285 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
Could anybody suggest what is the problem ??