Unable to start hive using tez execution engine - hadoop

Im using Hadoop 2.7.3 version and hive 1.2.1 version.
I face problem with hive using tez engine. Is there any setup error or other kind of error ??
Logging initialized using configuration in jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hduser/tez/tez/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type
[org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.RuntimeException:org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown.
Application application_1568628322588_0002 failed 2 times due to AM Container for appattempt_1568628322588_0002_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://rohan-VirtualBox:8088/cluster/app/application_1568628322588_0002Then, click on links to logs of each attempt.
Diagnostics: Exception from
container-launch.
Container id: container_1568628322588_0002_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: at
org.apache.hadoop.util.Shell.runCommand(Shell.java:582) at
org.apache.hadoop.util.Shell.run(Shell.java:479) at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

This is likely similar to Application failed 2 times due to AM Container: exited with exitCode: 1.
Essentially the code you pasted does not contain the actual error code - so we cannot help much with it. You need to find the exact error message, by going to your Resource Manager and looking at the logs there.

Related

Not able to run(schedule) oozie example map reduce job || java.net.ConnectException

I am using Hadoop 2.6.0 and oozie 5.2.0 version.
Trying to run example Map reduce job in oozie but getting below error.
hadoop1#ip-172-31-84-37:/usr/local/oozie-5.2.0/examples/target/examples/apps/map-reduce$ oozie job -oozie http://localhost:11000/oozie -config job.properties -run
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/oozie/oozie-5.2.0/embedded-oozie-server/webapp/WEB-INF/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/oozie/oozie-5.2.0/embedded-oozie-server/webapp/WEB-INF/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/oozie/oozie-5.2.0/libext/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: E0307 : E0307: Runtime error [Could not check whether file [hdfs://localhost:8020/user/hadoop1/examples/apps/map-reduce/workflow.xml] exists on HDFS. Error message: Call From ip-172-31-84-37/172.31.84.37 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused]
error screenshot
Not sure if I have to change any configurations. I tried giving public dns in job.properties file instead of localhost but still got the same error. Any help is appreciated.
Call From ip-172-31-84-37/172.31.84.37 to localhost:8020 seems to say that your Oozie server (on localhost) is not running HDFS on port 8020. Use the wiki page in the error to debug your network setup...
Assuming you have an external HDFS address, then fs.defaultFS in $HADOOP_CONF_DIR/core-site.xml needs updated with the correct address.

Unable to read HiveServer2 configs from ZooKeeper

I use HDP3.1. And I Ambari to deploy hadoop cluster and hive. After deployed, I can run hive in shell successfully. And then I deploy Apache Kylin2.6, it can sync hive table. But when I build the cube, I got the following error:
java.io.IOException: OS command error exit with return code: 1, error message: SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://datacenter1:2181,datacenter2:2181,datacenter3:2181/default;password=hdfs;serviceDiscoveryMode=zooKeeper;user=hdfs;zooKeeperNamespace=hiveserver2
19/02/15 10:04:53 [main]: INFO jdbc.HiveConnection: Connected to datacenter3:10000
19/02/15 10:04:53 [main]: WARN jdbc.HiveConnection: Failed to connect to datacenter3:10000
19/02/15 10:04:53 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify dfs.replication at runtime. It is not in list of params that are allowed to be modified at runtime (state=08S01,code=0)
Cannot run commands specified using -e. No current connection
The command is:
hive -e "USE default;
I run hive command in shell. It's success. The connection string is same as the string when run build cube in kylin. I'm confused why it is success in shell but failed in building cube.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://datacenter1:2181,datacenter2:2181,datacenter3:2181/default;password=hdfs;serviceDiscoveryMode=zooKeeper;user=hdfs;zooKeeperNamespace=hiveserver2
19/02/15 12:10:19 [main]: INFO jdbc.HiveConnection: Connected to datacenter3:10000
Connected to: Apache Hive (version 3.1.0.3.1.0.0-78)
Driver: Hive JDBC (version 3.1.0.3.1.0.0-78)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.1.0.3.1.0.0-78 by Apache Hive
0: jdbc:hive2://datacenter1:2181,datacenter2:>
You can try to add these two properties to hive-site.xml.
<property>
<name>hive.security.authorization.sqlstd.confwhitelist</name>
<value>mapred.*|hive.*|mapreduce.*|spark.*</value>
</property>
<property>
<name>hive.security.authorization.sqlstd.confwhitelist.append</name>
<value>mapred.*|hive.*|mapreduce.*|spark.*</value>
</property>
Finally, I found the root cause. There is 'Cannot modify dfs.replication at runtime.' error message in the error log. Kylin set this property in $KYLIN_HOME/conf/kylin_hive_conf.xml. And when it is running hive command, it will auto append the properties in that file. The final command likes: hive --hiveconf dfs.replication=2 ..........
It looks like that dfs.replication property can't be appened to hive command. I removed this property in kylin_hive_conf.xml. And it works now.

Kettle - pan.sh "No repository provided, can't load transformation"

I've created a kettle transformation and i've tested on my pc and it works. However, i've insered it in the server and starting as bash script by pan.sh. It was working but after few times it started to give this problem.
server$ bash pan.sh file="API_Mining_LatestVersion.ktr"
#######################################################################
WARNING: no libwebkitgtk-1.0 detected, some features will be unavailable
Consider installing the package with apt-get or yum.
e.g. 'sudo apt-get install libwebkitgtk-1.0-0'
#######################################################################
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
14:56:00,682 INFO [KarafBoot] Checking to see if org.pentaho.clean.karaf.cache is enabled
14:56:00,803 INFO [KarafInstance]
*******************************************************************************
*** Karaf Instance Number: 2 at /data/Fernando/data-integration_updated/./s ***
*** ystem/karaf/caches/pan/data-1 ***
*** FastBin Provider Port:52902 ***
*** Karaf Port:8803 ***
*** OSGI Service Port:9052 ***
*******************************************************************************
Nov 20, 2018 2:56:01 PM org.apache.karaf.main.Main$KarafLockCallback lockAquired
INFO: Lock acquired. Setting startlevel to 100
*ERROR* [org.osgi.service.cm.ManagedService, id=255, bundle=53/mvn:org.apache.aries.transaction/org.apache.aries.transaction.manager/1.1.1]: Updating configuration org.apache.aries.transaction caused a problem: null
org.osgi.service.cm.ConfigurationException: null : null
at org.apache.aries.transaction.internal.TransactionManagerService.<init>(TransactionManagerService.java:136)
at org.apache.aries.transaction.internal.Activator.updated(Activator.java:63)
at org.apache.felix.cm.impl.helper.ManagedServiceTracker.updateService(ManagedServiceTracker.java:148)
at org.apache.felix.cm.impl.helper.ManagedServiceTracker.provideConfiguration(ManagedServiceTracker.java:81)
at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceUpdate.provide(ConfigurationManager.java:1448)
at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceUpdate.run(ConfigurationManager.java:1404)
at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.objectweb.howl.log.LogConfigurationException: Unable to obtain lock on /data/Fernando/data-integration/system/karaf/caches/pan/data-1/txlog/transaction_1.log
at org.objectweb.howl.log.LogFile.open(LogFile.java:191)
at org.objectweb.howl.log.LogFileManager.open(LogFileManager.java:784)
at org.objectweb.howl.log.Logger.open(Logger.java:304)
at org.objectweb.howl.log.xa.XALogger.open(XALogger.java:893)
at org.apache.aries.transaction.internal.HOWLLog.doStart(HOWLLog.java:233)
at org.apache.aries.transaction.internal.TransactionManagerService.<init>(TransactionManagerService.java:133)
... 7 more
2018-11-20 14:56:04.508:INFO:oejs.Server:jetty-8.1.15.v20140411
2018-11-20 14:56:04.544:INFO:oejs.AbstractConnector:Started NIOSocketConnectorWrapper#0.0.0.0:9052
[...]
INFO: New Caching Service registered
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/Fernando/data-integration_updated/launcher/../lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/Fernando/data-integration_updated/plugins/pentaho-big-data-plugin/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2018/11/20 14:56:09 - Pan - Start of run.
ERROR: No repository provided, can't load transformation.
I don't understand where the problem is. The transformation file hasn't been changed and it contains also repo, user and pass paramethers.

Flink job on yarn Not starting

I have written a simple flink job of word count. I am trying to run the job on yarn and getting the error :
2017-10-04 13:15:19,037 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Diagnostics for container ResourceID{resourceId='container_e27_1506324726020_9534_01_000002'} in state COMPLETE : exitStatus=1 diagnostics=Exception from container-launch.
Container id: container_e27_1506324726020_9534_01_000002
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:944)
at org.apache.hadoop.util.Shell.run(Shell.java:848)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1142)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Container exited with a non-zero exit code
When i remove "logback.xml" from The configuration directory of flink /hdfs/flink-1.2.1/conf than the same is working fine.
Please help me to understand what is the issue with logback.xml . Not able to understand the cause of problem.

Getting Stack trace: ExitCodeException exitCode=255 during MapReduce

I am running some map reduce task on huge dataset on four node cluster but getting exception with exit code - 255
16/08/04 08:07:19 INFO mapreduce.Job: map 0% reduce 0%
16/08/04 08:07:27 INFO mapreduce.Job: Task Id : attempt_1470297644642_0001_m_000000_0, Status : FAILED
Exception from container-launch.
Container id: container_1470297644642_0001_01_000003
Exit code: 255
Stack trace: ExitCodeException exitCode=255:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:543)
at org.apache.hadoop.util.Shell.run(Shell.java:460)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:720)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:210)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 255
In log file - SLF4J:
Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J:
Defaulting to no-operation (NOP) logger implementation SLF4J: See
http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details. ��h��׶9�A#���P VERSIONAPPLICATION_ACL MODIFY_APPVIEW_APP
APPLICATION_OWNEcentos(&container_1470297644642_0007_01_000007��stderr0stdout0syslog4142016-08-04
10:39:56,439 FATAL [main]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[main,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.JVMId.(Lorg/apache/hadoop/mapred/JobID;ZJ)V
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:86)
2016-08-04 10:39:56,444 INFO [main] org.apache.hadoop.util.ExitUtil:
Exiting with status -1
(&container_1470297644642_0007_01_000013��stderr0stdout0syslog4142016-08-04
10:40:03,359 FATAL [main]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[main,5,main] threw an Error. Shutting down now...

Resources