Apache Oozie throws classnotfound exception while creating mysql DB - hadoop

I am trying to use mysql DB with apache OOzie.
my $OOZIE_HOME is
-bash: /opt/oozie_install/oozie-3.3.0-cdh4.2.2: Is a directory
But I copied mysql-connector-java-5.1.29-bin.jar in almost every possible places.
Like I copied it inside
/opt/oozie_install/oozie-3.3.0-cdh4.2.2
/opt/oozie_install/oozie-3.3.0-cdh4.2.2/libs
/opt/oozie_install/oozie-3.3.0-cdh4.2.2/libtools
/usr/lib/jvm/jdk/libs
/user/home/hadoop/
But I am still getting ClassnotFoundException.
java.lang.Exception: Could not connect to the database: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:473)
at org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:179)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:118)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:64)
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:462)
at org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:469)
Exactly where am I supposed to copy the Mysql connector.
I have verified my oozie-site.xml-
I followed the following steps to use mysql in Oozie
My oozie directory looks like-

You have to copy mysql-connector-java-5.1.29-bin.jar to /opt/oozie_install/oozie-3.3.0-cdh4.2.2/libext directory then restart oozie instance. Make sure that mysql user oozie has suffient privileges to the database oozie, if not, grant suffient permission using grant command in mysql server.

i meet the same problem finally i solve it by edit the oozie-env.sh and append the JAVA_HOME at last export JAVA_HOME=/usr/local/jdk1.7 the java_home is yourself javapath

I ran into this issue when I was converting my local Derby instance to MySql. The difference between my issue and the others, is that I did not install an RPM. My Oozie instance was pre-compiled in a tar.gz file. I had to copy the mysql-connector-java-bin.jar to the oozie-server/lib directory. This was in addition to copying it to the lib, libext, and libtools directories. I am not sure if all of those are needed, but I do know that oozie-server/lib is needed for Oozie to start. Hope this helps someone!

Related

How can Apache Spark history-server refer to Amazon S3?

[version]
Apache Spark 2.2.0
Hadoop 2.7
I want to set up Apache Spark histroy server.
Spark events log located in Amazon S3.
I can save log file in S3, but cannot read from history server.
Apache Spark installed at /usr/local/spark
so, $SPARK_HOME is /usr/local/spark
$ cd /usr/local/spark/sbin
$ sh start-history-server.sh
I got following error
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.hadoop.fs.s3a.S3AFileSystem
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:230)
....
my spark-defaults.conf is below
spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
spark.history.provider org.apache.hadoop.fs.s3a.S3AFileSystem
spark.history.fs.logDirectory s3a://xxxxxxxxxxxxx
spark.eventLog.enabled true
spark.eventLog.dir s3a://xxxxxxxxxxxxxxx
I installed this 2 jar files in /usr/local/spark/jars/
aws-java-sdk-1.7.4.jar
hadoop-aws-2.7.3.jar
but error is same.
What is wrong?
Please add the following in spark-defaults.conf file and retry again.
spark.driver.extraClassPath :/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/* :
spark.executor.extraClassPath :/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:

Geomesa Configuration Erorr

I am using Hadoop 2.7 with geoserver 2.8.0, but while I am trying to configure Geomesa 1.2.0, I am getting this error message:
$ geomesa
Using GEOMESA_HOME = /usr/local/geomesa/dist/tools/geomesa-tools-1.2.0
Warning: you have not set ACCUMULO_HOME and/or HADOOP_HOME as environment variables.
GeoMesa tools will not run without the appropriate Accumulo and Hadoop jars in the tools classpath.
Please ensure that those jars are present in the classpath by running 'geomesa classpath' .
To take corrective action, please place the necessary jar files in the lib directory of geomesa-tools.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/accumulo/core/client/TableNotFoundException
at org.locationtech.geomesa.tools.commands.TableConfCommand.<init>(TableConfCommand.scala:32)
at org.locationtech.geomesa.tools.Runner$.createCommand(Runner.scala:50)
at org.locationtech.geomesa.tools.Runner$.main(Runner.scala:21)
at org.locationtech.geomesa.tools.Runner.main(Runner.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.accumulo.core.client.TableNotFoundException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 4 more
How can I fix this?
The GeoMesa tools need Hadoop and Accumulo jars in order to connect to Accumulo.
One quick option is to run the GeoMesa tools from a tablet server or another machine already configured to be part of the Hadoop cluster. If you are using another machine, you can mirror the $HADOOP_HOME and $ACCUMULO_HOME directories from a cluster node locally.
As another alternative, you can download the install-hadoop-accumulo.sh script in the geomesa-tools/bin directory to download a set of Hadoop and Accumulo jars.
verify that corresponding jar file is present in the classpath, you can check this with the help of command:- Geomesa classpath
If jar is not present then copy the jar in the Geomesa directoryin my case it is in following path:
/*/geomesa-1.2.4/dist/tools/geomesa-tools-1.2.4/lib/common/

HiveServer Class Not Found Exception

I'm trying to run hive from the command prompt it is working absolutely fine. But when I try running hiveserver using "hive --service hiveserver" command, I'm getting the following exception.
Starting Hive Thrift Server
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.hadoop.hive.service.HiveServer
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
So I then tried with the command "hive --service hiveserver2"; still I'm not finding any solution.
Can anybody please suggest a solution for this problem.
May be another process (another hiveserver) already listening on port 10000.
can you check it by :
netstat -ntulp | grep ':10000' to see it and if found then kill the process.
Otherwise start the server on another port.
By the way which version you are using ?
This error occurred to me when it can't find hive-service-*.jar in hadoop classpath. Just copy the hive-service-*.jar to your hadoop lib folder or export classpath in hadoop-env.sh. I have mentioned how to add classpath below.
Add this line in hadoop-env.sh:
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/local/hive/lib/hive-*.jar
I have mentioned the path for hive as /usr/local/hive since i have hive installed at that location. Change it to point to your hive installation.

hcatalog with mapreduce

I get the following error while executing a MapReduce program.
I have placed all jars in hadoop/lib directory and have also mentioned the jars in -libjars.
This is the cmd I am executing:
$HADOOP_HOME/bin/hadoop --config $HADOOP_HOME/conf jar /home/shash/distinct.jar HwordCount -libjars $LIB_JARS WordCount HWordCount2
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hcatalog.mapreduce.HCatOutputFormat at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996) at
org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248) at org.apache.hadoop.mapred.Task.initialize(Task.java:501) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:306) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at
org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.lang.ClassNotFoundException: org.apache.hcatalog.mapreduce.HCatOutputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:423) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at
java.lang.ClassLoader.loadClass(ClassLoader.java:356) at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:264) at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:994) ...
8 more
Make sure LIB_JARS is a comma-separated list (as opposed to colon-separated like CLASSPATH)
Applies To CDH 5.0.x CDH 5.1.x CDH 5.2.x CDH 5.3.x Sqoop
Cause Sqoop cannot pick up the HCatalog libraries because Cloudera
Manager does not set the HIVE_HOME environment. It needs to be set
manually.
This problem is tracked with below JIRA:
https://issues.apache.org/jira/browse/SQOOP-2145
The fix of this JIRA has been included in CDH since version 5.4.0.
Workaround: Applicable to CDH versions lower than 5.4.0.
Execute below commands in shell before calling Sqoop command or adding them to /etc/sqoop/conf/sqoop-env.sh (create one, if it does not already exists):
export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive (for parcel installation)
export HIVE_HOME=/usr/lib/hive (for package installation)

Error starting Hive: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf

I have downloaded latest stable release of Hive, when I start /usr/local/hive/bin/hive it gives me this error:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 3 more
Hadoop DFS is started and working, and I have changed /usr/local/hive/conf/hive-env.sh to export HADOOP_HOME.
Does anyone know what else can I do?
Thanks.
Apart from editing hive-env.sh, you also need to edit your bash_profile.
vim ~/.bash_profile
Add the following lines to your bash_profile
export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin
Save this file and then
source ~/.bash_profile
If this still doesnt work, please include your hive-env.sh file and hive-site.xml file. Also please tell me if you are using derby or mysql as metastore.
Solved moving:
export HADOOP_CLASSPATH=/usr/local/hbase/hbase-0.94.1.jar:/usr/local/hbase/hbase-0.94.1-test.jar:/usr/local/hbase/conf:/usr/local/hbase/lib/zookeeper-3.4.3.jar:/usr/local/hive/lib/*.jar:/usr/local/hbase
From /usr/local/hadoop/conf/hadoop-env.sh to ~/.bashrc
Thanks for the help.
if you are facing this issue in sqoop while hive-imports
then you need to copy hive-common-3.1.2.jar (or whichever version of hive-common-x.x.x.jar) from the $HIVE_HOME/lib folder and paste it into the $SQOOP_HOME/lib folder and the error will be gone

Resources