How to establish connection with HIVE using JDBC? - hadoop

HOW TO CONFIGURE JDBC WITH HIVE
import java.sql.SQLException;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;
import java.sql.DriverManager;
public class table {
private static String driverName = "org.apache.hadoop.hive.mysql.jdbc.Driver";
public static void main(String[] args) throws SQLException {
// Register driver and create driver instance
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Connection con = DriverManager.getConnection("jdbc:mysql://localhost:1000/default", "", "");
Statement stmt = con.createStatement();
stmt.executeQuery("CREATE DATABASE userdb");
// System.out.println(“Database userdb created successfully”);
con.close();
}
}
akshay#akshay:~$ javac table.java
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
akshay#akshay:~$ java table
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
java.lang.ClassNotFoundException: org.apache.hadoop.hive.mysql.jdbc.Driver
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at table.main(table.java:14)
Exception in thread "main" java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:1000/default
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at table.main(table.java:20)
My hive-site.xml contains
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>MySQL JDBC driver class</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/metastore?createDatabaseIfNotExist=true</value>
<description>metadata is stored in a MySQL server </description>
</property>
I have configured meta store of hive with MySQL. So what should my ConnectionURL and Drivername in JAVA connection code?
I am not getting where I am going wrong. Please provide solution for above problem.

For running hive queries using JDBC API's, you need to start your hiveserver2 first. Configure the thrift server port in your hive-site.xml file as shown below
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
<description>TCP port number to listen on, default 10000</description>
</property>
Start the hiveserver2 using the command
cd $HIVE_HOME/bin
./hiveserver2
Also you need to add the below dependencies to your project.
Hive-jdbc-*-standalone.jar
hive-jdb-*.jar
hive-metastore-*.jar
hive-service-*.jar
After that try running the program. You can refer to this blog for more information on step by step procedure to run hive queries using java programs.

Related

Not able to connect to Aster database through jdbc driver using java

I am trying to connect to Aster server with jdbc drivers from java. I have added already the Jar files to the classpath.
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.*;
public class TeradataJDBCConnection {
public static void main(String[] args) throws Exception {
Class.forName("com.asterdata.ncluster.Driver");
String url="jdbc:ncluster://<ip_address>:2406/test";
Connection conn=DriverManager.getConnection(url, "user123", "test");
}
}
But I am getting the below error.
Exception in thread "main" java.sql.SQLException: [AsterData][ASTERJDBCDSII](34) : Failed to connect to 10.99.186.92. Please check the host address. ()
at com.asterdata.ncluster.jdbc.core.NClusterConnection.connect(Unknown Source)
at com.simba.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
at com.simba.jdbc.common.AbstractDriver.connect(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at Tera.TeradataJDBCConnection.main(TeradataJDBCConnection.java:17)
Caused by: com.asterdata.ncluster.jdbc.core.MuleException: [AsterData][ASTERJDBCDSII](34) : Failed to connect to 10.99.186.92. Please check the host address. ()
... 6 more
No bug in the code.
Everything is running fine.
Download the jdbc driver from here.
https://aster-community.teradata.com/docs/DOC-2254
You could also download the driver from here: http://downloads.teradata.com/download/aster/aster-client-tools-for-windows
You can use the jar noarch-aster-jdbc-driver from AsterJDBC__indep_indep.06.10.00.02.zip file. This jar file works for my connection.

Spark not reading hive-site.xml?

I am trying to access hive metastore and I am using SparkSql for this . I have setup sparksession , but when I run my program and see log I see this exception
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:188)
... 61 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
... 62 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
... 68 more
Caused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details.
I am running a servlet which accesses following code
public class HiveReadone extends HttpServlet {
private static final long serialVersionUID = 1L;
/**
* #see HttpServlet#HttpServlet()
*/
public HiveReadone() {
super();
// TODO Auto-generated constructor stub
}
/**
* #see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse response)
*/
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
// TODO Auto-generated method stub
response.getWriter().append("Served at: ").append(request.getContextPath());
SparkSession spark = SparkSession
.builder()
.appName("Java Spark SQL basic example")
.enableHiveSupport()
.config("spark.sql.warehouse.dir", "hdfs://saurab:9000/user/hive/warehouse")
.config("mapred.input.dir.recursive", true)
.config("hive.mapred.supports.subdirectories", true)
.config("hive.vectorized.execution.enabled", true)
.master("local")
.getOrCreate();
response.getWriter().println(spark);
Nothing gets print on browser accept output from response.getWriter().append("Served at:
").append(request.getContextPath()); which is Served at: /hiveServ
Please take a look at my conf/hive-site.xml
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://saurab:3306/metastore_db?createDatabaseIfNotExist=true</value>
<description>metadata is stored in a MySQL server</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>MySQL JDBC driver class</description>
</property>
<property>
<name>hive.aux.jars.path</name>
<value>/home/saurab/hadoopec/hive/lib/hive-serde-2.1.1.jar</value>
</property>
<property>
<name>spark.sql.warehouse.dir</name>
<value>hdfs://saurab:9000/user/hive/warehouse</value>
</property>
<property>
<name>hive.metastore.uris</name>
<!--Make sure that <value> points to the Hive Metastore URI in your cluster -->
<value>thrift://saurab:9083</value>
<description>URI for client to contact metastore server</description>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10001</value>
<description>Port number of HiveServer2 Thrift interface.
Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT
</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hiveuser</value>
<description>user name for connecting to mysql server</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hivepassword</value>
<description>password for connecting to mysql server</description>
</property>
As far as I have read if we configure hive.metastore.uris spark will connect to hive metastore, but in my case it is not and giving me above error.
To configure spark on hive try to copy your hive-site.xml to the spark/conf directory

JDBC simple program

This is my simple JDBC program but I am not able to establish connection. The class path is set up with ojdbc6.jar.
package Demo;
import java.sql.*;
public class JdbcDemo {
public static void main(String[] args)throws Exception {
// TODO Auto-generated method stub
System.out.println("hi");
Class.forName("oracle.jdbc.driver.OracleDriver");
System.out.println("hi1");
Connection con=DriverManager.getConnection("jdbc:oracle:thin:#localhost:1521:system","scott","tiger");
}
}
This is the output and the exception I am getting:
Exception in thread "main" java.sql.SQLRecoverableException: Io exception: The Network Adapter could not establish the connection
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:101)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:173)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:229)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:458)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:411)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:490)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:202)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:465)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at Demo.JdbcDemo.main(JdbcDemo.java:13)
Please check the following
Have you started Oracle TNS Listener, if not start by lsnrctl utility.
Have you put the correct port
Is your hostname correct in the database server?
The issue could be any one of the above.

Oracle ArrayIndexOutOfBoundsException: -1

For special needs, I use the unzipped jar format ojdbc6.jar.
The code source of my connection to the database oracle is :
import java.sql.DriverManager;
import java.sql.SQLException;
....
try
{
String driver = "oracle.jdbc.driver.OracleDriver";
String url = "jdbc:oracle:thin:#x.x.x.x:1521:dev";
String login = "dev";
String password = "dev";
Class.forName(driver);
DriverManager.getConnection(url, login, password); // Error is generated by this line
}
catch (SQLException e)
{
System.err.println("error");
}
.....
When i launch the program, i have this exception :
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at oracle.jdbc.driver.T4CTTIoauthenticate.setSessionFields(T4CTTIoauthenticate.java:999)
at oracle.jdbc.driver.T4CTTIoauthenticate.<init>(T4CTTIoauthenticate.java:235)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:370)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
.......
I use
ojdbc6.jar (11.2.0.2)
JRE 1.6.0_24
Oracle 11g
The database is accessible and the data connection are correct.
I was looking into the same error above for a RMI server issue. I added -Djava.security.policy=$RMIPATH"/policy.txt" to my runtime script and created a policy.txt file that stated.
grant {
permission java.security.AllPermission;
};
I hate disabling security but it works... My runtime is 1.8.31

Not able to load OracleDriver class in simple java class

// DbConnector.java
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.SQLException;
public class DbConnector {
public static void main(String[] argv) {
System.out.println("-------- Oracle JDBC Connection Testing ------");
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
} catch (ClassNotFoundException e) {
System.out.println("Where is your Oracle JDBC Driver?");
e.printStackTrace();
return;
}
System.out.println("Oracle JDBC Driver Registered!");
Connection connection = null;
try {
connection = DriverManager.getConnection(
"jdbc:oracle:thin:#172.17.119.108:1526:GESSDEV", "username",
"password");
} catch (SQLException e) {
System.out.println("Connection Failed! Check output console");
e.printStackTrace();
return;
}
if (connection != null) {
System.out.println("You made it, take control your database now!");
} else {
System.out.println("Failed to make connection!");
}
}
}
I have a jdbc jar with name odbc14.jar
javac -classpath odbc14.jar; DbConnector.java
compilation successfull
java -classpath odbc14.jar; DbConnector
OUTPUT ::
-------- Oracle JDBC Connection Testing ------
Where is your Oracle JDBC Driver?
java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:164)
at DbConnector.main(DbConnector.java:13)
Please help me out to resolve this
I have sorted out the above problem. Now I'm gettin new exception
-------- Oracle JDBC Connection Testing ------
Oracle JDBC Driver Registered!
Connection Failed! Check output console
java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:420)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at java.sql.DriverManager.getConnection(DriverManager.java:525)
The JDBC driver for use with ojdbc14.jar is oracle.jdbc.OracleDriver.
See: Loading JDBC Driver Class - ojdbc14.jar
Put your ojdbc14.jar in <jdk_home>\jre\lib\ext and ADD in CLASSPATH variable then you do not have to specify the classpath while compiling or running
If you are using java6 then oracle.jdbc.driver.OracleDriver is okay but it is deprecated. So it's better to use oracle.jdbc.OracleDriver
Just put the odbc14.jar this JAR in required JRE path like "C:\Program Files\Java\jdk1.7.0_65\jre\lib\ext".
This should solve your issue.
you should
try {
Class.forName("oracle.jdbc.OracleDriver");
}
and place the odbc14.jar in class path (if already not done)
Only you need to put into appsclassloader that is your classpath. I am not sure, why we need to put into jre/lib/ext

Resources