Hive over Tez via Hive JDBC - Error - hadoop

I am using Hortonworks Hadoop HDP-2.3.2.0-2950
Hive over Tez engine
Below 2 queries are from Java code.
select * from ascii -- works well
select count(*) from ascii or select count(1) from ascii -- fails with error out
My code:
package com.hadoop.hive;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
/**
*
* #author hdpadmin
*
*/
public class HiveReadJDBCMain {
private static String driverName = "org.apache.hive.jdbc.HiveDriver";
/**
* #param args
* #throws SQLException
*/
public static void main(String[] args) throws SQLException {
Connection con = null;
PreparedStatement pst = null;
ResultSet res = null;
try {
// Load Hive Driver Clazz
Class.forName(driverName);
// No username and password required.(running in a local machine)
con = DriverManager.getConnection("jdbc:hive2://localhost:10000/labs");
//Connection con = DriverManager.getConnection("jdbc:hive2://localhost:10000/labs", "<Username>", "<Password>");
String tableName = "ascii";
// select * query
String sql = "select * from " + tableName;
System.out.println("Select Query Running: " + sql);
pst = con.prepareStatement(sql);
res = pst.executeQuery();
while (res.next()) {
System.out.println(String.valueOf(res.getInt(1)) + "\t" + res.getString(2));
}
pst.close();
res.close();
// regular hive query
sql = "select count(*) from " + tableName;
System.out.println("Count Query Running: " + sql);
pst = con.prepareStatement(sql);
res = pst.executeQuery();
while (res.next()) {
System.out.println(res.getString(1));
}
} catch (ClassNotFoundException e) {
e.printStackTrace();
System.exit(1);
} catch (Exception e) {
e.printStackTrace();
System.exit(1);
} finally {
res.close();
pst.close();
con.close();
}
}
}
Error
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:282)
at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:378)
at org.apache.hive.jdbc.HivePreparedStatement.executeQuery(HivePreparedStatement.java:109)
at com.hadoop.hive.HiveReadJDBCMain.main(HiveReadJDBCMain.java:48)

Yes i have fixed by passing the below connection objects with my user name
Thanks.... :)
con = DriverManager.getConnection("jdbc:hive2://localhost:10000/labs", "hdpadmin","");

i also fixed by passing username in connection string . Thanks :)

In case your using the AWS Hive JDBC to talk with a default EMR cluster the following setup in Intellij worked for me
The trick was using user hadoop

Related

JMS-Loadtest using Gatling: unable to initialize ContextFactory of IBM MQ

I'm implementing a load test scenario against a IBM MQ with Gatling.
The setup is basically the same as mentioned here
Problem: I'm not able to initialize the required ContextFactory with IBM MQ - which should be com.ibm.mq.jms.context.WMQInitialContextFactory.
The WMQInitialContextFactory cannot be found.
But my build.sbt is containing the IBM MQ dependency correctly (being successfully retrieved by our internal Nexus :
"com.ibm.mq" % "com.ibm.mq" % "8.0.0.3"
My gatling scenario:
package com.myawesomecompany.loadtest
import com.ibm.mq._
import com.ibm.mq.jms.MQConnectionFactory
import com.ibm.mq.jms.JMSMQ_Messages
import com.ibm.msg.client.wmq.common.CommonConstants
import io.gatling.core.Predef._
import io.gatling.core.scenario.Simulation
import io.gatling.jms.Predef._
import javax.jms._
import scala.concurrent.duration._
class JMSLoadTest extends Simulation {
val jmsConfiguration = jms
.connectionFactoryName("ConnectionFactory")
.url("devmq01.company.dmz")
.credentials("mqm", "mqm")
.contextFactory(classOf[com.ibm.mq.jms.context.WMQInitialContrextFactory].getName)
.listenerCount(1)
.usePersistentDeliveryMode
val scn = scenario("Load testing GPRSForwarder").repeat(1) {
exec(jms("testing GPRSForwarder...").reqreply
.queue("COMPANY.TEST.QUEUE")
.textMessage("00001404020611100E033102C20EBB51CC1C036EFFFF00010002323802000200FE05001400000000FFFFFFFFFFFFFFFFFF040010038B0D6912EB10CE070206110F37298C")
.jmsType("test_jms_type")
)
}
setUp(scn.inject(rampUsersPerSec(10) to 1000 during (2 minutes)))
.protocols(jmsConfiguration)
}
The setup is equivalent to this one - but instead of using ActiveMQInitalContextFactory, I'm forced to use the "counterpart" of IBM MQ.
According to official docks the WMQInitialContextFactory should be in com.ibm.mq.jms.context but it is not. Ore is there some constant in CommonConstants which I can use to initialize a ContextFactory ?
Thanks a lot in advance.
Problem: I'm not able to initialize the required ContextFactory with
IBM MQ - which should be
com.ibm.mq.jms.context.WMQInitialContextFactory. The
WMQInitialContextFactory cannot be found.
Do not use WMQInitialContextFactory. It was an MQ SupportPac someone created where they wanted to use MQ as the JNDI repository. It is not a good idea plus the SupportPac does not support any form of security (i.e. SSL/TLS or security exit).
You should use a file based MQ JNDI. Here's a basic example:
import java.util.Hashtable;
import javax.jms.JMSException;
import javax.jms.Queue;
import javax.jms.QueueConnection;
import javax.jms.QueueConnectionFactory;
import javax.jms.QueueSender;
import javax.jms.QueueSession;
import javax.jms.Session;
import javax.jms.TextMessage;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import com.ibm.mq.MQException;
/**
* Program Name
* TestJMS01
*
* Description
* This java JMS class will connect to a remote queue manager
* using JNDI and put a message to a queue.
*
* Sample Command Line Parameters
* -x myQCF -q dev.test.q -f C:\JNDI-Directory\roger\mqjndi
*
* Sample MQ JNDI Commands:
* DEFINE QCF(myQCF) QMANAGER(MQA1) CHANNEL(TEST.CHL) HOSTNAME(127.0.0.1) PORT(1415) TRANSPORT(CLIENT) FAILIFQUIESCE(YES)
* DEFINE Q(dev.test.q) QUEUE(TEST1) QMANAGER(MQA1) TARGCLIENT(JMS) FAILIFQUIESCE(YES)
*
* #author Roger Lacroix, Capitalware Inc.
*/
public class TestJMS01
{
private static final String JNDI_CONTEXT = "com.sun.jndi.fscontext.RefFSContextFactory";
private QueueConnectionFactory cf;
private Queue q;
private Hashtable<String,String> params = null;
private String userID = "tester";
private String password = "mypwd";
public TestJMS01() throws NamingException
{
super();
}
/**
* Make sure the required parameters are present.
* #return true/false
*/
private boolean allParamsPresent()
{
return (params.containsKey("-x") && params.containsKey("-q") && params.containsKey("-f"));
}
/**
* Extract the command-line parameters and initialize the MQ variables.
* #param args
* #throws IllegalArgumentException
*/
private void init(String[] args) throws IllegalArgumentException
{
params = new Hashtable<String,String>(10);
if (args.length > 0 && (args.length % 2) == 0)
{
for (int i = 0; i < args.length; i += 2)
{
params.put(args[i], args[i + 1]);
}
}
else
{
throw new IllegalArgumentException();
}
if (allParamsPresent())
{
Hashtable<String,Object> env = new Hashtable<String,Object>();
env.put(Context.INITIAL_CONTEXT_FACTORY, JNDI_CONTEXT);
env.put(Context.PROVIDER_URL, "file:/"+(String) params.get("-f"));
try
{
Context ctx = new InitialContext(env);
cf = (QueueConnectionFactory) ctx.lookup((String) params.get("-x"));
q = (Queue) ctx.lookup((String) params.get("-q"));
}
catch (NamingException e)
{
System.err.println(e.getLocalizedMessage());
e.printStackTrace();
throw new IllegalArgumentException();
}
}
else
{
throw new IllegalArgumentException();
}
}
/**
* Test the connection to the queue manager.
* #throws MQException
*/
private void testConn() throws JMSException
{
QueueConnection connection = null;
QueueSession session = null;
try
{
connection = cf.createQueueConnection(userID, password);
connection.start();
session = connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
sendMsg(session);
}
catch (JMSException e)
{
System.err.println("getLinkedException()=" + e.getLinkedException());
System.err.println(e.getLocalizedMessage());
e.printStackTrace();
}
finally
{
try
{
session.close();
}
catch (Exception ex)
{
System.err.println("session.close() : " + ex.getLocalizedMessage());
}
try
{
connection.stop();
}
catch (Exception ex)
{
System.err.println("connection.stop() : " + ex.getLocalizedMessage());
}
try
{
connection.close();
}
catch (Exception ex)
{
System.err.println("connection.close() : " + ex.getLocalizedMessage());
}
}
}
/**
* Send a message to a queue.
* #throws MQException
*/
private void sendMsg(QueueSession session) throws JMSException
{
QueueSender sender = null;
try
{
TextMessage msg = session.createTextMessage();
msg.setText("Nice simple test. Time in 'ms' is -> " + System.currentTimeMillis());
// msg.setJMSReplyTo(tq);
// msg.setJMSDeliveryMode( DeliveryMode.NON_PERSISTENT);
System.out.println("Sending request to " + q.getQueueName());
System.out.println();
sender = session.createSender(q);
sender.send(msg);
}
catch (JMSException e)
{
System.err.println("getLinkedException()=" + e.getLinkedException());
System.err.println(e.getLocalizedMessage());
e.printStackTrace();
}
finally
{
try
{
sender.close();
}
catch (Exception ex)
{
System.err.println("sender.close() : " + ex.getLocalizedMessage());
}
}
}
/**
* main line
* #param args
*/
public static void main(String[] args)
{
try
{
TestJMS01 tj = new TestJMS01();
tj.init(args);
tj.testConn();
}
catch (IllegalArgumentException e)
{
System.err.println("Usage: java TestJMS01 -x QueueConnectionFactoryName -q JMS_Queue_Name -f path_to_MQ_JNDI");
System.exit(1);
}
catch (NamingException ex)
{
System.err.println(ex.getLocalizedMessage());
ex.printStackTrace();
}
catch (JMSException e)
{
System.err.println("getLinkedException()=" + e.getLinkedException());
System.err.println(e.getLocalizedMessage());
e.printStackTrace();
}
catch (Exception ex)
{
System.err.println(ex.getLocalizedMessage());
ex.printStackTrace();
}
}
}
Not directly related to your problem, but JMSToolBox with its scripting feature, is a tool that allows to perform bulk/load test on IBM MQ.
With it you can define a script, composed of steps, that will read a template with variable placeholders and repeatedly put messages in one or multiple destinations. It can also directly read saved messages from a directory etc.
You can download it from SourceForge here
Documentation of the script feature is here

Getting ambiguous result using JDBC Metadata API for Hive

I am trying to get Table names for hive using DatabaseMetaData in a similar way like RDBMS.
Sample code:
try (Connection con = getJdbcConnection(connectionUri, driverName, username, password);) {
DatabaseMetaData metadata = con.getMetaData();
ResultSet rs = metadata.getTables(null, null, tableName, null);
while (rs.next()) {
System.out.println(rs.getString(3));
}
} catch (SQLException e) {
}
private static void registerDriver(String driverName) {
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
LOG.error("No class found for " + driverName + ". Details: " + e);
}
}
private static Connection getJdbcConnection(String connectionUri, String driverName, String username,
String password) throws SQLException{
registerDriver(driverName);
return DriverManager.getConnection(connectionUri, username,password);
}
There is no table in a particular database. Using different different table names, I am getting different output.
For example:
I put table name emp, there are 3 records with name emp
I put table name employee, there are 5 records with name employee
I put table name emp12, it is returning no records (which is expected)
Am I doing something wrong?
Shouldn't I use DatabaseMetaData for checking table existence?
I need to pass schema name in getTables method
Signature:
ResultSet getTables(String catalog,
String schemaPattern,
String tableNamePattern,
String[] types)
throws SQLException
I passed following agruments:
catalog = null;
schemaPattern = Hive schema Name
tableNamePattern = Hive Table Name
types = new String[] { "TABLE" }
Sample code:
try (Connection con = getJdbcConnection(connectionUri, driverName, username, password);) {
DatabaseMetaData metadata = con.getMetaData();
ResultSet rs = metadata.getTables(null, schemaName, tableName, new String[] { "TABLE" });
while (rs.next()) {
String tName = rs.getString("TABLE_NAME");
if (tName != null && tName.equals(tableName)) {
LOG.info("Table [" + tableName + "] is present in the Database.");
return true;
}
}
rs.close();
LOG.info("Table [" + tableName + "] is not present in the Database.");
return false;
} catch (SQLException e) {
LOG.error("Not able to get Table Metadata . Caused By: " + e);
}

Hive "ANALYZE TABLE" how to execute from java

I need to compute the number of rows in a hive table, for that
I am using the query:
ANALYZE TABLE p_7 COMPUTE STATISTICS noscan
I want to fetch the results through java, I am trying with the below
code and have no luck. the error I get is :
Exception in thread "main" java.sql.SQLException: The query did not generate a result set!
at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:393)
at HiveJdbcClient.main(HiveJdbcClient.java:22)
code I am using is :
import java.sql.SQLException;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;
import java.sql.DriverManager;
public class HiveJdbcClient {
private static String driverName = "org.apache.hive.jdbc.HiveDriver";
public static void main(String[] args) throws SQLException {
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
e.printStackTrace();
System.exit(1);
}
Connection con = DriverManager.getConnection("jdbc:hive2://localhost:10000/default", "hive", "");
System.out.println("connected");
Statement statement = con.createStatement();
String query = "ANALYZE TABLE p_7 COMPUTE STATISTICS noscan";
ResultSet res = statement.executeQuery(query);
}
}
I dont know how to execute a query such as:
ANALYZE TABLE p_7 COMPUTE STATISTICS noscan
through java. Any help on this would be of great help to me. Thanks.
Use the ANALYZE TABLE statement without 'NOSCAN' to compute the number of rows.
Note: This statement does not produce resultSet object.
To fetch the stored stats, use the following statement.
DESCRIBE FORMATTED tableName
In the output, the number of rows is listed in parameters array. Use regex to extract it.
Here is the sample code:
String analyzeQuery = "ANALYZE TABLE p_7 COMPUTE STATISTICS";
String describeQuery = "DESCRIBE FORMATTED p_7";
stmt.execute(analyzeQuery);
StringBuilder sb = new StringBuilder();
try (ResultSet rs = stmt.executeQuery(describeQuery)) {
while (rs.next()) {
int count = rs.getMetaData().getColumnCount();
for (int j = 1; j <= count; j++) {
sb.append(rs.getString(j));
}
}
}
System.out.println("Output: "+ sb.toString());
Refer https://cwiki.apache.org/confluence/display/Hive/StatsDev for details on Table and Partition statistics.
Try the below code for getting number of rows of a table:
public static Connection createConnection(String hive_ip)
{
String hive_url="jdbc:hive2://"+hive_ip;
Connection con=null;
try {
Class.forName("org.apache.hive.jdbc.HiveDriver");
System.out.println(hive_url+"/");
con = DriverManager.getConnection(
hive_url+"/",
hive_username,hive_password);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return con;
}
public static int getHiveColumnRowCount(String tablename,String db_name)
{
int count=0;
Connection con=createConnection();
try {
Statement st=con.createStatement();
int i=0;
String count_query="show tblproperties "+db_name+"."+tablename;
ResultSet rs=st.executeQuery(count_query);
while(rs.next())
{
i++;
if(i==3)
{
count=Integer.parseInt(rs.getString(2));
}
}
System.out.println("COUNT:"+count);
rs.close();
st.close();
con.close();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return count;
}
Hope it helps :)

SQLException Handling

I am trying to run a simple Java program which fetches data from the oracle database and display it. I connected the oracle database. Here is my code:
DataHandler Class:
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.ResultSet;
import oracle.jdbc.pool.OracleDataSource;
public class DataHandler {
public DataHandler() {
super();
}
String jdbcUrl = "jdbc:oracle:thin:#localhost:1521:ORCL";
//I already added the above line but still getting error.
String userid = "scott";
String password = "tiger";
Connection conn;
public void getDBConnection() throws SQLException{
OracleDataSource ds;
ds = new OracleDataSource();
ds.setUser(jdbcUrl);
conn = ds.getConnection(userid,password);
}
Statement stmt;
ResultSet rset;
String query;
String sqlString;
public ResultSet getAllEmployees() throws SQLException{
getDBConnection();
stmt = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_READ_ONLY);
query = "SELECT * FROM emp ORDER BY empno";
System.out.println("\nExecuting query: " + query);
rset = stmt.executeQuery(query);
return rset;
}
}
and the JavaClient Class as
JavaCLient CLass:
import java.sql.ResultSet;
public class JavaClient {
public JavaClient() {
super();
}
public static void main(String[] args) throws Exception{
DataHandler datahandler = new DataHandler();
ResultSet rset = datahandler.getAllEmployees();
while (rset.next()) {
System.out.println(rset.getInt(1) + " " +
rset.getString(2) + " " +
rset.getString(3) + " " +
rset.getString(4));
}
}
}
I get no compilation error but while running it I get following exception error
Error:
Exception in thread "main" java.sql.SQLException: Invalid Oracle URL specified: OracleDataSource.makeURL
at oracle.jdbc.pool.OracleDataSource.makeURL(OracleDataSource.java:1277)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:185)
at student_attendence_iem.DataHandler.getDBConnection(DataHandler.java:22)
at student_attendence_iem.DataHandler.getAllEmployees(DataHandler.java:31)
at student_attendence_iem.JavaClient.main(JavaClient.java:9)
Process exited with exit code 1.
Please help me. Thanks in advance. :)
You have not set URL of your database.
Add setURL(url) method which takes URL of database as parameter. Below is the code.
OracleDataSource ds;
ds = new OracleDataSource();
ds.setURL(jdbcUrl);
Also, with ds.setUser(jdbcUrl); you are trying to setUser with the URL of database which is wrong.
You don't have to setUser like this as you are already doing that in the following line of code conn = ds.getConnection(userid,password);

Doesn't match table_names jdbc program showing result and sql showing result

My jdbc program code
package table;
import java.sql.*;
public class sdfjksjk {
static final String JDBC_DRIVER = "oracle.jdbc.driver.OracleDriver";
static final String DB_URL = "jdbc:oracle:thin:#192.168.1.12:1521:aftdb";
static final String USER = "system";
static final String PASS = "manager";
public static void main(String[] args) {
Connection conn = null;
Statement stmt = null;
try{
Class.forName("oracle.jdbc.driver.OracleDriver");
System.out.println("Connecting to database...");
conn = DriverManager.getConnection(DB_URL,USER,PASS);
System.out.println("Creating statement...");
stmt = conn.createStatement();
String sql;
DatabaseMetaData md = conn.getMetaData();
ResultSet rs = md.
getTables(null, "SYSTEM", "%", null);
while (rs.next())
{
System.out.println(rs.getString(3));
}
rs.close();
stmt.close();
conn.close();
}catch(SQLException se){
se.printStackTrace();
}catch(Exception e){
e.printStackTrace();
}finally{
try{
if(stmt!=null)
stmt.close();
}catch(SQLException se2){
}
try{
if(conn!=null)
conn.close();
}catch(SQLException se){
se.printStackTrace();
}
}
}
}
this code showing 231 table name,but in my sql developer select table_name from user_tables it showing 207 table names. What is the wrong in my program?
The last parameter of md.getTables is a list of table types.
You supply the value null which lists all table types ("TABLE", "VIEW" and possibly other types).
In sql developer you are just seeing regular tables (not views).
Edit:
Make the call like this to just get regular tables:
String regularTables[] = new String[] {"TABLE"};
ResultSet rs = md.
getTables(null, "SYSTEM", "%", regularTables);

Resources