Error while trying to load data into hive table - hadoop

I was able to create a table into hbase using hive now I'm trying to load data into a hive table then overwrite the data into the hbase table :
public class HiveJdbcClient {
private static String driverName = "org.apache.hadoop.hive.jdbc.HiveDriver";
/**
* #param args
* #throws SQLException
**/
public static void main(String[] args) throws SQLException {
try {
Class.forName(driverName);
} catch (ClassNotFoundException e){
// TODO Auto-generated catch block
e.printStackTrace();
System.exit(1);
}
Connection con = DriverManager.getConnection("jdbc:hive://localhost:10000/default", "", "");
Statement stmt = con.createStatement();
String tableNameHive = "hbase_trades";
String tableNameHbase= "trades";
stmt.executeQuery("drop table " + tableNameHive);
ResultSet res = stmt.executeQuery("create table " + tableNameHive + " (key string, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (\"hbase.columns.mapping\" = \":key,cf1:val\") TBLPROPERTIES (\"hbase.table.name\" = \"trades\")");
String sql = "show tables '" + tableNameHive + "'";
System.out.println("Running: " + sql);
res = stmt.executeQuery(sql);
if (res.next()) {
System.out.println(res.getString(1));
}
sql = "describe " + tableNameHive;
System.out.println("Running: " + sql);
res = stmt.executeQuery(sql);
while (res.next()) {
System.out.println(res.getString(1) + "\t" + res.getString(2));
}
String filepath = "/tmp/test_hive_server.txt";
sql = "load data local inpath '" + filepath + "' into table " + tableNameHive;
System.out.println("Running: " + sql);
res = stmt.executeQuery(sql);
stmt.executeQuery("insert overwrite " + tableNameHbase+"select * from"+tableNameHive);
}
}
and I get the following error:
Running: load data local inpath '/tmp/test_hive_server.txt' into table hbase_trades
Exception in thread "main" java.sql.SQLException: Query returned non-zero code: 10101, cause: FAILED: SemanticException [Error 10101]: A non-native table cannot be used as target for LOAD
at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:194)
at com.palmyra.nosql.HiveJdbcClient.main(HiveJdbcClient.java:53)
could someone tell me what's the problem??

Related

Output SQL Table to existing Excel file/ new worksheet in SSIS / C# in script task

Good day,
I had a request to create a SSIS package that would output an SQL table to an Excel file. I had no problem creating this. However, the client came back asking that they wanted to be able to output the SQL table content to an existing Excel file in a new worksheet. If the worksheet does not exists in my following script, it is being created. However, it just goes back in the loop and fail because now it exists.
Here is my code:
public void Main()
{
string datetime = DateTime.Now.ToString("yyyyMMddHHmmss");
try
{
//Declare Variables
string ExcelFileName = Dts.Variables["$Package::ExcelFileName"].Value.ToString();
string FolderPath = Dts.Variables["$Package::FolderPath"].Value.ToString();
string TableName = Dts.Variables["$Package::SQLTableName"].Value.ToString();
string SchemaName = Dts.Variables["$Package::SQLTableSchema"].Value.ToString();
string SheetName = Dts.Variables["$Package::SheetName"].Value.ToString();
string lastChar = FolderPath.Substring(FolderPath.Length - 1);
string currentTab;
DataTable ExcelFileTabs;
//Validate format of FolderPath
if (lastChar != "\\")
{
FolderPath = FolderPath + "\\";
}
string FullExcelFilePath = FolderPath + ExcelFileName + ".xlsx";
OleDbConnection Excel_OLE_Con = new OleDbConnection();
OleDbCommand Excel_OLE_Cmd = new OleDbCommand();
//Construct ConnectionString for Excel
string connstring = "Provider=Microsoft.ACE.OLEDB.12.0;" + "Data Source=" + FullExcelFilePath + ";" + "Extended Properties=\"Excel 12.0 Xml;HDR=YES;\"";
//USE ADO.NET Connection from SSIS Package to get data from table
SqlConnection myADONETConnection = new SqlConnection();
myADONETConnection = (SqlConnection)(Dts.Connections["ADO_DBConnection"].AcquireConnection(Dts.Transaction) as SqlConnection);
//Load Data into DataTable from SQL ServerTable
// Assumes that connection is a valid SqlConnection object.
string queryString = "SELECT * from " + SchemaName + "." + TableName;
SqlDataAdapter adapter = new SqlDataAdapter(queryString, myADONETConnection);
DataSet ds = new DataSet();
adapter.Fill(ds);
//Get Header Columns
string TableColumns = "";
// Get the Column List from Data Table so can create Excel Sheet with Header
foreach (DataTable table in ds.Tables)
{
foreach (DataColumn column in table.Columns)
{
TableColumns += column + "],[";
}
}
// Replace most right comma from Columnlist
TableColumns = ("[" + TableColumns.Replace(",", " Text,").TrimEnd(','));
TableColumns = TableColumns.Remove(TableColumns.Length - 2);
//Use OLE DB Connection and Create Excel Sheet
Excel_OLE_Con.ConnectionString = connstring;
Excel_OLE_Con.Open();
Excel_OLE_Cmd.Connection = Excel_OLE_Con;
// Verify if file exists
if (File.Exists(FullExcelFilePath))
{
//Verify if the sheet exists
foreach (DataTable table in ds.Tables)
{
ExcelFileTabs = Excel_OLE_Con.GetSchema("Tables");
foreach (DataRow excelTable in ExcelFileTabs.Rows)
{
currentTab = excelTable["TABLE_NAME"].ToString();
if (currentTab == SheetName)
{
// Create Log File for Errors
using (StreamWriter sw = File.CreateText(Dts.Variables["$Package::FolderPath"].Value.ToString() + "\\" + Dts.Variables["$Package::ExcelFileName"].Value.ToString() + "_" + datetime + ".log"))
{
sw.WriteLine("The sheet " + SheetName + " that your are trying to create in " + FullExcelFilePath + " already exists.");
sw.WriteLine("Please enter another sheet name or delete the Excel file and try again.");
}
Excel_OLE_Con.Close();
Dts.TaskResult = (int)ScriptResults.Failure;
}
else
{
// Create the worksheet in the existing Excel file
Excel_OLE_Cmd.CommandText = "Create table " + SheetName + " (" + TableColumns + ")";
Excel_OLE_Cmd.ExecuteNonQuery();
}
}
}
}
else
{
Excel_OLE_Cmd.CommandText = "Create table " + SheetName + " (" + TableColumns + ")";
Excel_OLE_Cmd.ExecuteNonQuery();
}
//Write Data to Excel Sheet from DataTable dynamically
foreach (DataTable table in ds.Tables)
{
ExcelFileTabs = Excel_OLE_Con.GetSchema("Tables");
foreach (DataRow excelTable in ExcelFileTabs.Rows)
{
String sqlCommandInsert = "";
String sqlCommandValue = "";
foreach (DataColumn dataColumn in table.Columns)
{
sqlCommandValue += dataColumn + "],[";
}
sqlCommandValue = "[" + sqlCommandValue.TrimEnd(',');
sqlCommandValue = sqlCommandValue.Remove(sqlCommandValue.Length - 2);
sqlCommandInsert = "INSERT into " + SheetName + "(" + sqlCommandValue.TrimEnd(',') + ") VALUES(";
int columnCount = table.Columns.Count;
foreach (DataRow row in table.Rows)
{
string columnvalues = "";
for (int i = 0; i < columnCount; i++)
{
int index = table.Rows.IndexOf(row);
columnvalues += "'" + table.Rows[index].ItemArray[i] + "',";
}
columnvalues = columnvalues.TrimEnd(',');
var command = sqlCommandInsert + columnvalues + ")";
Excel_OLE_Cmd.CommandText = command;
Excel_OLE_Cmd.ExecuteNonQuery();
}
Excel_OLE_Con.Close();
Dts.TaskResult = (int)ScriptResults.Success;
}
}
}
catch (Exception exception)
{
// Create Log File for Errors
using (StreamWriter sw = File.CreateText(Dts.Variables["$Package::FolderPath"].Value.ToString() + "\\" + Dts.Variables["$Package::ExcelFileName"].Value.ToString() + "_" + datetime + ".log"))
{
sw.WriteLine(exception.ToString());
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
}
Can somebody please help me with this ? I am pretty new to C#, and English is not my primary language. Please let me know if this is not clear enough.
Thanks in advance for you time :-)
Mylene
You know when you are trying to find the solution way to fare when it is just in front of you ?
My code, before wanting to catch the error if the user was trying to add a worksheet that already exists, already was trowing an exception in those cases...
public void Main()
{
string datetime = DateTime.Now.ToString("yyyyMMddHHmmss");
try
{
//Declare Variables
string ExcelFileName = Dts.Variables["$Package::ExcelFileName"].Value.ToString();
string FolderPath = Dts.Variables["$Package::FolderPath"].Value.ToString();
string TableName = Dts.Variables["$Package::SQLTableName"].Value.ToString();
string SchemaName = Dts.Variables["$Package::SQLTableSchema"].Value.ToString();
string SheetName = Dts.Variables["$Package::SheetName"].Value.ToString();
ExcelFileName = ExcelFileName + "_" + datetime;
string lastChar = FolderPath.Substring(FolderPath.Length - 1);
//Validate format of FolderPath
if (lastChar != "\\")
{
FolderPath = FolderPath + "\\";
}
OleDbConnection Excel_OLE_Con = new OleDbConnection();
OleDbCommand Excel_OLE_Cmd = new OleDbCommand();
//Construct ConnectionString for Excel
string connstring = "Provider=Microsoft.ACE.OLEDB.12.0;" + "Data Source=" + FolderPath + ExcelFileName
+ ";" + "Extended Properties=\"Excel 12.0 Xml;HDR=YES;\"";
//USE ADO.NET Connection from SSIS Package to get data from table
SqlConnection myADONETConnection = new SqlConnection();
myADONETConnection = (SqlConnection)(Dts.Connections["ADO_DBConnection"].AcquireConnection(Dts.Transaction) as SqlConnection);
//Load Data into DataTable from SQL ServerTable
// Assumes that connection is a valid SqlConnection object.
string queryString = "SELECT * from " + SchemaName + "." + TableName;
SqlDataAdapter adapter = new SqlDataAdapter(queryString, myADONETConnection);
DataSet ds = new DataSet();
adapter.Fill(ds);
//Get Header Columns
string TableColumns = "";
// Get the Column List from Data Table so can create Excel Sheet with Header
foreach (DataTable table in ds.Tables)
{
foreach (DataColumn column in table.Columns)
{
TableColumns += column + "],[";
}
}
// Replace most right comma from Columnlist
TableColumns = ("[" + TableColumns.Replace(",", " Text,").TrimEnd(','));
TableColumns = TableColumns.Remove(TableColumns.Length - 2);
//Use OLE DB Connection and Create Excel Sheet
Excel_OLE_Con.ConnectionString = connstring;
Excel_OLE_Con.Open();
Excel_OLE_Cmd.Connection = Excel_OLE_Con;
Excel_OLE_Cmd.CommandText = "Create table " + SheetName + " (" + TableColumns + ")";
Excel_OLE_Cmd.ExecuteNonQuery();
//Write Data to Excel Sheet from DataTable dynamically
foreach (DataTable table in ds.Tables)
{
String sqlCommandInsert = "";
String sqlCommandValue = "";
foreach (DataColumn dataColumn in table.Columns)
{
sqlCommandValue += dataColumn + "],[";
}
sqlCommandValue = "[" + sqlCommandValue.TrimEnd(',');
sqlCommandValue = sqlCommandValue.Remove(sqlCommandValue.Length - 2);
sqlCommandInsert = "INSERT into " + SheetName + "(" + sqlCommandValue.TrimEnd(',') + ") VALUES(";
int columnCount = table.Columns.Count;
foreach (DataRow row in table.Rows)
{
string columnvalues = "";
for (int i = 0; i < columnCount; i++)
{
int index = table.Rows.IndexOf(row);
columnvalues += "'" + table.Rows[index].ItemArray[i] + "',";
}
columnvalues = columnvalues.TrimEnd(',');
var command = sqlCommandInsert + columnvalues + ")";
Excel_OLE_Cmd.CommandText = command;
Excel_OLE_Cmd.ExecuteNonQuery();
}
}
Excel_OLE_Con.Close();
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception exception)
{
// Create Log File for Errors
using (StreamWriter sw = File.CreateText(Dts.Variables["$Package::FolderPath"].Value.ToString() + "\\" + Dts.Variables["$Package::ExcelFileName"].Value.ToString() + datetime + ".log"))
{
sw.WriteLine(exception.ToString());
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
}
}
}
Error log :
System.Data.OleDb.OleDbException (0x80040E14): Table 'Test9' already exists.
at System.Data.OleDb.OleDbCommand.ExecuteCommandTextErrorHandling(OleDbHResult hr)
at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(tagDBPARAMS dbParams, Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
at System.Data.OleDb.OleDbCommand.ExecuteNonQuery()
at ST_a21007570143466693913591932c30b7.ScriptMain.Main()
Problem resolved.

How to make Hibernate Transaction object in recursive call

I am getting below error in my console when I call recursively a method. The update query is running fine but it will not be updating record in the database.
org.springframework.transaction.TransactionSystemException: Could not commit Hibernate transaction; nested exception is org.hibernate.TransactionException: Transaction not successfully started
public boolean abcMethod() {
Transaction txn = session.beginTransaction();
String querySqlSold = "UPDATE abc_table SET inventory_type='SOLD', status='ACTIVE' where set_id="
+ setId + " and game_num=" + gameMaster.getGameNum() + " and priceScheme=" + prizeSchemeId;
SQLQuery querySold = session.createSQLQuery(querySqlSold);
querySold.executeUpdate();
String querySqlSelect = "SELECT set_id FROM abc_table where inventory_type='UPCOMING' and `status`='ACTIVE' and game_num="
+ gameMaster.getGameNum() + " and priceScheme=" + prizeSchemeId;
List list = session.createSQLQuery(querySqlSelect).list();
int newSetId = Integer.valueOf(list.get(0).toString());
if (newSetId != 0) {
String querySqlCurrent = "UPDATE abc_table SET inventory_type='CURRENT' where game_num="
+ gameMaster.getGameNum() + " and priceScheme=" + prizeSchemeId + " and set_id=" + newSetId;
SQLQuery queryCurrent = session.createSQLQuery(querySqlCurrent);
queryCurrent.executeUpdate();
txn.commit();
return true;
} else {
JSONObject jsonObject = new JSONObject();
jsonObject.put("errorCode", "809");
jsonObject.put("errorMsg", "finished");
throw new CustomException(jsonObject.toString());
}
public void xyzMethod() {
abcMethod();
abcMethod();
}
You might try something like this
try {
tx = session.beginTransaction();
if (!tx.wasCommitted()){
tx.commit();
}
} catch (Exception exp) {
tx.rollback();
}
It should help you understand the problem better.

oracle jdbc driver reports no primary key columns on a table that has a primary key

This was reported by HibernateTools Reverse Engineering, but it seems to be true.
oracle jdbc driver reports no primary key columns on a table that has a primary key.
#Test
public void checkTable() throws SQLException, IOException {
System.out.println("in check table");
assertNotNull(conn);
Statement s = conn.createStatement();
ResultSet rset = s.executeQuery("select user from dual");
rset.next();
String username = rset.getString(1);
rset.close();
try {
s.execute("drop table " + username + ".x");
} catch (Exception e) {
// nothing it might not exist
}
s.execute("create table " + username + ".x (y number)");
s.execute("alter table x add constraint x_pk primary key (y)");
DatabaseMetaData meta = conn.getMetaData();
final String[] tableTypes = new String[] { "TABLE", "VIEW" };
ResultSet rs = meta.getTables(null, username, "X",tableTypes);
rs.next();
String table = rs.getString("table_name");
System.out.println("table is " + table);
rs.close();
rs = s.executeQuery("select * from user_constraints where table_name = 'X'");
rs.next();
String type = rs.getString("constraint_type");
assertEquals("P",type); // primary key
rs.close();
rs = meta.getPrimaryKeys(null, username, "X");
rs.next();
logger.info("getting pk");
System.out.print("wtf");
int colCount = 0;
while (rs.next()) {
final String pkName = rs.getString("pk_name");
logger.info("pkName: {}", pkName);
int keySeq = rs.getShort("key_seq"); // TODO should probably be column seq
String columnName = rs.getString("column_name");
logger.warn("seq: {}, columnName: {}, keySeq, columnName");
colCount++;
}
System.out.println("colCount: " + colCount);
assertEquals(1,colCount);
}

Not able to access Hive Tables through JDBC in Hortonworks Sandbox 2.0

I am using Hortonworks Sandbox 2.0. I tried the following program in the Eclipse IDE, but was not table to access my Hive tables. Got the following errors. What do I have to do to resolve this?
I used this also: hive –service hiveserver, and got Not able to connect. I am using VMware.
import java.sql.SQLException;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;
import java.sql.DriverManager;
public class HiveJdbcClient {
private static String driverName = “org.apache.hadoop.hive.jdbc.HiveDriver”;
/**
* #param args
* #throws SQLException
*/
public static void main(String[] args) throws SQLException {
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
System.exit(1);
}
Connection con = DriverManager.getConnection(“jdbc:hive://localhost:10000/default”, “”, “”);
Statement stmt = con.createStatement();
String tableName = “testHiveDriverTable”;
stmt.executeQuery(“drop table ” + tableName);
ResultSet res = stmt.executeQuery(“create table ” + tableName + ” (key int, value string)”);
// show tables
String sql = “show tables ‘” + tableName + “‘”;
System.out.println(“Running: ” + sql);
res = stmt.executeQuery(sql);
if (res.next()) {
System.out.println(res.getString(1));
}
// describe table
sql = “describe ” + tableName;
System.out.println(“Running: ” + sql);
res = stmt.executeQuery(sql);
while (res.next()) {
System.out.println(res.getString(1) + “\t” + res.getString(2));
}
// load data into table
// NOTE: filepath has to be local to the hive server
// NOTE: /tmp/a.txt is a ctrl-A separated file with two fields per line
String filepath = “/tmp/a.txt”;
sql = “load data local inpath ‘” + filepath + “‘ into table ” + tableName;
System.out.println(“Running: ” + sql);
res = stmt.executeQuery(sql);
// select * query
sql = “select * from ” + tableName;
System.out.println(“Running: ” + sql);
res = stmt.executeQuery(sql);
while (res.next()) {
System.out.println(String.valueOf(res.getInt(1)) + “\t” + res.getString(2));
}
// regular hive query
sql = “select count(1) from ” + tableName;
System.out.println(“Running: ” + sql);
res = stmt.executeQuery(sql);
while (res.next()) {
System.out.println(res.getString(1));
}
}
}
—-
GOT ERROR
Exception in thread “main” java.sql.SQLException: Could not establish connection to 172.31.153.71:10000/default: java.net.ConnectException: Connection refused: connect
at org.apache.hadoop.hive.jdbc.HiveConnection.<init>(HiveConnection.java:117)
at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:106)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at com.coe.convert.hive.temp.htw.HiveJdbcClient.main(HiveJdbcClient.java:28)
Run hive on another port. And then use the new port number it will work. Use the following command to change the port number 10000 to 10001.
hive --service hiveserver -p 10001
Then Use ,
172.31.153.71:10000/default

glassfish 3.1.2 - ResultSetWrapper40 cannot be cast to oracle.jdbc.OracleResultSet

I recently migrate from glassfish 3.1.1 to 3.1.2 and I got the following error
java.lang.ClassCastException: com.sun.gjc.spi.jdbc40.ResultSetWrapper40 cannot be cast to oracle.jdbc.OracleResultSet
at the line
oracle.sql.BLOB bfile = ((OracleResultSet) rs).getBLOB("filename");
in the following routine:
public void fetchPdf(int matricola, String anno, String mese, String tableType, ServletOutputStream os) {
byte[] buffer = new byte[2048];
String query = "SELECT filename FROM "
+ tableType + " where matricola = " + matricola
+ " and anno = " + anno
+ ((tableType.equals("gf_blob_ced") || tableType.equals("gf_blob_car")) ? " and mese = " + mese : "");
InputStream ins = null;
//--------
try {
Connection conn = dataSource.getConnection();
//Connection conn = DriverManager.getConnection(connection, "glassfish", pwd);
java.sql.Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
if (rs.next()) {
logger.info("select ok " + query);
oracle.sql.BLOB bfile = ((OracleResultSet) rs).getBLOB("filename");
ins = bfile.getBinaryStream();
int length;
while ((length = (ins.read(buffer))) >= 0) {
os.write(buffer, 0, length);
}
ins.close();
} else {
logger.info("select Nok " + query);
}
rs.close();
stmt.close();
//conn.close();
} catch (IOException ex) {
logger.warn("blob file non raggiungibile: "+query);
} catch (SQLException ex) {
logger.warn("connessione non riuscita");
}
}
I'm using the glassfish connection pool
#Resource(name = "jdbc/ape4")
private DataSource dataSource;
and the jdbc/ape4 resource belongs to an oracle connection pool with the following param
NetworkProtocol tcp
LoginTimeout 0
PortNumber 1521
Password xxxxxxxx
MaxStatements 0
ServerName server
DataSourceName OracleConnectionPoolDataSource
URL jdbc:oracle:thin:#server:1521:APE4
User glassfish
ExplicitCachingEnabled false
DatabaseName APE4
ImplicitCachingEnabled false
The oracle driver is ojdbc6.jar, oracle DB is 10g.
Could anyone help me what is happening? On Glassfish 3.1.1 it was working fine.
There is no need for not using standard JDBC api in this code. You are not using any Oracle-specific functionality so rs.getBlob("filename").getBinaryStream() will work just as well.
If you insist on keeping this, turn off JDBC Object wrapping option for your datasource.

Resources