Reading hive data from soapUI framework - jdbc

I am new to SoapUI framework. I am trying to use soapUI framework for testing REST API. While testing for REST API, I need to verify data from backend database as well like Hive and Cassandra.
I could do the setup for SoapUI and could test a query on Cassandra using groovy script that is provided SoapUI framework. But when I searched for connecting to hive using SoapUI, I couldn't find any reference for that. Also on there sites, JDBC drivers are not provided but hive is not mentioned there.
So is there any option to connect to hive from SoapUI framework?
Should I think about using Hive JDBC driver from SoapUI?
Thanks for your help!

I believe you should be able to use it for different data bases using following ways:
JDBC test step
Groovy Script (you should be able to use almost java code)
Either of the ways, copy the drivers/libraries into SOAPUI_HOME/bin/ext directory and restart SoapUI
Here is the link for client code (in Java) to connect to Hive.
Sample connection code from above link(so should be able to use in groovy) :
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
System.exit(1);
}
Connection con = DriverManager.getConnection("jdbc:hive://localhost:10000/default", "", "");
Statement stmt = con.createStatement();
String tableName = "testHiveDriverTable";
stmt.executeQuery("drop table " + tableName);
ResultSet res = stmt.executeQuery("create table " + tableName + " (key int, value string)");
// show tables
String sql = "show tables '" + tableName + "'";
System.out.println("Running: " + sql);
res = stmt.executeQuery(sql);

Related

[Snowflake-jdbc]It hangs when get info from resetset object of connection.getMetadata().getColumns(...)

I try to test the jdbc connection of snowflake with codes below
Connection conn = .......
.......
ResultSet rs = conn.getMetaData().getColumns(**null**, "PUBLIC", "TAB1", null); // 1. set parameters to get metadata of table TAB1
while (rs.next()) { // 2. It hangs here if the first parameter is null in above liune; otherwise(set the corrent db name), it works fine
System.out.println( "precision:" + rs.getInt(7)
+ ",col type name:" + rs.getString(6)
+ ",col type:" + rs.getInt(5)
+ ",col name:" + rs.getString(4)
+ ",CHAR_OCTET_LENGTH:" + rs.getInt(16)
+ ",buf LENGTH:" + rs.getString(8)
+ ",SCALE:" + rs.getInt(9));
}
.......
I debug the codes above in Intellij IDEA, and find that the debugger can't get the details of the object, it always shows "Evaluating..."
The JDBC driver I used is snowflake-jdbc-3.12.5.jar
Is it a bug?
When the catalog (database) argument is null, the JDBC code effectively runs the following SQL, which you can verify in your Snowflake account's Query History UIs/Views:
show columns in account;
This is an expensive metadata query to run due to no filters and the wide requested breadth (columns across the entire account).
Depending on how many databases exist in your organization's account, it may require several minutes or upto an hour of execution to return back results, which explains the seeming "hang". On a simple test with about 50k+ tables dispersed across 100+ of databases and schemas, this took at least 15 minutes to return back results.
I debug the codes above in Intellij IDEA, and find that the debugger can't get the details of the object, it always shows "Evaluating..."
This may be a weirdness with your IDE, but in a pinch you can use the Dump Threads (Ctrl + Escape, or Ctrl + Break) option in IDEA to provide a single captured thread dump view. This should help show that the JDBC client thread isn't hanging (as in, its not locked or starved), it is only waiting on the server to send back results.
There is no issue with the 3.12.5 jar.I just tested the same version in Eclipse, I can inspect all the objects . Could be an issue with your IDE.
ResultSet columns = metaData.getColumns(null, null, "TESTTABLE123",null);
while (columns.next()){
System.out.print("Column name and size: "+columns.getString("COLUMN_NAME"));
System.out.print("("+columns.getInt("COLUMN_SIZE")+")");
System.out.println(" ");
System.out.println("COLUMN_DEF : "+columns.getString("COLUMN_DEF"));
System.out.println("Ordinal position: "+columns.getInt("ORDINAL_POSITION"));
System.out.println("Catalog: "+columns.getString("TABLE_CAT"));
System.out.println("Data type (integer value): "+columns.getInt("DATA_TYPE"));
System.out.println("Data type name: "+columns.getString("TYPE_NAME"));
System.out.println(" ");
}

Is there any way to view the physical SQLs executed by Calcite JDBC?

Recently I am studying Apache Calcite, by now I can use explain plan for via JDBC to view the logical plan, and I am wondering how can I view the physical sql in the plan execution? Since there may be bugs in the physical sql generation so I need to make sure the correctness.
val connection = DriverManager.getConnection("jdbc:calcite:")
val calciteConnection = connection.asInstanceOf[CalciteConnection]
val rootSchema = calciteConnection.getRootSchema()
val dsInsightUser = JdbcSchema.dataSource("jdbc:mysql://localhost:13306/insight?useSSL=false&serverTimezone=UTC", "com.mysql.jdbc.Driver", "insight_admin","xxxxxx")
val dsPerm = JdbcSchema.dataSource("jdbc:mysql://localhost:13307/permission?useSSL=false&serverTimezone=UTC", "com.mysql.jdbc.Driver", "perm_admin", "xxxxxx")
rootSchema.add("insight_user", JdbcSchema.create(rootSchema, "insight_user", dsInsightUser, null, null))
rootSchema.add("perm", JdbcSchema.create(rootSchema, "perm", dsPerm, null, null))
val stmt = connection.createStatement()
val rs = stmt.executeQuery("""explain plan for select "perm"."user_table".* from "perm"."user_table" join "insight_user"."user_tab" on "perm"."user_table"."id"="insight_user"."user_tab"."id" """)
val metaData = rs.getMetaData()
while(rs.next()) {
for(i <- 1 to metaData.getColumnCount) printf("%s ", rs.getObject(i))
println()
}
result is
EnumerableCalc(expr#0..3=[{inputs}], proj#0..2=[{exprs}])
EnumerableHashJoin(condition=[=($0, $3)], joinType=[inner])
JdbcToEnumerableConverter
JdbcTableScan(table=[[perm, user_table]])
JdbcToEnumerableConverter
JdbcProject(id=[$0])
JdbcTableScan(table=[[insight_user, user_tab]])
There is a Calcite Hook, Hook.QUERY_PLAN that is triggered with the JDBC query strings. From the source:
/** Called with a query that has been generated to send to a back-end system.
* The query might be a SQL string (for the JDBC adapter), a list of Mongo
* pipeline expressions (for the MongoDB adapter), et cetera. */
QUERY_PLAN;
You can register a listener to log any query strings, like this in Java:
Hook.QUERY_PLAN.add((Consumer<String>) s -> LOG.info("Query sent over JDBC:\n" + s));
It is possible to see the generated SQL query by setting calcite.debug=true system property. The exact place where this is happening is in JdbcToEnumerableConverter. As this is happening during the execution of the query you will have to remove the "explain plan for"
from stmt.executeQuery.
Note that by setting debug mode to true you will get a lot of other messages as well as other information regarding generated code.

TKProf (Oracle event 10046) on Spring Boot/JDBC

I'm trying to start oracle tracing through invoking direct JDBC calls. I'm obtaining my connection from Spring (boot/jdbc). Then I run the TKProf commands through statements... execute the query and print to the log.
The 3 statements below are returning false. If I use this same statements through Intellij's console I will get the intended results and my *.trc file is properly generated.
try (final Connection connection = DataSourceUtils.getConnection(dataSource)) {
log.debug(query);
final Long maxCount = findMaxCount();
boolean traceIdSet = connection.createStatement().execute("ALTER SESSION SET TRACEFILE_IDENTIFIER = '" + traceId + "'");
boolean traceEnabled = connection.createStatement().execute("ALTER SESSION SET EVENTS '10046 trace name context forever, level 8'");
final PreparedStatement stmt = connection.prepareStatement(query);
map(consumer, stmt.executeQuery(query));
boolean traceIdOff = connection.createStatement().execute("ALTER SESSION SET EVENTS '10046 trace name context off'");
log.debug("|" + traceIdSet + "|" + traceEnabled + "|" + traceIdOff + "| ____________________ DONE __________________________");
} catch (SQLException e) {
log.error("Error Performing the Query", e);
}
It has to be something in my configuration... I mean, java thin driver can do it because I can do it over the IDE... so I have to be missing some other stuff, maybe a Spring Boot convention that I should change.
Could you please help, any input is valuable.
Thanks!
My bad, the real issue was that I wasn't getting proper response from...
SELECT value FROM v$diag_info
Where I couldn't found the trace file, but only some others...
Nevertheless the trc files are in place, so there is no problem with Spring Boot/JDBC for enabling TKProf.

How do I make uCanAccess use Samba authentication, with special characters in username or password?

TL;DR: What Database.FileFormat constant should I use for an MS Access 2000-2003 database, when creating the Database object?
I have built a SAMBA test application using jCIFS. It allows me to create/overwrite files if given the correct authentication credentials, regardless of on which PC in the domain I use it.
I also have an application that uses uCanAccess/jackcess to connect to an MDB on a network share. However (from what I understand), it uses the credentials of the logged-in user, a number of whom have read-only access. Only system/network administrators have write permission.
The database in question is not password-protected. (I don't need to enter a password when opening it.)
My intention is to have the app ask for the administrator's Samba credentials before it writes to the DB, using those in the uCanAccess connection, so that it doesn't throw a java.nio.channels.NonWritableChannelException, as per the below stack trace:
java.nio.channels.NonWritableChannelException
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:747)
at com.healthmarketscience.jackcess.impl.PageChannel.writePage(PageChannel.java:310)
at com.healthmarketscience.jackcess.impl.PageChannel.writePage(PageChannel.java:247)
at com.healthmarketscience.jackcess.impl.TableImpl.writeDataPage(TableImpl.java:1980)
at com.healthmarketscience.jackcess.impl.TableImpl.addRows(TableImpl.java:2229)
at com.healthmarketscience.jackcess.impl.TableImpl.addRow(TableImpl.java:2067)
at net.ucanaccess.converters.UcanaccessTable.addRow(UcanaccessTable.java:44)
at net.ucanaccess.commands.InsertCommand.insertRow(InsertCommand.java:101)
at net.ucanaccess.commands.InsertCommand.persist(InsertCommand.java:148)
at net.ucanaccess.jdbc.UcanaccessConnection.flushIO(UcanaccessConnection.java:315)
at net.ucanaccess.jdbc.UcanaccessConnection.commit(UcanaccessConnection.java:205)
at net.ucanaccess.jdbc.AbstractExecute.executeBase(AbstractExecute.java:217)
at net.ucanaccess.jdbc.Execute.execute(Execute.java:46)
at net.ucanaccess.jdbc.UcanaccessPreparedStatement.execute(UcanaccessPreparedStatement.java:228)
at myapp.db.Digger.addTransaction(Digger.java:993)
at myapp.tasks.TransactionRunnable.run(TransactionRunnable.java:42)
at java.lang.Thread.run(Thread.java:745)
Update: I have tried using the smbFileChannel class by Gord Thompson and J. T. Alhborn, shown here. My code, based off the main class shown in that answer, looks like this:
// Ask the user for login credentials and the path to the database
String smbURL = (chosenDir.endsWith("/") ? chosenDir : chosenDir + '/')
+ dbName;
System.out.println("DB Path to use for URL: " + smbURL);
URL u = new URL(smbURL);
try (
// construct the SMB DB URL
SmbFileChannel sfc = new SmbFileChannel(smbURL);
Database db = new DatabaseBuilder().setChannel(sfc)
.setFileFormat(Database.FileFormat.GENERIC_JET4).create();
) {
// Model the table
Table tbl = new TableBuilder("Transactions")
.addColumn(new ColumnBuilder("TransactionID", DataType.LONG).setAutoNumber(true))
.addColumn(new ColumnBuilder("ControllerID", DataType.LONG).setAutoNumber(false))
.addColumn(new ColumnBuilder("ReaderID", DataType.LONG).setAutoNumber(false))
.addColumn(new ColumnBuilder("Event", DataType.LONG).setAutoNumber(false))
.addColumn(new ColumnBuilder("Timestamp", DataType.SHORT_DATE_TIME).setAutoNumber(false))
.addColumn(new ColumnBuilder("Number", DataType.LONG).setAutoNumber(false))
.addIndex(new IndexBuilder(IndexBuilder.PRIMARY_KEY_NAME).addColumns("TransactionID").setPrimaryKey())
.toTable(db);
// Add the row
Map<String, Object> values = new HashMap<>();
values.put("ControllerID", cid);
values.put("ReaderID", rid);
values.put("Event", evtNum);
values.put("Timestamp", ts); // Long; must be converted to DataType.SHORT_DATE_TIME
values.put("Number", accNum);
tbl.addRowFromMap(values);
} catch (IOException IOEx) {
System.err.println(
"Failed to write record to Transactions table in database: "
+ IOEx.getMessage()
);
IOEx.printStackTrace(System.err);
} catch (Exception ex) {
System.err.println(
'[' + ex.getClass().getSimpleName() + "]: Failed to write record to "
+ "Transactions table in database: " + ex.getMessage()
);
ex.printStackTrace(System.err);
}
Executing the above code results in the following output:
DB Path to use for URL: smb://machine.vpnName/Storage/me/dbs/DBName.mdb
Failed to write record to Transactions table in database: Logon failure: account currently disabled.
jcifs.smb.SmbAuthException: Logon failure: account currently disabled.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:546)
at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
at jcifs.smb.SmbSession.send(SmbSession.java:218)
at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
at jcifs.smb.SmbFile.connect(SmbFile.java:957)
at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
at jcifs.smb.SmbFile.open0(SmbFile.java:975)
at jcifs.smb.SmbFile.open(SmbFile.java:1009)
at jcifs.smb.SmbRandomAccessFile.<init>(SmbRandomAccessFile.java:57)
at jcifs.smb.SmbRandomAccessFile.<init>(SmbRandomAccessFile.java:42)
at samba.SmbFileChannel.<init>(SmbFileChannel.java:30)
at samba.SambaLanWriteTest.writeTest(SambaLanWriteTest.java:130)
at samba.SambaLanWriteTest.main(SambaLanWriteTest.java:181)
I have write access to a test copy of the database file in question when using Windows File Explorer. I am choosing that one when prompted.
Update 2: I realised that I neglected to add my username and password to the smb:// URL, as Thompson's example shows. I changed to code to this:
String smbCred = "smb://" + auth.getUsername() + ":" + auth.getPassword() + "#",
fixer = chosenDir.replace("\\", "/").replace("smb://", smbCred),
smbURL = fixer + dbName;
System.out.println("DB Path to use for URL: " + smbURL);
// URL u = new URL(smbURL);
The next problem I had was that my password contains special illegal characters (such as '#', ':', ';', '=' and '?'). I escaped these by using java.net.URLEncoder.encode() on auth.getUsername() and auth.getPassword() so the code doesn't throw a MalformedURLException when creating the SmbChannel. However, the next exception I encountered is as follows:
Failed to write record to Transactions table in database: File format GENERIC_JET4 [VERSION_4] does not support file creation for null
java.io.IOException: File format GENERIC_JET4 [VERSION_4] does not support file creation for null
at com.healthmarketscience.jackcess.impl.DatabaseImpl.create(DatabaseImpl.java:444)
What Database.FileFormat constant should I use for an MS Access 2000-2003 database, when creating the Database object?
It turns out that I needed to use Database.FileFormat.V2000.
After that, it was all plain sailing (although I still need to work out how to get the Long timestamp to convert correctly).

Netezza Batch Insert is very slow even in Batch execute mode

I am referring to this documentation. http://www-01.ibm.com/support/docview.wss?uid=swg21981328. As per the article if we use executeBatch method then inserts will be faster (The Netezza JDBC driver may detect a batch insert, and under the covers convert this to an external table load and external table load will be faster). I had to execute millions of insert statements and i am getting only a speed of 500 records per minute per connection max. Is there any better way to load data faster to netezza via jdbc connection? I am using spark and jdbc connection to insert the records.Why external table via loading is not happening even when i am executing in batches. Given below is the spark code i am using,
Dataset<String> insertQueryDataSet.foreachPartition( partition -> {
Connection conn = NetezzaConnector.getSingletonConnection(url, userName, pwd);
conn.setAutoCommit(false);
int commitBatchCount = 0;
int insertBatchCount = 0;
Statement statement = conn.createStatement();
//PreparedStatement preparedStmt = null;
while(partition.hasNext()){
insertBatchCount++;
//preparedStmt = conn.prepareStatement(partition.next());
statement.addBatch(partition.next());
//statement.addBatch(partition.next());
commitBatchCount++;
if(insertBatchCount % 10000 == 0){
LOGGER.info("Before executeBatch.");
int[] execCount = statement.executeBatch();
LOGGER.info("After execCount." + execCount.length);
LOGGER.info("Before commit.");
conn.commit();
LOGGER.info("After commit.");
}
}
//execute remaining statements
statement.executeBatch();
int[] execCount = statement.executeBatch();
LOGGER.info("After execCount." + execCount.length);
conn.commit();
conn.close();
});
I tried this approach(batch insert) but found very slow,
So I put all data in CSV & do external table load for each csv.
InsertReq="Insert into "+ tablename + " select * from external '"+ filepath + "' using (maxerrors 0, delimiter ',' unase 2000 encoding 'internal' remotesource 'jdbc' escapechar '\' )";
Jdbctemplate.execute(InsertReq);
Since I was using java so JDBC as source & note that csv file path is in single quotes .
Hope this helps.
If you find better than this approach, don't forget to post. :)

Resources