jdbc resultSet slow performance - jdbc

hiii,
i am calling a stored procedure in a java class in my web application, there are some reports that are taking to much time to render the jsp/HTML ,i have also chercked with stored procedure query its taking only 2 seconds to execute in oracle browser. i have checked my SP calling code where i found that my result set fetch time is very less,but when in trying to iterate result set with While(rst.next) it will almost take 3 mins to print SOP inside the while loop,i doubt there must be issue with ret.next(),
my code is as follows,
Connection connection = null;
CallableStatement stmt = null;
ResultSet rst = null ;
connection = DBConnector.getConnection();
stmt = connection.prepareCall("{call MIS_GSGR_ASON.MIS_DIVNETSALE_ASON(?,?,?,?,?,?,?,?,?)}");
stmt.setString(1,START_DT);
stmt.setString(2,END_DT);
stmt.setString(3,DIVISION);
stmt.setString(4,LOC_ID);
stmt.setInt(5,USER_GRP);
stmt.setInt(6,FIELD_ID);
stmt.setInt(7,Integer.parseInt(PERIOD_ID));
stmt.setString(8,zone);
stmt.registerOutParameter(9+INC,OracleTypes.CURSOR);
stmt.execute();
rst = (ResultSet) stmt.getObject(9+INC);
System.out.println("Got resultset . . . .");
data = new ArrayList<MainActionAll>();
while(rst.next()){
System.out.println("In loop");}
any help will be highly appriciated,please help me out
thanks ,amol

Maybe you run your query via slow network and if there is many rows returned it works very slow?
Add some time checking and instead of printing to console simply count records:
...
...
...
stmt.registerOutParameter(9+INC,OracleTypes.CURSOR);
long t0, t1, t2, t3, t4, t5;
t0 = System.currentTimeMillis();
stmt.execute();
t1 = System.currentTimeMillis();
rst = (ResultSet) stmt.getObject(9+INC);
t2 = System.currentTimeMillis();
System.out.println("Got resultset . . . .");
data = new ArrayList<MainActionAll>();
t3 = System.currentTimeMillis();
t4 = t3;
t5 = t3;
int rec_cnt = 0;
if (rst.next())
{
++rec_cnt;
t4 = System.currentTimeMillis();
while (rst.next())
++rec_cnt;
t5 = System.currentTimeMillis();
}
System.out.println("execute: " + (t1 - t0));
System.out.println("stmt.getObject: " + (t2 - t1));
System.out.println("Array: " + (t3 - t2));
System.out.println("1st next: " + (t4 - t3));
System.out.println("loop: " + (t5 - t4));
System.out.println("rec cnt: " + (rec_cnt));
Try to execute it as near database server as possible to reduce network transfers (localhost preferred). Edit your question with those times.

Related

How to set Hive configuration property hive.exec.dynamic.partition from Java code

I have made a java script that will connect to hive using Hiveserver2 and will create table and manage tables, for simple create, drop, insert data works fine.
I want to create external table with partition, for this I need to change the value for the following hive property,
hive.exec.dynamic.partition = true
hive.exec.dynamic.partition.mode = nonstrict
In hive cli I can do it using SET and the property name, but how can this be done in java code.
Here is my Java code:
public class HiveJdbcClient {
private static String strDriverName = "org.apache.hive.jdbc.HiveDriver";
public static void main(String[] args) throws SQLException {
try{
Class.forName(strDriverName);
} catch (ClassNotFoundException e){
e.printStackTrace();
System.out.println("No class found");
System.exit(1);
}
Connection con = DriverManager.getConnection("jdbc:hive2://172.11.1.11:10000/default","root","root123");
Statement stmt = con.createStatement();
String strTableName = "testtable";
//stmt.execute("drop table " + strTableName);
//creating staging table that will load the data to partition data
String strStagingTableSql = "create table if not exists "+strTableName+"_staging "+ " (SEQUENCE_NO DECIMAL, DATE_KEY INT, ACTIVITY_TIME_KEY INT, Ds_KEY INT, Ds_VALUE DECIMAL, TL_DATE_KEY INT) ROW FORMAT DELIMITED FIELDS TERIMANTED BY '~'";
String strMainTableSql = "create external table if not exists "+strTableName+" (SEQUENCE_NO DECIMAL, ACTIVITY_TIME_KEY INT, Ds_KEY INT, Ds_VALUE DECIMAL, TL_DATE_KEY INT) PARTITIONED BY (DATE_KEY INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '~' LOCATION '/informatica/dwh/teradata/testtable'";
String strCreateSql = "create external table if not exists "+ strTableName + " (key int, value string) row format delimited fields terminated by ','";
boolean res = stmt.execute(strCreateSql);
//show tables
String sql = "show tables '" + strTableName + "'";
ResultSet res1 = stmt.executeQuery(sql);
if (res1.next()){
System.out.println(res1.getString(1));
}
sql = "describe "+ strTableName;
System.out.println("Running: "+ sql);
res1 = stmt.executeQuery(sql);
while (res1.next()){
System.out.println(res1.getString(1) + "\t" + res1.getString(2));
}
// load data into table
// NOTE: filepath has to be local to the hive server
// NOTE: /tmp/a.txt is a ctrl-A separated file with two fields per line
String strFilepath = "/informatica/testing_hive_client_java.txt";
sql = "load data inpath '" + strFilepath + "' into table " + strTableName;
System.out.println("Running: " + sql);
res = stmt.execute(sql);
sql = "select count(1) from "+ strTableName;
System.out.println("Running: "+ sql);
res1 = stmt.executeQuery(sql);
while(res1.next()){
System.out.println(res1.getString(1));
}
}// end of main
}// end of class
Experts please pour in your thoughts.
I was able to solve my problem by following code.
boolean resHivePropertyTest = stmt
.execute("SET hive.exec.dynamic.partition = true");
resHivePropertyTest = stmt
.execute("SET hive.exec.dynamic.partition.mode = nonstrict");
As the code is JDBC client code , so the execute will just go and execute this in hive and so that worked for me.

MongoDB Native Query vs C# LINQ Performance

I am using the following two options, the Mongo C# driver seems to be taking more time. I'm using StopWatch to calculate the timings.
Case 1: Native Mongo QueryDocument (takes 0.0011 ms to return data)
string querytext = #"{schemas:{$elemMatch:{name: " + n + ",code : " + c + "} }},{schemas:{$elemMatch:{code :" + c1 + "}}}";
string printQueryname = "Query: " + querytext;
BsonDocument query1 = MongoDB.Bson.Serialization.BsonSerializer.Deserialize<BsonDocument>(querytext);
QueryDocument queryDoc1 = new QueryDocument(query1);
var queryResponse = collection.FindAs<BsonDocument>(queryDoc1);
Case 2: Mongo C# Driver (takes more than 3.2 ms to return data)
Schema _result = new Schema();
_result = (from c in _coll.AsQueryable<Schema>()
where c.schemas.Any(s => s.code.Equals(c) && s.name.Equals(n) ) &&
c.schemas.Any(s => s.code.Equals(c1))
select c).FirstOrDefault();
Any thoughts ? Anything wrong here ?

Drools - Creating a KnowlegeBuilder Faster ?

I've a use case where I need to create a rule dynamically for a single time use only. The rules are dynamic and in the worse case can be different for each iteration/request. Hence, I cannot store them.
My current understanding of Drools is that you need to create a KnowledgeBuilder and then add the rules.
I'm creating the KnowledgeBase as follows.
private KnowledgeBase readKnowledgeBase() throws Exception {
long t1 = System.currentTimeMillis();
KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
long t11 = System.currentTimeMillis();
kbuilder.add( org.drools.io.ResourceFactory.newByteArrayResource(getRule()), ResourceType.DRL);
long t2 = System.currentTimeMillis();
KnowledgeBuilderErrors errors = kbuilder.getErrors();
if (errors.size() > 0) {
for (KnowledgeBuilderError error: errors) {
System.err.println(error);
}
throw new IllegalArgumentException("Could not parse knowledge.");
}
long t3 = System.currentTimeMillis();
KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
long t4 = System.currentTimeMillis();
kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());
long t5 = System.currentTimeMillis();
Collection<KnowledgePackage> kpackages = kbase.getKnowledgePackages();
for (KnowledgePackage knowledgePackage : kpackages) {
System.out.println("Package -------- " + knowledgePackage.getName());
Collection<Rule> rules = knowledgePackage.getRules();
for (Rule rule : rules) {
System.out.println("****" + rule.getName());
}
}
long t6 = System.currentTimeMillis();
System.out.println( (t11-t1) + " " + (t2-t11) + " " + ( t3-t2) + " " + ( t4-t3) + " " + ( t5-t4) + " " + ( t6-t5)+ " " );
return kbase;
}
Based on the timing logs the code take majority (more than 80%) of time in only these two operations. Is there a way to make it faster? OR is there a way of achieving my use case ?
KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add( org.drools.io.ResourceFactory.newByteArrayResource(getRule()), ResourceType.DRL);
The following call will actually compile your code, i.e., it calls the JDT java compiler:
kbuilder.add( org.drools.io.ResourceFactory.newByteArrayResource(getRule()), ResourceType.DRL);
There is no way of avoiding that unless you use internal APIs, that are in fact very complicated to use.
There is an intermediate step that is to generate the canonical model using the descriptor builder API. It will save you a few cycles, but not much as the heaviest operation is the java compilation. Examples of the use of the Descr Builder API:
https://github.com/droolsjbpm/drools/blob/master/drools-compiler/src/test/java/org/drools/lang/api/DescrBuilderTest.java

Storing .NET double value in Oracle DB

I'm using ODP.NET to access Oracle DB from C# .NET.
Please see following code:
OracleConnection con = new OracleConnection();
con.ConnectionString = "User Id=user;Password=pass;Data Source=localhost/orcl";
con.Open();
/* create table */
DbCommand command = con.CreateCommand();
command.CommandType = CommandType.Text;
try
{
command.CommandText = "DROP TABLE TEST";
command.ExecuteNonQuery();
}
catch
{
}
//command.CommandText = "CREATE TABLE TEST (VALUE BINARY_DOUBLE)";
command.CommandText = "CREATE TABLE TEST (VALUE FLOAT(126))";
command.ExecuteNonQuery();
/* now insert something */
double val = 0.8414709848078965;
command.CommandText = "INSERT INTO TEST VALUES (" + val.ToString(System.Globalization.CultureInfo.InvariantCulture) + ")";
command.ExecuteNonQuery();
/* and now read inserted value */
command.CommandText = "SELECT * FROM TEST";
DbDataReader reader = command.ExecuteReader();
reader.Read();
double res = (double) (decimal)reader[0];
Console.WriteLine("Inserted " + val + " selected " + res);
The output from this is always:
Inserted 0,841470984807897 selected 0,841470984807897
But looking at variable values under debugger
val == 0.8414709848078965
res == 0,841470984807897
Why res is rounded up?
I looked into DB and there is stored rounded-up value.
On the other hand I used Oracle SQL Developer to modify this value, and I'm able to store 0.8414709848078965 in database?
I tried types NUMBER, FLOAT(126), BINARY_DOUBLE... always the same result.
Why there is a problem using ODP.NET?
OK, I have found that it works if parameter type is OracleDbType.BinaryDouble. But it causes my code to be dependent of ODP.NET. I wanted to use ADO.NET types (DbType) to achieve my code independency.
Oracle actually has a higher precision for it's numbers than .net!
I tried this in straight Oracle and it works fine, I recommend changing to use a param
e.g.
-- CREATE TABLE TEST (VALUE NUMBER(38,38)); (initial test)
INSERT INTO TEST VALUES (0.8414709848078965);
SELECT * FROM TEST;
VALUE
----------------------
0.8414709848078965
(recommendation)
OracleParameter param = cmd.CreateParameter();
param.ParameterName = "NUMBERVALUE";
param.Direction = ParameterDirection.Input;
param.OracleDbType = OracleDbType.Decimal;
param.Value = "0.8414709848078965";
command.Parameters.Add(param);

Execute sql statement via JDBC with CLOB binding

I have the following query (column log is of type CLOB):
UPDATE table SET log=? where id=?
The query above works fine when using the setAsciiStream method to put a value longer than 4000 characters into the log column.
But instead of replacing the value, I want to append it, hence my query looks like this:
UPDATE table SET log=log||?||chr(10) where id=?
The above query DOES NOT work any more and I get the following error:
java.sql.SQLException: ORA-01461: can bind a LONG value only for insert into a LONG column
It looks to me like you have to use a PL/SQL block to do what you want. The following works for me, assuming there's an entry with id 1:
import oracle.jdbc.OracleDriver;
import java.sql.*;
import java.io.ByteArrayInputStream;
public class JDBCTest {
// How much test data to generate.
public static final int SIZE = 8192;
public static void main(String[] args) throws Exception {
// Generate some test data.
byte[] data = new byte[SIZE];
for (int i = 0; i < SIZE; ++i) {
data[i] = (byte) (64 + (i % 32));
}
ByteArrayInputStream stream = new ByteArrayInputStream(data);
DriverManager.registerDriver(new OracleDriver());
Connection c = DriverManager.getConnection(
"jdbc:oracle:thin:#some_database", "user", "password");
String sql =
"DECLARE\n" +
" l_line CLOB;\n" +
"BEGIN\n" +
" l_line := ?;\n" +
" UPDATE table SET log = log || l_line || CHR(10) WHERE id = ?;\n" +
"END;\n";
PreparedStatement stmt = c.prepareStatement(sql);
stmt.setAsciiStream(1, stream, SIZE);
stmt.setInt(2, 1);
stmt.execute();
stmt.close();
c.commit();
c.close();
}
}
BLOBs are not mutable from SQL (well, besides setting them to NULL), so to append, you would have to download the blob first, concatenate locally, and upload the result again.
The usual solution is to write several records to the database with a common key and a sequence which tells the DB how to order the rows.

Resources