How to close refcursor in spring when using simplejdbccall - spring

I am using spring simpleJdbcCall to call oracle stored procedure and i am using oracle 11g .
I stumbled on a couple of posts which suggests there might be memory leak as the ref cursors are not properly closed by spring.
Is there anyway to explicitly close cursor while using spring simplejdbccall? or is increasing the oracle OPEN_CURSOR the only way out?.
I am planning to scale up my application to handle around one million transactions every hour .Any suggestions will be helpful.

Actually there is no such an issue with Spring JDBC. It closes all resources within finally after all execute. SimpleJdbcCall uses JdbcTemplate:
public <T> T execute(CallableStatementCreator csc, CallableStatementCallback<T> action)
throws DataAccessException {
try {
...
}
catch (SQLException ex) {
...
}
finally {
if (csc instanceof ParameterDisposer) {
((ParameterDisposer) csc).cleanupParameters();
}
JdbcUtils.closeStatement(cs);
DataSourceUtils.releaseConnection(con, getDataSource());
}
}
The same for ResultSet OUT parameters:
protected Map<String, Object> processResultSet(ResultSet rs, ResultSetSupportingSqlParameter param) throws SQLException {
....
finally {
JdbcUtils.closeResultSet(rs);
}
return returnedResults;
}
From other side I have a big experience with Spring JDBC and Oracle in high-loaded systems and want to say that we noticed enough open resources on Oracle with at peak loads, but they have been released properly after that.
Although we used JBOSS Pooled DataSource and its TransactionMaanger

I use CallableStatement directly and I can release statements and connections quickly and safely, try both methods and measure memory consumption, it worked perfectly for me to solve memory consumption and connection retention problems that proved many waiting and rejection of connections the applications.
try {
log.info("**** RepositoryPSostgres.getAllProducts ******** ");
Connection conn = jdbcTemplate.getDataSource().getConnection();
conn.setAutoCommit(false);
// Procedure call.
CallableStatement proc = conn.prepareCall("{? = call get_all_products() }");
proc.registerOutParameter(1, Types.OTHER);
proc.execute();
ResultSet results = (ResultSet) proc.getObject(1);
**proc.close();
proc.isClosed();
conn.close();**
ArrayList <Products> resp = new ArrayList <Products>();
while (results.next()) {
Products resp1 = new Products();
resp1.setId(results.getInt("id"));
resp1.setName((String) results.getString("name"));
resp1.setPrice((BigDecimal) results.getBigDecimal("price"));
resp.add(resp1);
log.info("***" + results.getInt("id") + "***** ");
log.info("***" + results.getString("name") + "***** ");
log.info("***" + results.getBigDecimal("price") + "***** ");
}
results.close();
return resp;
} catch (Exception e) {
e.printStackTrace();
log.error(new StringBuffer("Error en transaccion en saldo CashPooling : ").append(e.getLocalizedMessage()).toString());
return null;
}

Related

MariaDB Connector J : autoReconnect does not wok for Basic Faillover

From https://mariadb.com/kb/en/library/about-mariadb-connector-j/, for option autoReconnect, When this parameter enabled when a Failover and Load Balancing Mode is not in use, the connector will simply try to reconnect to its host after a failure. This is referred to as Basic Failover.
But the problem is that the reconnect does not work after server failure. The test code is as follows:
#Test
public void waitTimeoutResultSetTest() throws SQLException, InterruptedException {
try (Connection connection = setBlankConnection("&autoReconnect=true")) {
Statement stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery("SELECT 1");
assertTrue(rs.next());
stmt.execute("set session wait_timeout=1");
Thread.sleep(3000); // Wait for the server to kill the connection
try {
rs = stmt.executeQuery("show databases;");
assertTrue(rs.next());
System.out.println("position 1");
} catch (SQLException e) {
//normal exception
System.out.println("position 2");
}
}
}
With autoReconnect, I think the expected result is that it will get to position 1, but actually it will get to position 2, with exception that "Connection reset by peer: socket write error."
My question is that whether the basic failover does not work, or my test code is wrong? I cannot find other information from web, could you kindly give me some explanation if you know about it?

Using BulkInsert with OracleDB

I have some kind of promblem with BulkInsert on my OracleDB. I need to insert couple of thousand objects so I decided to use EF.BulkInsert.Oracle added by Nuget which is extension of EF6.BulkInsert for Oracle.
private IOracleDbContext _context;//Class property
//method body:
EF6.BulkInsert.ProviderFactory.Register<EF6.BulkInsert.Providers.OracleBulkInsertProvider>("BulkInsertProvider");
using (var context = (OracleDbContext)_context)
{
using (var dbContextTransaction = context.Database.BeginTransaction())
{
try
{
//Preparing list of objects
var opt = new EF6.BulkInsert.BulkInsertOptions();
opt.Connection = context.Database.Connection;
await context.BulkInsertAsync<ObjectType>(ObjectList,opt);
await context.SaveChangesAsync();
dbContextTransaction.Commit();
stopwatch.Stop();
}
catch (Exception ex)
{
dbContextTransaction.Rollback();
throw ex;
}
}
}
Without adding opt (BulkInsertOptions object) as parameter of BulkInsert it is trying to connect with SQLServer (which don't exist so I get connection failure). After add this BulkOptions with connection I get exception that connection is already part of transaction :/
Traditional way (_context.TableName.Add() ) of course works but It takes unacceptable amount of time.
Any idea what I did wrong here?
I found better way (BulkInsert still do not cooperate). I used Array Binding
mentioned here
It reduced insert time from ~6 minutes to ~1-1.5 seconds :D (7770 records)

JDBC statement pooling with DB2 does not have significant time difference

I'm using JDBC db2 driver, a.k.a. JT400 to connect to db2 server on Application System/400, a midrange computer system.
My goal is to insert into three Tables, from outside of IBM mainframe, which would be cloud instance(eg. Amazon WS).
To make the performance better
1) I am using already established connections to connect to db2.
(using org.apache.commons.dbcp.BasicDataSource or com.ibm.as400.access.AS400JDBCManagedConnectionPoolDataSource, both are working fine.)
public class AS400JDBCManagedConnectionPoolDataSource extends AS400JDBCManagedDataSource implements ConnectionPoolDataSource, Referenceable, Serializable {
}
public class AS400JDBCManagedDataSource extends ToolboxWrapper implements DataSource, Referenceable, Serializable, Cloneable {
}
2) I want to cache the insert into statements for all three tables, so that I don't have to send query every time and compile every time, which is expensive. I would instead just pass the parameters only. (Obviously I am doing this using JDBC prepared statements)
Based on an official IBM document Optimize Access to DB2 for i5/OS
from Java and WebSphere, page 17-20 - Enabling Extended Dynamic Support, it's possible to cache the statement with AS400JDBCManagedConnectionPoolDataSource.
BUT, the problem is the insert into queries are being compiled each time, which is taking 200ms * 3 queries = 600ms each time.
Example I'm using,
public class CustomerOrderEventHandler extends MultiEventHandler {
private static Logger logger = LogManager.getLogger(CustomerOrderEventHandler.class);
//private BasicDataSource establishedConnections = new BasicDataSource();
//private DB2SimpleDataSource nativeEstablishedConnections = new DB2SimpleDataSource();
private AS400JDBCManagedConnectionPoolDataSource dynamicEstablishedConnections =
new AS400JDBCManagedConnectionPoolDataSource();
private State3 orderState3;
private State2 orderState2;
private State1 orderState1;
public CustomerOrderEventHandler() throws SQLException {
dynamicEstablishedConnections.setServerName(State.server);
dynamicEstablishedConnections.setDatabaseName(State.DATABASE);
dynamicEstablishedConnections.setUser(State.user);
dynamicEstablishedConnections.setPassword(State.password);
dynamicEstablishedConnections.setSavePasswordWhenSerialized(true);
dynamicEstablishedConnections.setPrompt(false);
dynamicEstablishedConnections.setMinPoolSize(3);
dynamicEstablishedConnections.setInitialPoolSize(5);
dynamicEstablishedConnections.setMaxPoolSize(50);
dynamicEstablishedConnections.setExtendedDynamic(true);
Connection connection = dynamicEstablishedConnections.getConnection();
connection.close();
}
public void onEvent(CustomerOrder orderEvent){
long start = System.currentTimeMillis();
Connection dbConnection = null;
try {
dbConnection = dynamicEstablishedConnections.getConnection();
long connectionSetupTime = System.currentTimeMillis() - start;
state3 = new State3(dbConnection);
state2 = new State2(dbConnection);
state1 = new State1(dbConnection);
long initialisation = System.currentTimeMillis() - start - connectionSetupTime;
int[] state3Result = state3.apply(orderEvent);
int[] state2Result = state2.apply(orderEvent);
long state1Result = state1.apply(orderEvent);
dbConnection.commit();
logger.info("eventId="+ getEventId(orderEvent) +
",connectionSetupTime=" + connectionSetupTime +
",queryPreCompilation=" + initialisation +
",insertionOnlyTimeTaken=" +
(System.currentTimeMillis() - (start + connectionSetupTime + initialisation)) +
",insertionTotalTimeTaken=" + (System.currentTimeMillis() - start));
} catch (SQLException e) {
logger.error("Error updating the order states.", e);
if(dbConnection != null) {
try {
dbConnection.rollback();
} catch (SQLException e1) {
logger.error("Error rolling back the state.", e1);
}
}
throw new CustomerOrderEventHandlerRuntimeException("Error updating the customer order states.", e);
}
}
private Long getEventId(CustomerOrder order) {
return Long.valueOf(order.getMessageHeader().getCorrelationId());
}
}
And the States with insert commands look like below,
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
public class State2 extends State {
private static Logger logger = LogManager.getLogger(DetailState.class);
Connection connection;
PreparedStatement preparedStatement;
String detailsCompiledQuery = "INSERT INTO " + DATABASE + "." + getStateName() +
"(" + DetailState.EVENT_ID + ", " +
State2.ORDER_NUMBER + ", " +
State2.SKU_ID + ", " +
State2.SKU_ORDERED_QTY + ") VALUES(?, ?, ?, ?)";
public State2(Connection connection) throws SQLException {
this.connection = connection;
this.preparedStatement = this.connection.prepareStatement(detailsCompiledQuery); // this is taking ~200ms each time
this.preparedStatement.setPoolable(true); //might not be required, not sure
}
public int[] apply(CustomerOrder event) throws StateException {
event.getMessageBody().getDetails().forEach(detail -> {
try {
preparedStatement.setLong(1, getEventId(event));
preparedStatement.setString(2, getOrderNo(event));
preparedStatement.setInt(3, detail.getSkuId());
preparedStatement.setInt(4, detail.getQty());
preparedStatement.addBatch();
} catch (SQLException e) {
logger.error(e);
throw new StateException("Error setting up data", e);
}
});
long startedTime = System.currentTimeMillis();
int[] inserted = new int[0];
try {
inserted = preparedStatement.executeBatch();
} catch (SQLException e) {
throw new StateException("Error updating allocations data", e);
}
logger.info("eventId="+ getEventId(event) +
",state=details,insertionTimeTaken=" + (System.currentTimeMillis() - startedTime));
return inserted;
}
#Override
protected String getStateName() {
return properties.getProperty("state.order.details.name");
}
}
So the flow is each time an event is received(eg. CustomerOrder), it gets the establishedConnection and then asks the states to initialise their statements.
The metrics for timing look as below,
for the first event, it takes 580ms to create the preparedStatements for 3 tables.
{"timeMillis":1489982655836,"thread":"ScalaTest-run-running-CustomerOrderEventHandlerSpecs","level":"INFO","loggerName":"com.xyz.customerorder.events.handler.CustomerOrderEventHandler",
"message":"eventId=1489982654314,connectionSetupTime=1,queryPreCompilation=580,insertionOnlyTimeTaken=938,insertionTotalTimeTaken=1519","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":1,"threadPriority":5}
for the second event, takes 470ms to prepare the statements for 3 tables, which is less than the first event but just < 100ms, I assume it to be drastically less, as it should not even make it to compilation.
{"timeMillis":1489982667243,"thread":"ScalaTest-run-running-PurchaseOrderEventHandlerSpecs","level":"INFO","loggerName":"com.xyz.customerorder.events.handler.CustomerOrderEventHandler",
"message":"eventId=1489982665456,connectionSetupTime=0,queryPreCompilation=417,insertionOnlyTimeTaken=1363,insertionTotalTimeTaken=1780","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":1,"threadPriority":5}
What I'm thinking is since I'm closing preparedStatement for that particular connection, it does not even exist for new connection. If thats the case whats the point of having statement caching at all in multi-threaded environment.
The documentation has similar example, where its making transactions inside the same connection which is not the case for me, as I need to have multiple connections at the same time.
Questions
Primary
Q1) Is DB2 JDBC driver caching the statements at all, between multiple connections? Because I don't see much difference while preparing the statement. (see example, first one takes ~600ms, second one takes ~500ms)
References
ODP = Open Data Path
SQL packages
SQL packages are permanent objects used to store information related
to prepared SQL statements. They can be used by the IBM iSeries Access
for the IBM Toolbox for
Java JDBC driver. They are also used by applications which use the
QSQPRCED (SQL Process Extended Dynamic) API interface.
In the case JDBC, the existence of the SQL package is
checked when the client application issues the first prepare of a SQL
Statement. If the package does not exist, it is created at that time
(even though it may not yet contain any SQL statements)
Tomcat jdbc connection pool configuration - DB2 on iSeries(AS400)
IBM Data Server Driver for JDBC and SQLJ statement caching
A couple of important things to note regarding statement caching:
Because Statement objects are child objects of a given Connection, once the Connection is closed all child objects (e.g. all Statement objects) must also be closed.
It is not possible to associate a statement from one connection with a different connection.
Statement pooling may or may not be done be by a given JDBC driver. Statement pooling may also be performed by a connection management layer (i.e. application server)
Per JDBC spec, default value for Statement.isPoolable() == false and PreparedStatement.isPoolable() == true, however this flag is only a hint to the JDBC driver. There is no guarantee from the spec that statement pooling will occur.
First off, I am not sure if the JT400 driver does statement caching. The document you referenced in your question comment, Optimize Access to DB2 for i5/OS from Java and WebSphere, is specific to using the JT400 JDBC driver with WebSphere application server, and on slide #3 it indicates that statement caching comes from the WebSphere connection management layer, not the native JDBC driver layer. Given that, I'm going to assume that the JT400 JDBC driver doesn't support statement caching on its own.
So at this point you are probably going to want to plug into some sort of app server (unless you want to implement statement caching on your own, which is sort of re-inventing the wheel). I know for sure that both WebSphere Application Server products (traditional and Liberty) support statement caching for any JDBC driver.
For WebSphere Liberty (the newer product), the data source config is the following:
<dataSource jndiName="jdbc/myDS" statementCacheSize="10">
<jdbcDriver libraryRef="DB2iToolboxLib"/>
<properties.db2.i.toolbox databaseName="YOURDB" serverName="localhost"/>
</dataSource>
<library id="DB2iToolboxLib">
<fileset dir="/path/to/jdbc/driver/dir" includes="jt400.jar"/>
</library>
The key bit being the statementCacheSize attribute of <dataSource>, which has a default value of 10.
(Disclaimer, I'm a WebSphere dev, so I'm going to talk about what I know)
First off, the IBM i Java documentation is here: IBM Toolbox for Java
Secondly, I don't see where you are setting the "extended dynamic" property to true which provides
a mechanism for caching dynamic SQL statements on the server. The first
time a particular SQL statement is prepared, it is stored in a SQL
package on the server. If the package does not exist, it is
automatically created. On subsequent prepares of the same SQL
statement, the server can skip a significant part of the processing by
using information stored in the SQL package. If this is set to "true",
then a package name must be set using the "package" property.
I think you're missing some steps in using the managed pool...here's the first example in the IBM docs
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.sql.DataSource;
import com.ibm.as400.access.AS400JDBCManagedConnectionPoolDataSource;
import com.ibm.as400.access.AS400JDBCManagedDataSource;
public class TestJDBCConnPoolSnippet
{
void test()
{
AS400JDBCManagedConnectionPoolDataSource cpds0 = new AS400JDBCManagedConnectionPoolDataSource();
// Set general datasource properties. Note that both connection pool datasource (CPDS) and managed
// datasource (MDS) have these properties, and they might have different values.
cpds0.setServerName(host);
cpds0.setDatabaseName(host);//iasp can be here
cpds0.setUser(userid);
cpds0.setPassword(password);
cpds0.setSavePasswordWhenSerialized(true);
// Set connection pooling-specific properties.
cpds0.setInitialPoolSize(initialPoolSize_);
cpds0.setMinPoolSize(minPoolSize_);
cpds0.setMaxPoolSize(maxPoolSize_);
cpds0.setMaxLifetime((int)(maxLifetime_/1000)); // convert to seconds
cpds0.setMaxIdleTime((int)(maxIdleTime_/1000)); // convert to seconds
cpds0.setPropertyCycle((int)(propertyCycle_/1000)); // convert to seconds
//cpds0.setReuseConnections(false); // do not re-use connections
// Set the initial context factory to use.
System.setProperty(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.fscontext.RefFSContextFactory");
// Get the JNDI Initial Context.
Context ctx = new InitialContext();
// Note: The following is an alternative way to set context properties locally:
// Properties env = new Properties();
// env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.fscontext.RefFSContextFactory");
// Context ctx = new InitialContext(env);
ctx.rebind("mydatasource", cpds0); // We can now do lookups on cpds, by the name "mydatasource".
// Create a standard DataSource object that references it.
AS400JDBCManagedDataSource mds0 = new AS400JDBCManagedDataSource();
mds0.setDescription("DataSource supporting connection pooling");
mds0.setDataSourceName("mydatasource");
ctx.rebind("ConnectionPoolingDataSource", mds0);
DataSource dataSource_ = (DataSource)ctx.lookup("ConnectionPoolingDataSource");
AS400JDBCManagedDataSource mds_ = (AS400JDBCManagedDataSource)dataSource_;
boolean isHealthy = mds_.checkPoolHealth(false); //check pool health
Connection c = dataSource_.getConnection();
}
}

Configure Datasource with timezone in jboss/weblogic server

I am going to handle multiple time zone in my web application by resetting database session time zone every time getting DB connection.
The code looks like:
public Connection getConnection(GmDNSNamesEnum enDNSName) throws AppError
{
...
DataSource ds = (DataSource) ic.lookup(enDNSName.getDNSName());
conn = ds.getConnection();
conn.setAutoCommit(false);
setTimeZone(strTimeZone);
...
return conn;
}
private void setTimeZone(String strTimeZone){
Statement stmt = null;
if(conn != null) {
try{
stmt = conn.createStatement();
stmt.execute("alter session set time_zone=\'" + strTimeZone+"\'");
} catch (Exception e)
{
String strErrorMsg = GmCommonClass.getExceptionStackTrace(e);
log.error("Exception e "+strErrorMsg);
throw new AppError(e);
}
}
}
Is there any alternate way to set database session time zone?
now, I am looking for configure datasource with different timezone in jboss/weblogic server and use appropriate datasource specific to user timezone instead of every time reset the session time zone by executing alter session script.
thanks in advance
Wow ok.
Perhaps you could try persisting the dates and times into timezone-aware database columns instead of contorting the data source each time you connect? The convention is to store dates in UTC time, then do timezone conversion in the presentation layer (or at least post-retrieval).

Oracle MAF Error "Unable to read DataControl Usages, on loadDataControl for id:"

I have created a simple SOAP web service for a remote DB which is working fine in WebLogic 12c server and also in Jdeveloper HTTP analyzer. When I am using the same web service in the code below, I am getting the following error (I have included the System.out.println outputs in ref. to the flow of the code) I am using Jdeveloper 12C MAF 2.0.0.0.41 on Mac OSx 10.9.
Chk #0
Processing row# 1
Chk #1
Chk #2
[SEVERE - oracle.adfmf.framework - AmxBindingContext - loadDataControlById] Unable to read DataControl Usages, on loadDataControl for id: WLFNewActWS.
[SEVERE - oracle.adfmf.framework - SynchronizationDC - syncDataFromOfflineToOnline] [Ljava.lang.StackTraceElement;#467c53d3
public void syncDataFromOfflineToOnline() {
Trace.log(Utility.FrameworkLogger, Level.INFO, this.getClass(), "syncDataFromOfflineToOnline",
"Executing syncDataFromOfflineToOnline Method");
try {
Connection conn = DBConnectionFactory.getConnection();
conn.setAutoCommit(false);
String select = "SELECT * FROM DEPARTMENTS";
PreparedStatement pStmt = conn.prepareStatement(select);
ResultSet rs = pStmt.executeQuery();
System.out.println("Chk #0");
int rowcount = 0;
while (rs.next()) {
rowcount++;
System.out.println("Processing row# " + rowcount);
List namesList = new ArrayList(1);
List paramsList = new ArrayList(1);
List typesList = new ArrayList(1);
Department dept = new Department();
dept.setDeptId(rs.getInt("DEPARTMENT_ID"));
dept.setDeptName(rs.getString("DEPARTMENT_NAME"));
dept.setMgrId(rs.getInt("MANAGER_ID"));
dept.setLocId(rs.getInt("LOCATION_ID"));
System.out.println("Chk #1");
GenericType gtDept =
GenericTypeBeanSerializationHelper.toGenericType("WLFNewActWS.Types.persistDepartments.arg0", dept);
System.out.println("Chk #2");
namesList.add("arg0");
paramsList.add(gtDept);
typesList.add(GenericType.class);
AdfmfJavaUtilities.invokeDataControlMethod("WLFNewActWS", null, "persistDepartments", namesList,
paramsList, typesList);
System.out.println("Chk #3");
}
} catch (SQLException e) {
Trace.log(Utility.FrameworkLogger, Level.SEVERE, this.getClass(), "syncDataFromOfflineToOnline",
e.getMessage());
} catch (Exception e) {
Trace.log(Utility.FrameworkLogger, Level.SEVERE, this.getClass(), "syncDataFromOfflineToOnline",
e.getStackTrace());
}
}
This may not be the answer to this question, but it may be the answer for someone searching for "Unable to read DataControl Usages, on loadDataControl for id"
In my case it is with Oracle MAF, but this also applies to Oracle ADF.
Your web service needs to be in your DataBindings.cpx file.
Something like:
<dataControlUsages>
<dc id="WLFNewActWS " path="mobile.WLFNewActWS "/>
</dataControlUsages>
I cheat and drag something from the WS into a one of pages and have jdeveloper add the record in DataBindings.cpx for me.
This occurs of the binding issue. Please create a reference/binding to 'WLFNewActWS' in your amx page. It will solve your problem...!

Resources