I had read several articles on obtaining Connection using Spring DataSource.
But in our Company Setup, connection object is obtained through already configured environment. Following the sample code:
String pool = PropertyFileReader.getPropertyValue("database.properties", "development.connectionPool");
Connection connection = RequestUtils.getConnection(pool);
Hence, After reading this tutorial
I am confused on using JDBCTemplate using connection object from above code.
I believe JdbcTemplate is not designed to work against a Connection as what you expected. As a workaround, if you are fine to create a separate JdbcTemplate for each connection you created, you may wrap your connection in a thin wrapper of DataSource, and feed it to JdbcTemplate.
I think it should work but I haven't tried it anyway...
class SingleConnectionDataSource implements DataSource {
private Connection connection;
public SingleConnectionDataSource(Connection connection) {
this.connection = connection;
}
public Connection getConnection() {
return this.connection;
}
public Connection getConnection(String username, String password) {
return this.connection;
}
}
// at the place you want to use JdbcTemplate
Connection conn = blablabla; // your own way to get it
JdbcTemplate jdbcTemplate = new JdbcTemplate(new SingleConnectionDataSource(conn));
Actually, Spring already provided SingleConnectionDataSource implementation (have seen in version 4.1.7).
It is even allows you to supress connection closing by template.
Related
I have a Rest service and when it gets it has to do some insertion and updation to almost 25 database. So when I tried like the below code, it was working in my localhost but when I deploy to my staging server I was getting FATAL: too many connections for role "user123"
List<String> databaseUrls = null;
databaseUrls.forEach( databaseUrl -> {
DataSource dataSource = DataSourceBuilder.create()
.driverClassName("org.postgresql.Driver")
.url(databaseUrl)
.username("user123")
.password("some-password")
.build();
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.update("Some...Update...Query");
});
As per my understanding DataSource need not to be closed because it is never opened.
Note:
A DataSource implementation need not be closed, because it is never
“opened”. A DataSource is not a resource, is not connected to the
database, so it is not holding networking connections nor resources on
the database server. A DataSource is simply information needed when
making a connection to the database, with the database server's
network name or address, the user name, user password, and various
options you want specified when a connection is eventually made.
Can someone tell why I am getting this issue
The problem is in DataSourceBuilder, it actually creates of the connection pools which spawns some number of connections and keeps them running:
private static final String[] DATA_SOURCE_TYPE_NAMES = new String[] {
"org.apache.tomcat.jdbc.pool.DataSource",
"com.zaxxer.hikari.HikariDataSource",
"org.apache.commons.dbcp.BasicDataSource" };
Javadoc says:
/**
* Convenience class for building a {#link DataSource} with common implementations and
* properties. If Tomcat, HikariCP or Commons DBCP are on the classpath one of them will
* be selected (in that order with Tomcat first). In the interest of a uniform interface,
* and so that there can be a fallback to an embedded database if one can be detected on
* the classpath, only a small set of common configuration properties are supported. To
* inject additional properties into the result you can downcast it, or use
* <code>#ConfigurationProperties</code>.
*/
Try to use e.g. SingleConnectionDataSource, then your problem will gone:
List<String> databaseUrls = null;
Class.forName("org.postgresql.Driver");
databaseUrls.forEach( databaseUrl -> {
SingleConnectionDataSource dataSource;
try {
dataSource = new SingleConnectionDataSource(
databaseUrl, "user123", "some-password", true /*suppressClose*/);
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.update("Some...Update...Query");
} catch (Exception e) {
log.error("Failed to run queries for {}", databaseUrl, e);
} finally {
// release resources
if (dataSource != null) {
dataSource.destroy();
}
}
});
First thing it is very bad architecture decision to have single application managing 50 database. Anyway instead of creating DataSource in for loop, you should make use of Factory Design pattern to create DataSource for each DB. You should add some connection pooling mechanism to your system . HijariCP and TomcatPool are most widely used. Analyse logs of failure thread for any further issues.
I have a Java JDBC application, after 100 queries of Select Jetty server crashed and return below error:
ERROR com.zaxxer.hikari.pool.HikariPool - dev-server - Exception during pool initialization.
java.sql.SQLNonTransientConnectionException: Too many connections
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:526)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:513)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:115)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:1606)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:633)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:347)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:219)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:443)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:514)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:111)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
at com.jrg.platform.commons.hikari.HikariDataSourceSupport.getDataSource(HikariDataSourceSupport.java:70)
at com.jrg.platform.commons.hikari.HikariDataSourceSupport.getConnection(HikariDataSourceSupport.java:82)
at com.jrg.platform.services.internal.pcms.data.campaign.CampaignAlertDaoImpl.getCampaignAlerts(CampaignAlertDaoImpl.java:40)
at com.jrg.platform.services.internal.pcms.service.CampaignAlertServiceImpl.campaignAlerts(CampaignAlertServiceImpl.java:43)
at com.jrg.platform.services.internal.pcms.resource.CampaignAlertResource.getCampaignAlerts(CampaignAlertResource.java:52)
at com.jrg.platform.services.internal.pcms.resource.CampaignAlertResource_$$_jvstf5a_6._d5getCampaignAlerts(CampaignAlertResource_$$_jvstf5a_6.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Unable to find the solution. I am just getting the connection from HikariDataSourceSupport and performing the query.
the sample code of getting connection is given below:
default HikariDataSource getDataSource(E e, boolean readOnly) throws IOException {
String key = this.getKey(e, readOnly);
Map<String, HikariDataSource> sources = this.getDataSources();
if(!sources.containsKey(key)) {
synchronized(this.getMonitor()) {
if(!sources.containsKey(key)) {
if(logger.isDebugEnabled()) {
logger.debug("creating new DataSource for {}", key);
}
Config config = this.getConfig(e, readOnly);
if(!config.hasPathOrNull("jdbcUrl")) {
throw new EnvironmentNotConfigured(e, readOnly);
}
sources.put(key, new HikariDataSource(new HikariConfig(ConfigUtils.asProperties(config))));
}
}
}
return (HikariDataSource)sources.get(key);
}
default Connection getConnection(E env, boolean readOnly) throws SQLException, IOException {
return this.getDataSource(env, readOnly).getConnection();
}
There was problem in configuration HikariCP. When request generated and try to get connection, the create connection method was creating the new datasource because the last state was destroy due to view scope of the bean.
now i have converted that class into to singalton and created the implementation of that interface to inject in the code. it is saving the state of previous data source connection. Now it is working perfect with minimum idle connections.
the code can be seen here:
APP.java
bind(HikariLucktasticDataSourceSupportImpl.class)
.to(HikariDataSourceSupport.class)
.named("hdsSupport")
.in(Singleton.class);
in DAO Layer:
#Inject
#Named("hdsSupport")
private HikariDataSourceSupport hdsSupport;
I have an application that runs through multiple databases and for each database runs select query on all tables and dumps it to hadoop.
My design is to create one datasource connection at a time and use the connection pool obtained to run select queries in multiple threads. Once done for this datasource, close the connection and create new one.
Here is the Async code
#Component
public class MySampleService {
private final static Logger LOGGER = Logger
.getLogger(MySampleService.class);
#Async
public Future<String> callAsync(JdbcTemplate template, String query) throws InterruptedException {
try {
jdbcTemplate.query(query);
//process the results
return new AsyncResult<String>("success");
}
catch (Exception ex){
return new AsyncResult<String>("failed");
}
}
Here is the caller
public String taskExecutor() throws InterruptedException, ExecutionException {
Future<String> asyncResult1 = mySampleService.callAsync(jdbcTemplate,query1);
Future<String> asyncResult2 = mySampleService.callAsync(jdbcTemplate,query2);
Future<String> asyncResult3 = mySampleService.callAsync(jdbcTemplate,query3);
Future<String> asyncResult4 = mySampleService.callAsync(jdbcTemplate,query4);
LOGGER.info(asyncResult1.get());
LOGGER.info(asyncResult2.get());
LOGGER.info(asyncResult3.get());
LOGGER.info( asyncResult4.get());
//now all threads finished, close the connection
jdbcTemplate.getConnection().close();
}
I am wondering if this is a right way to do it or do any exiting/optimized solution that out of box I am missing. I can't use spring-data-jpa since my queries are complex.
Thanks
Spring Boot docs:
Production database connections can also be auto-configured using a
pooling DataSource. Here’s the algorithm for choosing a specific
implementation:
We prefer the Tomcat pooling DataSource for its performance and concurrency, so if that is available we always choose it.
Otherwise, if HikariCP is available we will use it.
If neither the Tomcat pooling datasource nor HikariCP are available and if Commons DBCP is available we will use it, but we
don’t recommend it in production.
Lastly, if Commons DBCP2 is available we will use it.
If you use the spring-boot-starter-jdbc or
spring-boot-starter-data-jpa ‘starters’ you will automatically get a
dependency to tomcat-jdbc.
So you should be provided with sensible defaults.
Currently I'm setting autocommit to false in spring through adding a property to a datasource bean id like below :
<property name="defaultAutoCommit" value="false" />
But i need to add it specifically in a single java method before executing my procedure.
I used the below code snippet.
getJdbcTemplate().getDataSource().getConnection().setAutoCommit(false);
But the above line was not setting autocommit to false?
Am i missing anything ?
or any alternative to set autocommit in a specific java method by spring
Thanks
The problem is that you are setting autocommit on a Connection, but JdbcTemplate doesn't remember that Connection; instead, it gets a new Connection for each operation, and that might or might not be the same Connection instance, depending on your DataSource implementation. Since defaultAutoCommit is not a property on DataSource, you have two options:
Assuming your concrete datasource has a setter for defaultAutoCommit (for example, org.apache.commons.dbcp.BasicDataSource), cast the DataSource to your concrete implementation. Of course this means that you can no longer change your DataSource in your Spring configuration, which defeats the purpose of dependency injection.
((BasicDataSource)getJdbcTemplate().getDataSource()).setDefaultAutoCommit(false);
Set the DataSource to a wrapper implementation that sets AutoCommit to false each time you fetch a connection.
final DataSource ds = getJdbcTemplate().getDataSource();
getJdbcTemplate().setDataSource(new DataSource(){
// You'll need to implement all the methods, simply delegating to ds
#Override
public Connection getConnection() throws SQLException {
Connection c = ds.getConnection();
c.setAutoCommit(false);
return c;
}
});
You need to get the current connection. e.g.
Connection conn = DataSourceUtils.getConnection(jdbcTemplate.getDataSource());
try {
conn.setAutoCommit(false);
/**
* Your Code
*/
conn.commit();
} catch (SQLException e) {
conn.rollback();
e.printStackTrace();
}
I'm posting this because I was looking for it everywhere: I used configuration property in Spring boot to achieve setting the default autocommit mode with:
spring.datasource.hikari.auto-commit: false
Spring Boot 2.4.x Doc for Hikari
You will have to do for each statement that the jdbcTemplate executes. Because for each jdbcTemplate.execute() etc it gets a new connection from the Datasource's connection pool. So you will have to set it for the connection that the connection the jdbcTemplate uses for that query. So you will have to do something like
jdbcTemplate.execute("<your sql query", new PreparedStatementCallback<Integer>(){
#Override
public Integer doInPreparedStatement(PreparedStatement stmt) throws SQLException, DataAccessException
{
Connection cxn = stmt.getConnection();
// set autocommit for that cxn object to false
cxn.setAutoCommit(false);
// set parameters etc in the stmt
....
....
cxn.commit();
// restore autocommit to true for that cxn object. because if the same object is obtained from the CxnPool later, autocommit will be false
cxn.setAutoCommit(true);
return 0;
}
});
Hope this helps
after 5 years still a valid question, i resolved my issue in this way :
set a connection with connection.setAutoCommit(false);
create a jbc template with that connection;
do your work and commit.
Connection connection = dataSource.getConnection();
connection.setAutoCommit(false);
JdbcTemplate jdbcTemplate =
new JdbcTemplate(newSingleConnectionDataSource(connection, true));
// ignore case in mapping result
jdbcTemplate.setResultsMapCaseInsensitive(true);
// do your stuff
connection.commit();
I just came across this and thought the solution would help someone even if it's too late.
As Yosef said, the connection that you get by calling getJdbcTemplate().getDataSource().getConnection() method may or may not be the one used for the communication with database for your operation.
Instead, if your requirement is to just test your script, not to commit the data, you can have a Apache Commons DBCP datasource with auto commit set to fault. The bean definition is given below:
/**
* A datasource with auto commit set to false.
*/
#Bean
public DataSource dbcpDataSource() throws Exception {
BasicDataSource ds = new BasicDataSource();
ds.setUrl(url);
ds.setUsername(username);
ds.setPassword(password);
ds.setDefaultAutoCommit(false);
ds.setEnableAutoCommitOnReturn(false);
return ds;
}
// Create either JdbcTemplate or NamedParameterJdbcTemplate as per your needs
#Bean
public NamedParameterJdbcTemplate dbcpNamedParameterJdbcTemplate() throws Exception {
return new NamedParameterJdbcTemplate(dbcpDataSource());
}
And use this datasource for any such operations.
If you wish to commit your transactions, I suggest you to have one more bean of the datasource with auto commit set to true which is the default behavior.
Hope it helps someone!
I needed it to do some unit testing
In fact Spring already provides the SingleConnectionDataSource implementation with the setAutoCommit method
// import org.springframework.jdbc.datasource.SingleConnectionDataSource;
SingleConnectionDataSource dataSource = new SingleConnectionDataSource();
dataSourceRX71.setAutoCommit(false);
dataSourceRX71.setDriverClassName("xxx");
dataSourceRX71.setUrl("xxx");
dataSourceRX71.setUsername("xxx");
dataSourceRX71.setPassword("xxx");
In some case you could just add #Transactional in the method, e.g. After some batch insert, execute commit at last.
I'm using Hibernate with C3P0 connection pool. In Hibernate 3 I could get access to wrapped C3P0ProxyConnection through BorrowedConnectionProxy and then perform rawConnectionOperation. From what I see BorrowedConnectionProxy is not a part of Hibernate 4.1 anymore.
Is it any way I can run vendor specific queries ? (An instance of proxy connection inside Work.execute does not work for me, I need to execute Oracle stored procedure that takes collection of custom object type).
Thank you .
You can get access to the unproxied Connection in Work by calling:
public void execute(Connection connection) throws SQLException {
Connection unproxiedConnection = connection.unwrap( Connection.class );
...
}
That form leverages the JDBC 4 unwrap method, we simply delegate that to the underlying connection. Or if you specifically need an OracleConnection:
public void execute(Connection connection) throws SQLException {
OracleConnection oracleConnection = connection.unwrap( OracleConnection.class );
...
}
You could also use:
public void execute(Connection connection) throws SQLException {
Connection unproxiedConnection = ( (JdbcWrapper<Connection>) connection ).getWrappedObject();
...
}
I have gone back and forth in terms of contemplating allowing the Work to signify that it wants an unproxied Connection, but given the availability of Connection#unwrap I am not so sure there is an real benefit.