In spring boot-jpa if I am using hikaricp pool configuration in which I have set idle-timeout as 5 mins, max life time 2 mins and I have set jpa's ddl-auto property as create-drop, then if the connection which created the table sits idle for 7 mins, will drop the table in db?
Tables are dropped when entire SessionFactory is closed.
SessionFactory has a data source, which in your case maintains a pool of connections. Closing a connection doesn't imply that SessionFactory will get closed.
As the SessionFactory is not closed by a closed connection, the table is not dropped.
Related
In my Spring Boot microservice I have autoconfigured JdbcTemplate.
spring.datasource.url=jdbc:sqlserver://${DB_SERVER};PortNumber=${DB_PORT};failoverPartner=${DB_FAILOVER_SERVER};databaseName=${DB_NAME};
spring.datasource.username=${DB_USER}
spring.datasource.password=${DB_PASSWORD}
spring.datasource.driverClassName= com.microsoft.sqlserver.jdbc.SQLServerDriver
But after more that 20 hours of inactivity I see that connection is closed and my web service is blocked.
Does Spring Boot closes all connection in case of inactivity automatically after some time?
EDIT 21 March
If I don't add connection test every X minutes or before taking it from the pool does it mean that jdbcTemplate will not recreate them in cas of long inactivity?
spring.datasource.tomcat.testWhileIdle = true
spring.datasource.tomcat.timeBetweenEvictionRunsMillis = 50000
spring.datasource.tomcat.validationQuery = SELECT 1
Add these values to application.properties file
We have a Spring Boot application that uses Spring Integration and Spring Batch. We drop a file in the poller and it processes. This process inserts records into a database and then reads them back out does some processing and writes a file. Let's say there are 10 records. The first time we get 10 records read and 10 written. Without stopping the server, we delete all the records through a SQL client on the database, run the same file again and we get 10 records read with 20 written. I believe there is some JPA or caching going on with the datasource. We've tried turning off several auto configuration options for JPA and caching but we haven't found the right configuration option to turn off caching.
Adding a bit more detail to the question.
Basically we have cron scheduler that has a FileHandler. This the handleFile methods we have the following.
public File handleFile(File file) throws Throwable {
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
Job job = (Job) appContext.getBean("processInitialFileJob");
JobExecution jb = jobLauncher.run(job, jobParametersBuilder.toJobParameters());
....
}
What can we do to the code above to ensure that it has a new JPA session or not use the JPA session at all? This job needs to read from the database each time and not a cached representation of the database.
Are u using Hibernate. Hibernate First Level cache may be creating the problem for u. Hibernate manages a First Level cache which is local to your Session. So once u create a session and do any transactions in that hibernate syncs that within. But when u do any changes to the table outside hibernate then hibernate wont sync that until flush is called on the session and session is closed.
To make sure this is not happening, inside your poller logic try creating new Session(or EntityManager in case of JPA) and close the session for every read/process/write cycle.
Also make sure this hibernate.current_session_context_class is not set to Thread. Since thread can be reused by the poller so the same Hibernate Session may be injected again.
This ended up not being an issue with Hibernate or JPA, but an issue of a StringBuilder holding on to data from previous runs. I believe this will need to be setup as #JobScope so that it is not reused across different executions of the job.
I'm working on a Spring Boot application with Liquibase integration to setup the database. We use a different user for the database changes which we configured using the application.properties file
liquibase.user=abc
liquibase.password=xyz
liquibase.url=jdbc:postgresql://something.eu-west-1.rds.amazonaws.com:5432/app?ApplicationName=${appName}-liquibase
liquibase.enabled=true
liquibase.contexts=dev,postgres
We have at this moment 3 different microservices in deployment and we noticed that for every running instance, Liquibase opens 10 connections and it never closes these connections unless we stop the application. This basically means that in development we regularly hit the connection limit of our Amazon RDS instance.
Right now, in development, 40 of 74 active connections are occupied by Liquibase. If we ever want to go to production with this, having autoscaling enabled for all the microservices, that would mean we'll have to over-scale the database in order not to hit any connection limits.
Is there a way to
tell liquibase to not use a connection pool of 10 connections
tell liquibase to stop or close the connections
So far I found no documentation on how to do this.
Thanks to the response of Slava I managed to fix the problem with following datasource configuration class
#Configuration
public class LiquibaseDataSourceConfiguration {
private static final Logger LOG = LoggerFactory.getLogger(LiquibaseDataSourceConfiguration.class);
#Autowired
private LiquibaseDataSourceProperties liquibaseDataSourceProperties;
#LiquibaseDataSource
#Bean
public DataSource liquibaseDataSource() {
DataSource ds = DataSourceBuilder.create()
.username(liquibaseDataSourceProperties.getUser())
.password(liquibaseDataSourceProperties.getPassword())
.url(liquibaseDataSourceProperties.getUrl())
.driverClassName(liquibaseDataSourceProperties.getDriver())
.build();
if (ds instanceof org.apache.tomcat.jdbc.pool.DataSource) {
((org.apache.tomcat.jdbc.pool.DataSource) ds).setInitialSize(1);
((org.apache.tomcat.jdbc.pool.DataSource) ds).setMaxActive(2);
((org.apache.tomcat.jdbc.pool.DataSource) ds).setMaxAge(1000);
((org.apache.tomcat.jdbc.pool.DataSource) ds).setMinIdle(0);
((org.apache.tomcat.jdbc.pool.DataSource) ds).setMinEvictableIdleTimeMillis(60000);
} else {
// warnings or exceptions, whatever you prefer
}
LOG.info("Initialized a datasource for {}", liquibaseDataSourceProperties.getUrl());
return ds;
}
}
The documentation of the properties can be found on the site of Tomcat: https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html
initialSize: The initial number of connections that are created when the pool is started
maxActive: The maximum number of active connections that can be allocated from this pool at the same time
minIdle: The minimum number of established connections that should be kept in the pool at all times
maxAge: Time in milliseconds to keep this connection. When a connection is returned to the pool, the pool will check to see if the now - time-when-connected > maxAge has been reached, and if so, it closes the connection rather than returning it to the pool. The default value is 0, which implies that connections will be left open and no age check will be done upon returning the connection to the pool.
minEvictableIdleTimeMillis: The minimum amount of time an object may sit idle in the pool before it is eligible for eviction.
So it does not appear to be a connection leak, it's just the default configuration of the datasource which is not optimal for Liquibase if you use a dedicated datasource. I don't expect this to be a problem if the liquibase datasource is your primary datasource.
Update: This has been fixed in 2.5.0-M2 and Liquibase now uses a SimpleDriverDataSource without a connection pool.
Original answer: This change to connection pool management was introduced in Spring Boot version 2.0.6.RELEASE, and only takes effect if you use Spring Boot Actuator. There is an actuator endpoint (enabled by default) which allows you to get change sets applied by Liquibase. For this to work Liquibase keeps its database connections open. You can disable the endpoint with management.endpoint.liquibase.enabled = false, in which case the connection pool used by Liquibase will be shutdown after the initial run.
GitHub issue related to this change: https://github.com/spring-projects/spring-boot/issues/13832
Spring Boot Actuator (see 12. Liquibase: https://docs.spring.io/spring-boot/docs/2.0.6.RELEASE/actuator-api/html/
I don't know why liquibase doesn't close a connection, maybe it's a bug and you should create an issue for that.
To set connection pool for liquibase you have to create a custom data source and mark it with #LiquibaseDataSource annotation.
Related issues provide more details:
Possibility to specify custom dataSource configuration for liquibase only
Add LiquibaseDataSource annotation
I am getting the following error below
Unexpected error
org.springframework.transaction.CannotCreateTransactionException:
**CannotCreateTransactionException:** Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBC
Exception: Could not open connection
Here is the scenario in which it occurs :
I have recently moved some table from mysql to mongo. The code is written in such a way that either data would be taken from mongo/mysql.
The code is written in a method block which is annotated with #Transactional provided by spring framework.
There is hibernate layer which is using transaction provided by spring. c3p0 is the connection pool.
The parameter of connection pool is
hibernate.c3p0.min_size=5
hibernate.c3p0.timeout=1200
hibernate.c3p0.max_size=35
hibernate.c3p0.max_statements=50
The problem comes when we try to pull the data from mongo.Looks like the transaction is not getting closed because of mongo operation.The database connection is not getting released .It reaches the max size defined in the pool.
Tried the query in DB to find out the connection
show status like '%onn%';
Any suggestion to resolve this would really help.
Thanks
Writing a EJB 3 Service with container managed transaction. If i set autocommit false perform SQL updates and close the JDBC connection within the EJB transaction,will the transaction be still active and commit after the method completes without any exception with the connection currently closed
If you use CMT and connection from the JTA enabled datasource, then when you call connection.close() the connection is not actually closed but returned to the pool. You should also not change the autocommit with the connections taken from such datasource as they are by default set to false and shouldn't be changed. Yes, transaction will still be active, and will be commited/rollbacked, when the transaction scope ends (it doesn't have to be exactly when this particular method completes, since transaction could be started earlier and this method may only participate in it via REQUIRED, instead of running in separate as with REQIRES_NEW.