I am deploying my Spring Boot REST API on AWS Fargate, which connects to AWS Aurora Postgresql Serverless V1.
I have configured the Aurora to scale the ACU to 0 when idle as in the following picture, so that I am not charge too much when I don't use the API.
Initially, my Spring Boot App maintains 10 idle connections by default, so I have tried to make it zero by adding the this to application.properties
spring.datasource.minimumIdle=0
And then I see from AWS console that the database connection has been reduced. But it remains 1 connection forever.
Please help suggest if you know how to make it zero.
The Spring Boot database configuration is basically like this
#Bean
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
Edit 1
I used the suggestion in the comment to check if the connection really comes from Spring Boot.
It turns out there is no active connection but /actuator/metrics/hikaricp.connections.idle always return the value of 1
{"name":"hikaricp.connections.idle","description":"Idle connections","baseUnit":null,"measurements":[{"statistic":"VALUE","value":1.0}],"availableTags":[{"tag":"pool","values":["HikariPool-1"]}]}
And it seems does not relate to health check because I have tried running it locally and the result of /actuator/metrics/hikaricp.connections.idle remains 1.
I set logging.level.root = trace to see what is happening.
There are only 2 things keep printing in the log periodically
The Hikari connection report, showing 1 idle connection
{"level":"DEBUG","ref":"|","marker":"INTERNAL","message":"HikariPool-1 - Before cleanup stats (total=1, active=0, idle=1, waiting=0)","logger":"com.zaxxer.hikari.pool.HikariPool","timestamp":"2022-06-14 16:15:16.799","thread":"HikariPool-1 housekeeper"}
{"level":"DEBUG","ref":"|","marker":"INTERNAL","message":"HikariPool-1 - After cleanup stats (total=1, active=0, idle=1, waiting=0)","logger":"com.zaxxer.hikari.pool.HikariPool","timestamp":"2022-06-14 16:15:16.800","thread":"HikariPool-1 housekeeper"}
{"level":"DEBUG","ref":"|","marker":"INTERNAL","message":"HikariPool-1 - Fill pool skipped, pool is at sufficient level.","logger":"com.zaxxer.hikari.pool.HikariPool","timestamp":"2022-06-14 16:15:16.800","thread":"HikariPool-1 housekeeper"}
Tomcat NioEndpoint, but I think it is not relevant
{"level":"DEBUG","ref":"|","marker":"INTERNAL","message":"timeout completed: keys processed=0; now=1655198117181; nextExpiration=1655198117180; keyCount=0; hasEvents=false; eval=false","logger":"org.apache.tomcat.util.net.NioEndpoint","timestamp":"2022-06-14 16:15:17.181","thread":"http-nio-8445-Poller"}
Thanks to the suggestion in the comment, this is because of the actuator health check, which can be solved by the following settings
management.health.db.enabled=false
Related
At the company i work with we use Spring for Kafka without authentication and lately we did some experiments to setup the security in Kafka and we enabled authentication for a brief moment which cause a crush in all our consumers/producers within our microservices ! (the microservices stayed up)
The exception :
Authorization Exception and no authorizationExceptionRetryInterval set
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: foo-group
after some researchs we found out that this is the expected behavior by kafka clients and we needed to set the authorizationExceptionRetryInterval property
public void setAuthorizationExceptionRetryInterval​(java.time.Duration authorizationExceptionRetryInterval)
Set the interval between retries after AuthorizationException is
thrown by KafkaConsumer. By default the field is null and retries are
disabled. In such case the container will be stopped. The interval
must be less than max.poll.interval.ms consumer property.
Here is some other useful links
Setting authorizationExceptionRetryInterval for Spring Kafka
Why does the spring KafkaConsumer suspend all consumption from n topics when one fails to authorize
What i want to know is :
Is a failed authentication the only case when
consumers/producers goes down ?
If there are some other cases, how to make sure that our
consumers/producers recover without human intervention (restarting
the microservices) ? In other word how to check if the
consumers/producers are up and restart them otherwise ?
Containers are stopped only under the following circumstances:
AuthorizationException with no authorizationExceptionRetryInterval
NoOffsetForPartitionException - thrown when ConsumerConfig.AUTO_OFFSET_RESET_CONFIG is not earliest or latest and there is no existing offset for a partition with this consumer group.
FencedInstanceIdException - using transactions and static group members (meaning some other instance is using this instance id).
StopAfterFenceException - when stopContainerWhenFenced is true (default false) - only applies with transactions
Any Error (such as OOME)
I'm using Spring Boot 2.0.3 release. I want to increase the maximum pool size of HikariCP which is 10 by default.
I tried changing it in applicaiton.properties with
spring.datasource.hikari.maximum-pool-size=200
but it is not working because in the logs it still show that max pool size is 10.
The reason I want to change is because I am somehow reaching that limit in staging and I have no idea what's causing it.
I faced similar issue recently (Spring Boot v2.1.3). Posting it here in case people bump into the same scenario.
Long story short: if you're initializing DataSource using #ConfigurationProperties, those properties don't seem to require hikari prefix for maximum-pool-size, unless I'm missing something. So spring.datasource.maximum-pool-size should work if you use #ConfigurationProperties(prefix = "spring.datasource").
Details: In my case I'm initializing DataSource myself using org.springframework.boot.jdbc.DataSourceBuilder, so that I could later use it in org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean:
#Bean
#Primary
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource primaryDataSource()
{
return DataSourceBuilder.create().build();
}
In this case property spring.datasource.hikari.maximum-pool-size taken from Common App Properties section in Spring Boot doc did not help. Neither did suggested above maximumPoolSize.
So I went and debugged Spring Boot code which lead me to org.springframework.boot.context.properties.bind.JavaBeanBinder and it's method bind. Value for property name for HikariDataSource setter setMaximumPoolSize was "maximum-pool-size", so just for sake of testing I renamed my property to be spring.datasource.maximum-pool-size and it finally worked.
Hope it helps. Please let me know in the comments if I missed something or took wrong path. Thanks!
As per HikariCP Github README it's maximumPoolSize so try using:
spring.datasource.hikari.maximumPoolSize = 200
But this will work only if you allow Spring Boot to create the DataSource. If you create the DataSource yourself Spring Boot properties have no effect.
Do note that 200 is a very high value that may negatively impact your database as each physical connection requires server resources. In most cases a lower value will yield better performance, see HikariCP wiki: About Pool Sizing.
I am trying to introduce spring security to angularjs app. Back end uses spring framework . I used the methodology explained here for back end security.
https://samerabdelkafi.wordpress.com/2016/01/25/secure-angularjs-application-with-spring-security/
This is working fine with single app instance.
The problem is application is clustered and therefore sessions must be replicated .
I tried to use hazelcast for session replication as explained here:
https://dzone.com/articles/spring-boot-hazelcast-for-session-replication
When I introduce hazelcast , first authenticate is successful. After that the first request is also successfull. But after that it seems that
org.springframework.security.web.context.HttpSessionSecurityContextRepository can not find the session...
As I said this start to occur after I configure com.hazelcast.web.WebFilter for sesion replication as below:
#Bean
public WebFilter webFilter(HazelcastInstance hazelcastInstance) {
Properties properties = new Properties();
properties.put("instance-name", hazelcastInstance.getName());
properties.put("sticky-session", "true");
return new WebFilter(properties);
}
Here are the related logs:
2017-08-22 15:17:31,593 : [DEBUG] [http-nio-7023-exec-2][HttpSessionSecurityContextRepository] No HttpSession currently exists
2017-08-22 15:17:31,593 : [DEBUG] [http-nio-7023-exec-2][HttpSessionSecurityContextRepository] No SecurityContext was available from the HttpSession: null. A new one will be created.
I am sure that client sends the same cookie after successfull login
I could not figure out the reason. Any guidance is appreciated.
I could make this work by using spring session..
https://docs.spring.io/spring-session/docs/current/reference/html5/guides/java-hazelcast.html
The sessions are successfully replicated between two instances behind a round robin load balancer. Many thanks to spring session developers...
I have a spring boot application that is deployed to Azure. This application uses JPA to connect to a SQL Server database. Everything seems to work nicely, until after ~30 minutes of inactivity. My guess is the database is closing the connection and the spring boot (tomcat) container isn't properly handling this situation.
I'm injecting the datasouce properties via:
#Bean
#Primary
#ConfigurationProperties(prefix = "spring.datasource")
public DataSource getDataSource() {
return DataSourceBuilder.create().build();
}
My application.properties are:
spring.datasource.url=jdbc:sqlserver://xxxxxx:1433;database=xxxxxx
spring.datasource.username=xxxxxxx
spring.datasource.password=xxxxxxxx
spring.datasource.driver-class-name=com.microsoft.sqlserver.jdbc.SQLServerDriver
spring.datasource.tomcat.test-on-borrow=true
spring.datasource.tomcat.validation-query=SELECT 1
spring.datasource.tomcat.validation-interval=0
spring.datasource.tomcat.removeAbandoned=true
spring.datasource.tomcat.removeAbandonedTimeout=120
spring.datasource.tomcat.timeBetweenEvictionRunsMillis=34000
spring.datasource.tomcat.minEvictableIdleTimeMillis=55000
spring.datasource.tomcat.initialSize=4
spring.datasource.tomcat.maxActive=16
However after a period of inactivity i'm getting SQLExceptions:
com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
I'm trying to determine the cause of this. I'm guesses connection validation isn't working correctly, but as I understand this, the connection should be validate by tomcat before the application is allowed to use it. Any help would be greatly appreciated.
I'm working on a Spring Boot application with Liquibase integration to setup the database. We use a different user for the database changes which we configured using the application.properties file
liquibase.user=abc
liquibase.password=xyz
liquibase.url=jdbc:postgresql://something.eu-west-1.rds.amazonaws.com:5432/app?ApplicationName=${appName}-liquibase
liquibase.enabled=true
liquibase.contexts=dev,postgres
We have at this moment 3 different microservices in deployment and we noticed that for every running instance, Liquibase opens 10 connections and it never closes these connections unless we stop the application. This basically means that in development we regularly hit the connection limit of our Amazon RDS instance.
Right now, in development, 40 of 74 active connections are occupied by Liquibase. If we ever want to go to production with this, having autoscaling enabled for all the microservices, that would mean we'll have to over-scale the database in order not to hit any connection limits.
Is there a way to
tell liquibase to not use a connection pool of 10 connections
tell liquibase to stop or close the connections
So far I found no documentation on how to do this.
Thanks to the response of Slava I managed to fix the problem with following datasource configuration class
#Configuration
public class LiquibaseDataSourceConfiguration {
private static final Logger LOG = LoggerFactory.getLogger(LiquibaseDataSourceConfiguration.class);
#Autowired
private LiquibaseDataSourceProperties liquibaseDataSourceProperties;
#LiquibaseDataSource
#Bean
public DataSource liquibaseDataSource() {
DataSource ds = DataSourceBuilder.create()
.username(liquibaseDataSourceProperties.getUser())
.password(liquibaseDataSourceProperties.getPassword())
.url(liquibaseDataSourceProperties.getUrl())
.driverClassName(liquibaseDataSourceProperties.getDriver())
.build();
if (ds instanceof org.apache.tomcat.jdbc.pool.DataSource) {
((org.apache.tomcat.jdbc.pool.DataSource) ds).setInitialSize(1);
((org.apache.tomcat.jdbc.pool.DataSource) ds).setMaxActive(2);
((org.apache.tomcat.jdbc.pool.DataSource) ds).setMaxAge(1000);
((org.apache.tomcat.jdbc.pool.DataSource) ds).setMinIdle(0);
((org.apache.tomcat.jdbc.pool.DataSource) ds).setMinEvictableIdleTimeMillis(60000);
} else {
// warnings or exceptions, whatever you prefer
}
LOG.info("Initialized a datasource for {}", liquibaseDataSourceProperties.getUrl());
return ds;
}
}
The documentation of the properties can be found on the site of Tomcat: https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html
initialSize: The initial number of connections that are created when the pool is started
maxActive: The maximum number of active connections that can be allocated from this pool at the same time
minIdle: The minimum number of established connections that should be kept in the pool at all times
maxAge: Time in milliseconds to keep this connection. When a connection is returned to the pool, the pool will check to see if the now - time-when-connected > maxAge has been reached, and if so, it closes the connection rather than returning it to the pool. The default value is 0, which implies that connections will be left open and no age check will be done upon returning the connection to the pool.
minEvictableIdleTimeMillis: The minimum amount of time an object may sit idle in the pool before it is eligible for eviction.
So it does not appear to be a connection leak, it's just the default configuration of the datasource which is not optimal for Liquibase if you use a dedicated datasource. I don't expect this to be a problem if the liquibase datasource is your primary datasource.
Update: This has been fixed in 2.5.0-M2 and Liquibase now uses a SimpleDriverDataSource without a connection pool.
Original answer: This change to connection pool management was introduced in Spring Boot version 2.0.6.RELEASE, and only takes effect if you use Spring Boot Actuator. There is an actuator endpoint (enabled by default) which allows you to get change sets applied by Liquibase. For this to work Liquibase keeps its database connections open. You can disable the endpoint with management.endpoint.liquibase.enabled = false, in which case the connection pool used by Liquibase will be shutdown after the initial run.
GitHub issue related to this change: https://github.com/spring-projects/spring-boot/issues/13832
Spring Boot Actuator (see 12. Liquibase: https://docs.spring.io/spring-boot/docs/2.0.6.RELEASE/actuator-api/html/
I don't know why liquibase doesn't close a connection, maybe it's a bug and you should create an issue for that.
To set connection pool for liquibase you have to create a custom data source and mark it with #LiquibaseDataSource annotation.
Related issues provide more details:
Possibility to specify custom dataSource configuration for liquibase only
Add LiquibaseDataSource annotation