I am troubleshooting an issue with a Spring Boot app connecting to a PostgreSQL database. The app runs normally, but under fairly moderate load it will begin to log errors like this:
java.sql.SQLException: Timeout after 30000ms of waiting for a connection.
This is running on an Amazon EC2 instance connecting to a PostgreSQL RDS. The app is configured like the following:
spring.datasource.url=jdbc:postgresql://[rds_path]:5432/[db name]
spring.datasource.username=[username]
spring.datasource.password=[password]
spring.datasource.max-active=100
In the AWS console, I see 60 connections active to the database, but that is across several Spring Boot apps (not all this app). When I query the database for current activity using pg_stat_activity, I see all but one or 2 connections in an idle state. It would seem the Spring Boot app is not using all available connections? Or is somehow leaking connections? I'm trying to interpret how pg_stat_activity would show so many idle connections and the app still getting connection pool time outs.
Figured it out. Spring is using the Hikari database connection pooling (didn't realize that until after more closely inspecting the stack trace). Hikari configuration parameters have different names, to set the pool size you use maximum-pool-size. Updated that and problem solved.
Related
I want a database connection that has gone idle to be dropped once its maxLifetime has been reached.
How can we do this in spring boot/hikari? Is this achieved with these 2 properties?
idleTimeout
maxLifetime
How can I check that idle connections are actually being dropped? Any logs I can activate?
What should be the ideal values?
read Hikari documentation
check DB activity via SQL (there you can see all transaction statistics)
add logging
try locally with Postgres in Docker
I would like to know if changing the password of the DB associated with the Spring boot application affects the existing connections.
If Hikari tries to connect by reading the properties at regular intervals when managing the connection, I think there could be a connection error in this process, is that right?
Yes it does affects the connection, it will cause something like "Error establishing a database connection".
I'm using a cloud database at elephantsql.com. As I'm using a free plan I can have only 5 concurrent connections.
I have a spring boot application that connects with this cloud database. The problem is that even when there is just one user at the website the spring boot application opens a lot of connections and some times it exceed the 5 connections.
I checked the connections at elephantsql.com and got this:
Its seems that the following unnecessary query is increasing the amount of connections...
SET application_name = 'PostgreSQL JDBC Driver'
How could I fix this to avoid the application to open unnecessary connections?
The setting you're looking for is probably
spring.datasource.max-active=5
Default value of this property is 10.
I am using hikariCP for connection pooling in my reactive spring boot application running in kubernetes cluster. There will be lots of blocking calls and multiple database queries, so ideally more no of database connections would help, provided the availability of cpu cores.
Providing all the cpu core to one kubernetes container will waste resource as the spike in requests will not always be there. So I am trying to explore how to utilize the autoscaler in kubernetes so that new application containers can be spinned up with increase in the no of requests. Two concerns:
I tried the hikari configuration com.zaxxer.hikari.blockUntilFilled=true to keep the no of connections filled up during the application startup. But when using autoscaler with increasing no of requests, this will cause delays in the response as connection creation in the pool would take time. Is it better to use hikari's dynamic connection creation based on spike in demand rather than creating all the connections at once (during the startup).
Also, each kubernetes container will be a new instance of application, how do we manage the no of database connections created.
I did a sample load test with jmeter and could see improved performance (and no timeouts etc) with large no of requests when using a fixed no of active database connections. There were large no of thread interrupted exceptions when there was no fixed connection pool size provided and connections were getting created dynamically with increased no of requests.
Any insights will help.
We have an application that uses several data sources. A DB underlying one of those data sources is down at the moment: IOError. Network adapter couldn't establish the connection & Socket read timed out.
Is there an annotation (or other means) of configuring Spring Boot such that it bypasses the culprit data source and still starts up: the DB is not essential in current development work. spring.datasource.continue-on-error=true doesn't seem to work. This is Spring 2.2.2.RELEASE.
using multiple datasource, so when your apps fail at start up your apps still work, i mean using memory db / sqlite to handle fail at connection error...