We are deploying Spring Boot microservices into Cloud Foundry in IBM Bluemix. The microservices all talk to a PostGres database, and it appears by default that each service is creating 10 connections that they hold open that I assume is for connection pooling. The problem we run into is that with our Postgres instance we are limited to 100 connections, so we are at times exceeding that limit.
We did some research and found that this setting may help:
spring.datasource.maxActive=5
We put that setting in our yml file for the microservices, and that helped with some of our deployments. It seems like Spring Boot sometimes adheres to the settings of maxActive and sometimes not.
I say that because when I look at the active connections in Postgres I see some of the services adhering to that limit and others that don't. We checked all the yml files, and they are all set the same. A few of us reviewed this closely to make sure that all of the yml files were the same.
Is maxActive the right setting to use to limit the connections to Postgres?
Are there other settings we should be using?
Any help is greatly appreciated.
Brian
Related
Set up - ActiveMQ Artemis 2.14.0 and Spring Boot.
Problem statement: I want to achieve throttling in terms of reading / limiting the messages to be read from ActiveMQ.
This can be achieved by configuring the consumerMaxRate during the start time and that works fine too. I want to change this parameter on the fly to increase / decrease the rate of consumption without stopping my application. I have tried by re-initializing the beans, setting the activemqconnectionfactories again but somehow the connection is maintained with the initial value only.
Any suggestion would be helpful.
I have tried searching the documentation but it only says about the parameter but with no examples.
The consumerMaxRate cannot be changed while the connection to the broker is active. You'd need to close the connection, set a new consumerMaxRate, and then create a connection with the new configuration.
I'm using a cloud database at elephantsql.com. As I'm using a free plan I can have only 5 concurrent connections.
I have a spring boot application that connects with this cloud database. The problem is that even when there is just one user at the website the spring boot application opens a lot of connections and some times it exceed the 5 connections.
I checked the connections at elephantsql.com and got this:
Its seems that the following unnecessary query is increasing the amount of connections...
SET application_name = 'PostgreSQL JDBC Driver'
How could I fix this to avoid the application to open unnecessary connections?
The setting you're looking for is probably
spring.datasource.max-active=5
Default value of this property is 10.
java.sql.SQLNonTransientConnectionException: Too many connections
Facing this issue very frequently. Can anyone please please help
It looks like your database complains that you hold too many connections open, consume even more but don't close them again.
If you have the default connection pool settings of Hikari (which you should have and not specify i.e. a poolSize of 1000), this problem should basically not happen.
I have two docker instances that I launch with docker-compose.
One holds a Cassandra instance
One holds a Spring Boot application that tries to connect to that application.
However, the Spring Boot application will always fail, because it's trying to connect to a Cassandra instance that is not ready yet to take connections.
I have tried:
Using restart:always in Docker-compose
This still doesn't always work, because the Cassandra might be up 'enough' to no longer crash the Spring Boot application, but not up 'enough' to have successfully created the Table/Column family. On top of that, this is a very hacky solution.
Using healthcheck
It seems like healthcheck in compose doesn't have restart capabilities
Using a bash script as entrypoint
In the hope that I could use netstat,ping,... whatever to determine that readiness state of Cassandra
Right now the only thing that really works is using that same bash script and sleep the process for x seconds, then start the jar. This is even more hacky...
Does anyone have an idea on how to solve this?
Thanks!
Does the spring boot service defined in the docker-compose.yml depends_on the cassandara service? If yes then the service is started only if the cassandra service is ready.
https://docs.docker.com/compose/compose-file/#depends_on
Take a look at this github repository, to find a healthcheck for the cassandra service.
https://github.com/docker-library/healthcheck
CONCLUSION
After some discussion we found out that docker-compose seems not to provide a functionality for waiting until services are up and healthy, such as Kubernetes and Openshift provide (See comments below). They recommend to use wrapper script (docker-entrypoint.sh) which waits for the depending service to come up, which make binaries necessary, the actual service shouldn't use such as the cassandra client binary. Additionally the service depending on cassandra could never get up if cassandra doesn't, which shouldn't happen.
A main thing with microservices is that they have to be resilient for failures and are not supposed to die or not to come up if a depending service is currently not available or unexpectedly disappears. Therefore the microservice should be implemented in a way so that it retries to get connection after startup or an unexpected disappearance. Unexpected is a word actually wrongly used in this context, because you should always expect such issues in a distributed environment, and even with docker-compose you will face issues like that as discussed in this topic.
The following link points to a tutorial which helped to integrate cassandra properly into a spring boot application. It provides a way to implement the retrieval of a cassandra connection with a retry behavior, therefore the service is resilient to a non existing cassandra database and will not fail to start anymore. Hope this helps others as well.
https://dzone.com/articles/containerising-a-spring-data-cassandra-application
I am troubleshooting an issue with a Spring Boot app connecting to a PostgreSQL database. The app runs normally, but under fairly moderate load it will begin to log errors like this:
java.sql.SQLException: Timeout after 30000ms of waiting for a connection.
This is running on an Amazon EC2 instance connecting to a PostgreSQL RDS. The app is configured like the following:
spring.datasource.url=jdbc:postgresql://[rds_path]:5432/[db name]
spring.datasource.username=[username]
spring.datasource.password=[password]
spring.datasource.max-active=100
In the AWS console, I see 60 connections active to the database, but that is across several Spring Boot apps (not all this app). When I query the database for current activity using pg_stat_activity, I see all but one or 2 connections in an idle state. It would seem the Spring Boot app is not using all available connections? Or is somehow leaking connections? I'm trying to interpret how pg_stat_activity would show so many idle connections and the app still getting connection pool time outs.
Figured it out. Spring is using the Hikari database connection pooling (didn't realize that until after more closely inspecting the stack trace). Hikari configuration parameters have different names, to set the pool size you use maximum-pool-size. Updated that and problem solved.