We have a Spring Boot application usin AWS Aurora serverless as a database. Our back-end is 99% of the time not used; therefore, we set the minimumidle parameter to 0 - hoping that Hikari would after a certain amount of time, close all the connections until they're needed again. That way our serverless database would also pause after a certain amount of inactive time.
Unfortunately, it's not working. We can see that Hikari is indeed closing the idle connections as we monitor our database. However, our serverless database isn't pausing due to inactivity.
My question is: Is Hikari is doing periodical polling to the database at certain intervals even after no idle/active connections are left? We suspect this is what's keeping our DB "active".
Related
I have a Spring boot app that use HikariCP for Postgres connection pooling.
Recently I've set up tracing to collect some data how time is spent when handling a request to a specific endpoint.
My assumptions are that when using HikariCP:
The first connection to the database while handling the request might be a bit slower
Subsequent connections to the database should be fast (< 10 ms)
However, as the trace shows, the first connection is fast (< 10 ms). And while some subsequent connections during the same request handling are also fast (< 10 ms), I frequently see some subsequent connections taking 50-100ms, which seems quite slow to me, although I'm not sure if this is to be expected or not.
Is there anything I can configure to improve this behavior?
Maybe good to know:
The backend in question doesn't really see any other traffic right now, so it's only handling traffic when I manually send requests to it
I've changed maximumPoolSize to 1 to rule out that the issue is that it uses different connections in the context of 1 request and that's what causes the issue. The same behavior is still seen.
I use the default Hikari settings, I don't change them.
I do think something is wrong with your pool configuration or your usage of the pool if it takes roughly 10 ms to get an already initialized connection from your pool. I would expect it to be sub-millisecond... Are you sure you are using the pool correctly?
Make sure you are using as new versions of pool and driver as possible, and make sure that connectionTestQuery is not set, as that would execute a query every time the connection is obtained from the pool. The defaults should be good enough for the rest of the settings.
Debug logs could be one thing help figure out what is happening, metrics on the pool another. Have a look at Spring Boot Actuator, it will help you with that...
To answer your actual question on how you can improve the situation given it actually takes roughly 10 ms to obtain a connection: Do not obtain and return the connection to the pool for every query... If you do not want to pass the connection around in your code, and if it suits your use case, you can make this happen easily by making sure your whole request is wrapped in a transaction. See the Spring guide on managing transactions.
My application currently polls database every n seconds to see if there are any new records.
To reduce network round trips, and CPU cycles of this polling i was thinking to replace it with CQN based approach where database will itself update subscribed application if there is any Commit to database.
The only problem is what if Oracle was NOT able to notify application due to any connection issue between oracle and subscribed application or if the application was crashed or killed due to any reason? ... Is there a way to know if application have missed any CQN notification?
Is polling database via application code itself the only way for mission critical applications?
You didn't say whether every 'n' seconds means you're expecting data every few seconds, or you just need your "staleness" to as low as that. That has an impact on the choice of CQN, because as per docs, https://docs.oracle.com/en/database/oracle/oracle-database/12.2/adfns/cqn.html#GUID-98FB4276-0827-4A50-9506-E5C1CA0B7778
"Good candidates for CQN are applications that cache the result sets of queries on infrequently changed objects in the middle tier, to avoid network round trips to the database. These applications can use CQN to register the queries to be cached. When such an application receives a notification, it can refresh its cache by rerunning the registered queries"
However, you have control over how persistent you want the notifcations to be:
"Reliable Option:
By default, a CQN registration is stored in shared memory. To store it in a persistent database queue instead—that is, to generate reliable notifications—specify QOS_RELIABLE in the QOSFLAGS attribute of the CQ_NOTIFICATION$_REG_INFO object.
The advantage of reliable notifications is that if the database fails after generating them, it can still deliver them after it restarts. In an Oracle RAC environment, a surviving database instance can deliver them.
The disadvantage of reliable notifications is that they have higher CPU and I/O costs than default notifications do."
I am using spring boot 2 for APIs, hosted on aws ecs fargate. and database is postgress 10.6 on RDS with 16 gb ram and 4 cpu.
my hikari configuration is as follows:
spring.datasource.testWhileIdle = true
spring.datasource.validationQuery = SELECT 1
spring.datasource.hikari.maximum-pool-size=100
spring.datasource.hikari.minimum-idle=80
spring.datasource.hikari.connection-timeout=30000
spring.datasource.hikari.idle-timeout=500000
spring.datasource.hikari.max-lifetime=1800000
Now generally this works perfectly.. but when load comes on server, say around 5000 concurrent API request..(which is also not huge though.. ), my application crashes.
Have enabled debug log for hikari.. so getting below messages:
hikaripool-1 - pool stats (total=100 active=100 idle=0 waiting=100)
Exception message says connection is not available:
HikariPool-1 - Connection is not available, request timed out after 30000ms.
org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC
At the same time when I see RDS postgress performance insighter, max query execution time is < 0.03 second..And CPU utilization also under 50%. So no issue with the Database server.
I am using entitymager and JPA only.. not using any query by opening connection manually. so closing connection or connection leak might not be an issue. But after enabling leak detection:
spring.datasource.hikari.leakDetectionThreshold=2000
Getting warn in logs saying apparent connection leak detected:
when I check the method pointing to this error: then it's just JPA findById() method..
so what should be the root cause of connection not available and request time out for just 10k api request.. with 100 pool size.. why its not releasing any connection after active connection goes to 100 and wait is 100? My ECS application server restarts automatically with this error and accessible only after 5-7 minutes after that..
HikariCP recommend removing minimumIdle when there's spike demands as you are testing
for maximum performance and responsiveness to spike demands, we recommend not setting this value and instead allowing HikariCP to act as a fixed size connection
And if you remove it. also idle-timeout is irrelevant
See also configure HikariCP for PostgreSQL
It is likely that your application is throttling it self into timeouts because of the wrong connection pool size in your configuration. A pool size of 100 is 10 times too many and this will affect performance and stability
HikariCP Pool size formula can be found in their wiki, but it looks like this:
((core_count * 2) + effective_spindle_count). Core count should not include
HT threads, even if hyperthreading is enabled.
If you have 4 cores then your connection pool size can be left at the default size of 10.
if this might help, I was facing this issue recently and it gave me a tough time.
The server accepts too many requests which hikari pool is not able to handle and so hikari tries to obtain extra connections to serve this spike in demand.
Eg. For tomcat with 200 default threads, if your maxPoolSize = 10, on spike demands, your server would try to serve 200 threads at the same time. If the connections in the pool are busy, hikari would try to obtain 190 connections and this is what you see in the waiting.
Here is how I am managing it.
I made sure that tomcat threads do not exceed the number of hikari maxPoolSize. In that way, there won't be need to ask for more connections during spike.
in spring boot, this is the config I used.
server.tomcat.threads.max = 50
spring.datasource.hikari.maximumPoolSize = 50
Note: 50 is variable based on your server capacity
We have a JBoss application where the database connections are managed via C3PO. Recently the DB node (Oracle DB) had gone down and ever since the restart of the DB we are seeing intermittent timeouts during the DB call. Ideally new connections should've been made after the DB node had come back up but looks like the connections were still persistent and had gone idle.
C3PO configurations I think could be responsible:
hibernate.c3p0.timeout=300
hibernate.c3p0.idle_test_period=1800
Upon some research i read that the idle_test_period should not be higher than the c3p0.timeout. Does this hold good and if yes what's how do these values determine when a connection is idle and has to be removed from the pool?
We have a web service written in Java and is connecting to Oracle database for data extraction. Recently, we encountered too many inactive session in Oracle database from JDBC which is our web service.
We are very sure that all the connection is being closed and set to null after every process.
Can anyone help us in this? Why is it causing inactive session in the database and what can be the solution to this.
Thank you.
What, exactly, is the problem?
Normally, the middle tier application server creates a connection pool. When your code requests a connection, it gets an already open connection from the pool rather than going through the overhead of spawning a new connection to the database. When your code closes a connection, the connection is returned to the pool rather than going through the overhead of physically closing the connection. That means that there will be a reasonable number of connections to the database where the STATUS in V$SESSION is "INACTIVE" at any given point in time. That's perfectly normal.
Even under load, most database connections from a middle tier are "INACTIVE" most of the time. A status of "INACTIVE" merely means that at the instant you ran the query, the session was not executing a SQL statement. Most connections will spend most of their time either sitting in the connection pool waiting for a Java session to open them or waiting on the Java session to do something with the data or waiting on the network to transfer data between the machines.
Are you actually getting an error (i.e. ORA-00020: maximum number of processes exceeded)? Or are you just confused by the number of entries in V$SESSION?