I have a few places in my code that does not properly close database connections. This gets periodically reported in catalina.out with messages like: org.apache.commons.dbcp.AbandonedTrace$AbandonedObjectException: DBCP object created 2013-08-29 02:55:00 by the following code was never closed. These messages are repeated for other unclosed connections for various times over the next few hours.
By looking at other info in catalina.out, I can see that these messages were printed to catalina.out around 7:40AM. I've seen other instances where these are reported in catalina.out the next day. My question is, what determines when these messages get printed to catalina.out? How does that work exactly?
DBCP is open source so you can look at the code yourself and find out. The way DBCP checks for abandoned connections is a form of cooperative garbage collection. When a connection is checked out from the connection pool, it first checks for abandoned connections and cleans them up.
So if no new connections have been requested for a few hours, abandoned connections will not be removed. And when (eg at start of a business day) a connection is requested from the pool it will first remove all abandoned connections.
If you look at the code of borrowObject(), depending on the configuration it will call removeAbandoned(), which in turn will revoke and log abandoned connections.
Related
I have a Spring boot app that use HikariCP for Postgres connection pooling.
Recently I've set up tracing to collect some data how time is spent when handling a request to a specific endpoint.
My assumptions are that when using HikariCP:
The first connection to the database while handling the request might be a bit slower
Subsequent connections to the database should be fast (< 10 ms)
However, as the trace shows, the first connection is fast (< 10 ms). And while some subsequent connections during the same request handling are also fast (< 10 ms), I frequently see some subsequent connections taking 50-100ms, which seems quite slow to me, although I'm not sure if this is to be expected or not.
Is there anything I can configure to improve this behavior?
Maybe good to know:
The backend in question doesn't really see any other traffic right now, so it's only handling traffic when I manually send requests to it
I've changed maximumPoolSize to 1 to rule out that the issue is that it uses different connections in the context of 1 request and that's what causes the issue. The same behavior is still seen.
I use the default Hikari settings, I don't change them.
I do think something is wrong with your pool configuration or your usage of the pool if it takes roughly 10 ms to get an already initialized connection from your pool. I would expect it to be sub-millisecond... Are you sure you are using the pool correctly?
Make sure you are using as new versions of pool and driver as possible, and make sure that connectionTestQuery is not set, as that would execute a query every time the connection is obtained from the pool. The defaults should be good enough for the rest of the settings.
Debug logs could be one thing help figure out what is happening, metrics on the pool another. Have a look at Spring Boot Actuator, it will help you with that...
To answer your actual question on how you can improve the situation given it actually takes roughly 10 ms to obtain a connection: Do not obtain and return the connection to the pool for every query... If you do not want to pass the connection around in your code, and if it suits your use case, you can make this happen easily by making sure your whole request is wrapped in a transaction. See the Spring guide on managing transactions.
We have a JBoss application where the database connections are managed via C3PO. Recently the DB node (Oracle DB) had gone down and ever since the restart of the DB we are seeing intermittent timeouts during the DB call. Ideally new connections should've been made after the DB node had come back up but looks like the connections were still persistent and had gone idle.
C3PO configurations I think could be responsible:
hibernate.c3p0.timeout=300
hibernate.c3p0.idle_test_period=1800
Upon some research i read that the idle_test_period should not be higher than the c3p0.timeout. Does this hold good and if yes what's how do these values determine when a connection is idle and has to be removed from the pool?
I am an Oracle DBA and not a java developer or websphere expert. We recently started using websphere in our environment. So the developers are still learning it. So I may not word my question properly. I did search the forums and saw 2 other questions like this. My question is more about how to trouble shoot this.
Websphere 8.5.0.2
Oracle 11.2.0.3
I see 20 open connections in the database. All are inactive. So they are not processing. From oracle it is v$session. Inactive means, you are open and not doing anything. Basically it is idle.
If they are inactive and not processing, they should be available for the connection pool to give to a new requester, assuming the DAO the java developer is using is being closed when done (this includes try/catch block). We confirmed that he is closing his connections.
Checks so far:
1. We reviewed the developers code. He is using standard java DAOs. He is closing his connection. He has a try/catch block and the first thing he does in the catch is close the connection.
2. My assumption is that this should cover the code path.
We don't see any errors raised in a log about 'closing' a connection.
My understanding of how a connection pool works
1. Pool Manager opens a configurable set of connections to the database. In our case it is 20.
2. When an application requests a connection, the connection manager does a lookup of the pool for the next available connection, then passes a pointer to that connect to the requesting function.
Possibility:
1. really slow server. We are using VMs for development/test. We do not have visibility to the servers to see if they are busy. So another VM could be using up CPU or disk.
Though lookups for available connections are light weight, it is possible that the server is hung up at 100% cpu and we timeout. Problem is, I don't have a way to look at this. No privileges and no access to someone who does.
not closing connections: We checked pretty thoroughly. We don't see any code passes (including exceptions) where he is not closing connections. First thing he does in a catch, is close the connection.
Any suggestions on where to look? I think its an issue with a slow server, but I want to rule other stuff out. I would like to state again that I am not a java developer or a websphere expert. So my question may be worded poorly.
the first thing he does in the catch is close the connection
Get the developer to introduce finally block after catch block and close connection in finally block, instead of catch block. Flow will move to catch only in case of error, but on normal flow the connection will not be released soon.
try {
//do something
}
catch(Exception ex) {
// log error
}
finally {
//close connection here
}
The symptoms you've described indicate a connection leak. Leaks are easy to solve (see ad-inf's response), but it can be hard to locate the source of the leak. Luckily, WAS comes with ConnLeakLogic mechanism. After enabling it, in the trace.log you'll find entries related to connections which have been retrieved from pool by application and haven't been returned for longer period of time. That connection information will also print a stack trace of a Java thread from the time of obtaining the connection. Seeing that stack traces, Java developer should be able to identify offending code.
I'm getting ActiveRecord::ConnectionTimeoutError in a daemon that runs independently from the rails app. I'm using Passenger with Apache and MySQL as the database.
Passenger's default pool size is 6 (at least that's what the documentation tells me), so it shouldn't use more than 6 connections.
I've set ActiveRecord's pool size to 10, even though I thought that my daemon should only need one connection. My daemon is one process with multiple threads that calls ActiveRecord here and there to save stuff to the database that it shares with the rails app.
What I need to figure out is whether the threads simply can't share one connection or if they just keep asking for new connections without releasing their old connections. I know I could just increase the pool size and postpone the problem, but the daemon can have hundreds of threads and sooner or later the pool will run out of connections.
The first thing I would like to know is that Passenger is indeed just using 6 connections and that the problem lies with the daemon. How do I test that?
Second I would like to figure out if every thread need their own connection or if they just need to be told to reuse the connection they already have. If they do need their own connections, maybe they just need to be told to not hold on to them when they're not using them? The threads are after all sleeping most of the time.
You can get to the connection pools that ActiveRecord is using through ActiveRecord::Base.connection_handler.connection_pools it should be an array of connection pools. You probably will only have one in there and it has a connections method on it. To get an array of connections it knows about.
You can also do a ActiveRecord::Base.connection_handler.connection_pools.each(&:clear_stale_cached_connections!) and it will checkin any checked out connections which thread is no longer alive.
Don't know if that helps or confuses more
As of February 2019, clear_state_cached_connections has been deprecated and moved to reap
Commit
Previous accepted answer updated:
ActiveRecord::Base.connection_handler.connection_pools.each(&:reap)
We have a web service written in Java and is connecting to Oracle database for data extraction. Recently, we encountered too many inactive session in Oracle database from JDBC which is our web service.
We are very sure that all the connection is being closed and set to null after every process.
Can anyone help us in this? Why is it causing inactive session in the database and what can be the solution to this.
Thank you.
What, exactly, is the problem?
Normally, the middle tier application server creates a connection pool. When your code requests a connection, it gets an already open connection from the pool rather than going through the overhead of spawning a new connection to the database. When your code closes a connection, the connection is returned to the pool rather than going through the overhead of physically closing the connection. That means that there will be a reasonable number of connections to the database where the STATUS in V$SESSION is "INACTIVE" at any given point in time. That's perfectly normal.
Even under load, most database connections from a middle tier are "INACTIVE" most of the time. A status of "INACTIVE" merely means that at the instant you ran the query, the session was not executing a SQL statement. Most connections will spend most of their time either sitting in the connection pool waiting for a Java session to open them or waiting on the Java session to do something with the data or waiting on the network to transfer data between the machines.
Are you actually getting an error (i.e. ORA-00020: maximum number of processes exceeded)? Or are you just confused by the number of entries in V$SESSION?