After rebooting the server, the oracle connection from the Tomcat server times out every night. Prior to the reboot, the connection didn't timeout. Now, in the morning, the application throws a JDBC connection error while accessing the DB. Restarting Tomcat corrects the issue. I'm assuming that's due to the connections being re-established. I think, this is due to the Oracle DB timing out the session. How can the session timeout be disabled in Oracle 11g?
Thanks!
Steve
Config.groovy with dev and test omitted.
dataSource {
pooled = true
}
hibernate {
cache.use_second_level_cache = true
cache.use_query_cache = true
cache.provider_class = 'net.sf.ehcache.hibernate.EhCacheProvider'
}
// environment specific settings
environments {
production {
dataSource {
driverClassName = "oracle.jdbc.driver.OracleDriver"
username = "XXXXX"
password = "XXXXXX"
dialect = "org.hibernate.dialect.Oracle10gDialect"
dbCreate = "update" // one of 'create', 'create-drop','update'
url = "jdbc:oracle:thin:#XXXXXX:1521:xxxx"
}
} }
That's generally controlled by the profile associated with the user Tomcat is connecting as.
SQL> SELECT PROFILE, LIMIT FROM DBA_PROFILES WHERE RESOURCE_NAME = 'IDLE_TIME';
PROFILE LIMIT
------------------------------ ----------------------------------------
DEFAULT UNLIMITED
SQL> SELECT PROFILE FROM DBA_USERS WHERE USERNAME = USER;
PROFILE
------------------------------
DEFAULT
So the user I'm connected to has unlimited idle time - no time out.
Adam has already suggested database profiles.
You could check the SQLNET.ORA file. There's an EXPIRE_TIME parameter but this is for detecting lost connections, rather than terminating existing ones.
Given it happens overnight, it sounds more like an idle timeout, which could be down to a firewall between the app server and database server. Setting the EXPIRE_TIME may stop that happening (as there'll be check every 10 minutes to check the client is alive).
Or possibly the database is being shutdown and restarted and that is killing the connections.
Alternatively, you should be able to configure tomcat with a validationQuery so that it will automatically restart the connection without a tomcat restart
This is likely caused by your application's connection pool; not an Oracle DBMS issue. Most connection pools have a validate statement that can execute before giving you the connection. In oracle you would want "Select 1 from dual".
The reason it started occurring after you restarted the server is that the connection pool was probably added without a restart and you are just now experiencing the use of the connection pool for the first time. What is the modification dates on your resource files that deal with database connections?
Validate Query example:
<Resource name="jdbc/EmployeeDB" auth="Container"
validationQuery="Select 1 from dual" type="javax.sql.DataSource" username="dbusername" password="dbpassword"
driverClassName="org.hsql.jdbcDriver" url="jdbc:HypersonicSQL:database"
maxActive="8" maxIdle="4"/>
EDIT:
In the case of Grails, there are similar configuration options for the grails pool. Example for Grails 1.2 (see release notes for Grails 1.2)
dataSource {
pooled = true
dbCreate = "update"
url = "jdbc:mysql://localhost/yourDB"
driverClassName = "com.mysql.jdbc.Driver"
username = "yourUser"
password = "yourPassword"
properties {
maxActive = 50
maxIdle = 25
minIdle = 5
initialSize = 5
minEvictableIdleTimeMillis = 60000
timeBetweenEvictionRunsMillis = 60000
maxWait = 10000
}
}
I came to this question looking for a way to enable oracle session pool expiration based on total session lifetime instead of idle time.
Another goal is to avoid force closes unexpected to application.
It seems it's possible by setting pool validation query to
select 1 from V$SESSION
where AUDSID = userenv('SESSIONID') and sysdate-LOGON_TIME < 30/24/60
This would close sessions aging over 30 minutes in predictable manner that doesn't affect application.
Does the DB know the connection has dropped, or is the session still listed in v$session? That would indicate, I think, that it's being dropped by the network. Do you know how long it can stay idle before encountering the problem, and if that bears any resemblance to the TCP idle values (net.ipv4.tcp_keepalive_time, tcp_keepalive_probes and tcp_keepalive_interval from sysctl if I recall correctly)? Can't remember whether sysctl changes persist by default, but that might be something that was modified and then reset by the reboot.
Also you might be able to reset your JDBC connections without bouncing the whole server; certainly can in WebLogic, which I realise doesn't help much, but I'm not familiar with the Tomcat equivalents.
Check applications connection Pool settings, rather than altering any session timout settings on the oracle db. It's normal that they time out.
Have a look here:
http://grails.org/doc/1.0.x/guide/3.%20Configuration.html#3.3%20The%20DataSource
Are you sure that you have set the "pooled" parameter correctly?
Greetings,
Lars
EDIT:
Your config seems ok on first glimpse.
I came across this issue today. Maybe it is related to your pain:
"Infinite loop of exceptions if the application is started when the database is down for maintenance"
Related
I am creating a VM in azure to upload a postgres instance in docker and connect to it with my local backend in Spring. What happens is that once connected to the DB after X time of inactivity when trying to make a request I get the following "HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#f162126 (This connection has been closed.). Possibly consider using a shorter maxLifetime value." digging around I realized that it is as if my VM has some kind of behavior that when a connection becomes inactive it closes it causing the above error. The curious thing here is that the sessions are not closed as you can see in the following image even shutting down my backend the sessions are maintained and the only options to delete them is restarting the container in which the DB is hosted.
I have tried to reproduce this behavior on local but it never happens even if I leave the backend idle for an hour if I do the request to the DB it works as if nothing, it only happens with my VM in azure.
I want to clarify that the sessions that appear in the attached image no longer work, i.e. if I try to consume the DB from spring, the error I mentioned appears and automatically Hikari creates new sessions for its pool and I can reproduce this behavior until I reach 100 sessions that after a while would not work again and that Spring never closes when shutting down the backend.
HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection#f162126 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
This error is thrown by method isConnectionDead. While checking the connection, if it's still alive & can be used and it will issue an above error if it has already been closed.
You can adjust your maxLifetime setting, to resolve this problem. 30000ms (30 seconds) is the shortest value that is permitted (30 seconds), 1800000ms (30 minutes) is by default value.
A connection in the pool can only last for a certain amount of time which is controlled by the maxLifetime attribute. the value of this property it should be several seconds below any connection time restrictions set by the infrastructure or any databases.
Reference: Hikari Configuration Github.
Well, after much research and reviewing various sources it turns out that azure when creating a VM has certain security policies as Pedro Perez says in the following post in Stack Excahnge: Azure closing idle network connections
You're hitting a design feature of the software load balancer in front of your VMs. By default it will close any idle connections after 4 minutes, but you can configure the timeout to be anything between those 4 and 30 minutes
So in order to overwrite this policy that governs your VM you must do the process of creating a load balancer, do all the relevant configuration and create a Load balancing rule for port 5432 which is the default port of postgres and set the Idle timeout in a range of 4 to 30 min according to your needs.
finally configure your VM so its public ip points to the LB(Load Balancer) public ip and everything will work normally.
It is true that if you simply want to take advantage of Azure's default security policies on the VMs you create you should set the maxLifetime to a maximum of 4 minutes in your Spring application.properties or appliation.yml as #PratikLad says.
in my case I prefer to leave the default Hikari configuration (maxLifetime of 30 mins) so I need to create the LB but if you prefer to change the property by setting it to a maximum of 4 min you would not need to do all the above mentioned on the LB.
We're testing the failover behaviour using the MariaDB JDBC connector Aurora specific features.
We've set the JDBC URL as the documentation suggest:
jdbc:mysql:aurora://cluster.cluster-xxxx.us-east-1.rds.amazonaws.com/db
The problem is that as soon as we add the aurora: part in the URL schema, we can see an increase in the connections to the database writer until the point that we've to rollback the change (it even reaches 3.000 connections).
Versions:
MariaDB connector: 2.0.1
HikariCP connection pool: 2.6.1
Play-Slick: 2.1.0
Slick: 3.2.0
Configuration:
master {
profile = "slick.jdbc.MySQLProfile$"
db {
driver = "org.mariadb.jdbc.Driver"
url = "jdbc:mysql:aurora://cluster-name.cluster-xxx.us-east-1.rds.amazonaws.com/db_name?characterEncoding=utf8mb4&rewriteBatchedStatements=true&usePipelineAuth=false"
user = "rw_user"
password = "rw_user_pass"
numThreads = 20
queueSize = 1000000
}
}
slaves = [
{
profile = "slick.jdbc.MySQLProfile$"
db {
driver = "org.mariadb.jdbc.Driver"
url = "jdbc:mysql:aurora://cluster-name.cluster-ro-xxx.us-east-1.rds.amazonaws.com/db_name?characterEncoding=utf8mb4&usePipelineAuth=false"
user = "ro_user"
password = "ro_user_pass"
numThreads = 20
queueSize = 1000000
}
}
]
We'd tried to add the aurora: part to the JDBC URL schema after upgrading the MariaDB connector version, but the number of connections to the Reader started to increase again:
If we run a show processlist on the read only endpoint, we can see all the opened connections in "cleaned up" state, and "Sleep" command.
We'd removed the aurora: part from the read only endpoint just in order to stabilize the number of connections to it. Is it possible that the driver searches for the cluster master while opening connections? That would explain this kind of behaviour.
When using the "aurora" keyword, driver , under the hood, create 2 connections:
a connection to the primary server,
a connection to one of the replicas if any.
The goal is always to save resources on the main server. Generally, only one pool is configured. The driver then uses the connection to the primary / replica according to [Connection.setReadOnly] [1].
When you have separate "write" / "read" pools, using the configuration "failover" will solve your issue: Driver will use only one real connection.
This way, there will be no "wasted" connection.
Failover will then be handled differently, but with the same results (for example, a query not in a transaction that is to be sent to a replica that just crashed will not directly use the primary connection as when using the "aurora" configuration, the driver will recreate a new connection to another replicas before executing the query).
Once you get past several dozen active connections, the database starts stumbling over itself. It is better to throttle the connections in the client instead of assuming you have infinite bandwidth to accept connections in Aurora.
Does SQL Azure allow 3-rd party connection pool like HikariCP or BoneCP?
We configured HikariCP it works when we just run app but later db doesnt response on request. Is it HikariCP issue or it's common connection poool issue and no need spending more time on investigation?
HikariConfig config = new HikariConfig();
config.setMaximumPoolSize(50);
config.setDriverClassName(env.getProperty("jdbc.driverClassName"));
config.setJdbcUrl(env.getProperty("jdbc.url"));
config.setUsername(env.getProperty("jdbc.user"));
config.setPassword(env.getProperty("jdbc.pass"));
config.addDataSourceProperty("cachePrepStmts", env.getProperty("jdbc.cachePrepStmts"));
config.addDataSourceProperty("prepStmtCacheSize", env.getProperty("jdbc.prepStmtCacheSize"));
config.addDataSourceProperty("prepStmtCacheSqlLimit", env.getProperty("jdbc.prepStmtCacheSqlLimit"));
config.addDataSourceProperty("useServerPrepStmts", env.getProperty("jdbc.useServerPrepStmts"));
See this SQL Azure page re: Connection Constraints.
Maximum allowable durations are subject to change depending on the resource usage.
A logged-in session that has been idle for 30 minutes will be terminated
automatically. We strongly recommend that you use the connection pooling and
always close the connection when you are finished using it so that the unused
connection will be returned to the pool. For more information about connection
pooling, see Connection Pooling.
See if any of these errors match up to your logs. Search that page for "terminated" and "busy" to find error codes that might be relevant to your issue.
I would suggest setting the maxLifetime property in HikariCP to 15 minutes, and the idleTimeout to 2 minutes.
There is nothing on the SQL Azure side that would prohibit you from using a 3rd party connection pool. My guess is that the connection failed between the server and the client and the client didn't remove the connection from the pool.
Moving forward, I'd ensure that whichever 3-rd part connection pool you end up using tests that the connection exists before taking it out of the pool for use.
Hope that helps.
We are using tomcat jdbc connection pooling with Oracle database.
Recently, we encountered problem that there are too many inactive session in Oracle database from JDBC THIN CLIENT.
Can anyone help us in this? Why is it causing inactive session in the database and what can be the solution to this.
Thank you.
Adjust settings of your pool, following the documentation. Setting maxIdle and minEvictableIdleTimeMillis to low values would ensure that idle connections are evicted fast, and that few idle connections stay open. Obviously, it will also make your pool less efficient, since connections will be closed and opened more frequently.
If you are getting SQLExceptions because connections have timed out, you should set a validationQuery (to something like SELECT 0 FROM DUAL) and your connections will be tested before they are checked-out of the pool. Any connection that has failed will be replaced by a working connection and then returned to your code.
We are using ADO to access Oracle 10g release 2, Oledb provider for Oracle 10g. We are facing some issue with the connection pooling. The database reside on the remote machine and connection pooling is occuring as it should. But if the remote machine goes down for some reason, the connection is returned from the pool and query on that connection fails. When this connection is closed, it is returned back to the pool instead of being invalid. The subsequent connection opening requests are sucessfull but query fails. This is strange behaviour, according to OLEDB specifications, provider must support DBPROP_CONNECTIONSTATUS property, thus in case of invalid connection, it would not be returned back to the pool.
Things get weired when the remote machine comes up. The connections in the pool are still invalid and although the connection opening succeeds, query on the connection fails. Oracle OLEDB is unable to connect to the server anymore and we have to restart our application. Well this is undesired cause our application is a critical application.
Any ideas on how to get over this.
Thanks
Mubashir
If you are doing this programmatically, use a try block, so that if something does happen, it won't fail. With a try block, you can catch an exception and ignore it, so that the errors are shushed.
You could tell the pool to not accept invalid connections, by marking the connection invalid before it is returned to the pool.
Connections are recovered after 10 minutes by default. Time can be set by the registry key SPTimeout under the oledb provider's root key.
Actually the default connection pool timeout is 120 seconds at least for this Oracle 11 32-bit OLEDB installation. You can find the registry settings at:
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Oracle\KEY_orac
Where KEY_orac is the KEY_<oracle_home_name>
The key name is ORAMTS_CONN_POOL_TIMEOUT and the default value is 120.
It does not appear that you can set connection pool parameters at the connection string level.
In most connection pool implementation it is possible to check the connection before using it. For example: you define a check query like select * from dual and if you pick up a connection from the pool this query will be executed. If it fails, then the connection will be excluded from the pool and a new one will be opened.