In development environment, spring boot with hikari, jdbc connection is unstable, if idle for some time, then call api again, it will fail(guess the network unstable cause it, because in production environment is ok)
2022-03-08 12:13:35.571 [http-nio-9090-exec-6] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate connection com.mysql.cj.jdbc.ConnectionImpl#72415749 (No operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value.
### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30038ms.] with root cause
com.mysql.cj.exceptions.ConnectionIsClosedException: No operations allowed after connection closed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.NativeSession.checkClosed(NativeSession.java:1209)
at com.mysql.cj.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:567)
at com.mysql.cj.jdbc.ConnectionImpl.setNetworkTimeout(ConnectionImpl.java:2484)
at com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(PoolBase.java:550)
at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:165)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:179)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157)
but next time it will ok.
Because it is development environment, no performance requirements, so I want to disable connection pool, that is every time use connection just create a new connection.
SO how to config spring.datasource.hikari.XXX to disable connection pool and create new jdbc connection every time when use?
You can use different datasource as SimpleDriverDataSource
as of Spring Boot 2.3.x, the following works out of the box with no need to exclude anything:
spring.datasource.type=org.springframework.jdbc.datasource.SimpleDriverDataSource
Also you can check MySQL recommended settings
Related
I'm having an issue that has gone from "annoyance" to "flip the desk".
I have a very basic Spring Boot MVC service with the following Hikari Connection Pool setup:
spring.datasource.username=ME
spring.datasource.password=mypassword
#spring.datasource.url=jdbc:oracle:thin:#SERVICE
spring.datasource.url=jdbc:oracle:thin:#localhost:1521/SERVICE
# need the duplicate because hikari is "special", but it's the spring default
#spring.datasource.jdbc-url=jdbc:oracle:thin:#SERVICE
spring.datasource.jdbc-url=jdbc:oracle:thin:#localhost:1521/SERVICE
# HikariCP settings
spring.datasource.hikari.minimumIdle=5
spring.datasource.hikari.maximumPoolSize=5
spring.datasource.hikari.idleTimeout=30000
spring.datasource.hikari.maxLifetime=2000000
spring.datasource.hikari.connectionTimeout=30000
spring.datasource.hikari.poolName=HikariPool
# leak detection - 30 seconds/30k ms
spring.datasource.hikari.leak-detection-threshold=30000
It seems that a connection is being held. I've tried restarting my Oracle Service and Listener with no luck, and the only solution seems to be to reboot.
Can anyone assist?
Update: I removed Hikari and used this:
spring.datasource.type=org.springframework.jdbc.datasource.SimpleDriverDataSource
I no longer think this is directly related to Hikari.
Edit 2:
Here's the actual "caused by" root exception:
Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor
(CONNECTION_ID=Z0U0SUlgTNuHsXgisq9YSA==)
at oracle.net.ns.NSProtocolNIO.createRefusePacketException(NSProtocolNIO.java:816) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
at oracle.net.ns.NSProtocolNIO.handleConnectPacketResponse(NSProtocolNIO.java:396) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:207) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:350) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:2372) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:657) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
... 85 common frames omitted
That particular CONNECTION_ID is randomized and different each time.
As a database we use PostgreSQL. Application uses spring libraries, and transaction is managed by spring. Transactional annotation is used correctly, where it is needed as read-only, and where it is used with write abilities. But in logs we occasinally see exception
2020-12-30 11:39:13.513 [jmsContainer-8] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - ERROR: cannot execute nextval() in a read-only transaction
2020-12-30 11:39:13.515 [jmsContainer-8] ERROR o.s.t.s.TransactionSynchronizationUtils - TransactionSynchronization.afterCompletion threw exception
javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not extract ResultSet
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:147)
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:155)
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:162)
at org.hibernate.internal.SessionImpl.firePersist(SessionImpl.java:780)
at org.hibernate.internal.SessionImpl.persist(SessionImpl.java:758)
at jdk.internal.reflect.GeneratedMethodAccessor2794.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:301)
at com.sun.proxy.$Proxy226.persist(Unknown Source)
Connections to PostgreSQL is provided through pg_bouncer, and pool_mode=sesssion, the server_reset_query=DISCARD ALL.
We tried restarting the pg_bouncer, though maybe some connection by default set read-only and being kept alive, but it did not help.
Any thoughts?
I'm trying to set up a simple test in Apache Nifi to connect to an existing PostgreSQL instance. I'm able to connect outside of nifi using other tools like dBeaver, and am fairly sure my connection string is proper. I have tried putting the postgresql jdbc driver in all sorts of places, but still keep seeing the "No suitable driver" error. I'll include some screenshots of my DBCPConnectionPool controller as well as my stack traces.
I have seen other posts like this, but none of them seem to lead to any solutions for me. Any help is appreciated.
Stack Trace
19-11-05 23:50:09,933 ERROR [Timer-Driven Process Thread-2] o.a.nifi.processors.standard.ExecuteSQL ExecuteSQL[id=3d68fb42-016e-1000-0ea4-abcc7dcc2e48] Unable to execute SQL select query select * from records; due to org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Cannot create JDBC driver of class 'org.postgresql.Driver' for connect URL 'jdbc:postgres://salt.db.elephantsql.com:5432/oickotoy'. No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Cannot create JDBC driver of class 'org.postgresql.Driver' for connect URL 'jdbc:postgres://salt.db.elephantsql.com:5432/oickotoy'
org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Cannot create JDBC driver of class 'org.postgresql.Driver' for connect URL 'jdbc:postgres://salt.db.elephantsql.com:5432/oickotoy'
at org.apache.nifi.dbcp.DBCPConnectionPool.getConnection(DBCPConnectionPool.java:442)
at org.apache.nifi.dbcp.DBCPService.getConnection(DBCPService.java:49)
at sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
at com.sun.proxy.$Proxy91.getConnection(Unknown Source)
at org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:223)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Cannot create JDBC driver of class 'org.postgresql.Driver' for connect URL 'jdbc:postgres://salt.db.elephantsql.com:5432/oickotoy'
at org.apache.commons.dbcp2.DriverFactory.createDriver(DriverFactory.java:75)
at org.apache.commons.dbcp2.BasicDataSource.createConnectionFactory(BasicDataSource.java:472)
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:538)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:753)
at org.apache.nifi.dbcp.DBCPConnectionPool.getConnection(DBCPConnectionPool.java:438)
... 19 common frames omitted
Caused by: java.sql.SQLException: No suitable driver
at org.apache.commons.dbcp2.DriverFactory.createDriver(DriverFactory.java:68)
... 23 common frames omitted
What solved this issue for me was oddly enough deleting the Database connection URL (jdbc:postgresql://.....), applying the empty connection string to the controller service, then re-typing the connection string and then applying the now re-typed connection string to the controller service.
Seems like some kind of special character has caused this hiccup
What ended up solving this in the end was a version problem. I was apparently working on a dev release of Nifi that this was broken in. After I upgraded to the latest stable release, my problem went away.
I've just switched from Spring Boot 1.X to Spring Boot 2.0 RC1, so now I'm using Spring Integration 5, which uses Lettuce as the default Redis library.
I have a RedisQueueMessageDrivenEndpoint that I want to wait until a new messages arrives, so I set the receive timeout to 0, which means it should block until something arrives. This worked with Spring Integration 4, however, with Lettuce there is a default connection timeout of 60 seconds, so I get the following exception:
Failed to execute listening task. Will attempt to resubmit in 5000 milliseconds.
org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: Command timed out
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:70)
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42)
at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:257)
at org.springframework.data.redis.connection.lettuce.LettuceListCommands.convertLettuceAccessException(LettuceListCommands.java:490)
at org.springframework.data.redis.connection.lettuce.LettuceListCommands.bRPop(LettuceListCommands.java:409)
at org.springframework.data.redis.connection.DefaultedRedisConnection.bRPop(DefaultedRedisConnection.java:478)
at org.springframework.data.redis.core.DefaultListOperations$5.inRedis(DefaultListOperations.java:215)
at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:59)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:95)
at org.springframework.data.redis.core.DefaultListOperations.rightPop(DefaultListOperations.java:211)
at org.springframework.data.redis.core.DefaultBoundListOperations.rightPop(DefaultBoundListOperations.java:154)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:195)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$200(RedisQueueMessageDrivenEndpoint.java:58)
at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:340)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:53)
at java.lang.Thread.run(Thread.java:745)
Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
at io.lettuce.core.AbstractRedisAsyncCommands.select(AbstractRedisAsyncCommands.java:1185)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:52)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy156.select(Unknown Source)
at org.springframework.data.redis.connection.lettuce.LettuceConnection.getDedicatedConnection(LettuceConnection.java:888)
at org.springframework.data.redis.connection.lettuce.LettuceListCommands.bRPop(LettuceListCommands.java:407)
... 13 common frames omitted
What is the recommended way with Lettuce to have a blocking brpop operation without running into timeouts (and getting this execptions)? I could increase the connection timeout using spring.redis.timeout, but this would affect all connections.
Lettuce is a non-blocking client based on netty and NIO. The latter two have some significance as socket read timeout (known from SocketInputStream) is not applicable.
Lettuce exposes a command timeout to protect synchronous calls against infinite wait. It makes sense to have a slight lower timeout on RedisQueueMessageDrivenEndpoint (say 50 seconds if your spring.redis.timeout is 60 seconds).
The stack trace above shows a timeout in the SELECT command. This happens before BRPOP. It looks as if spring.redis.timeout was set to 0 (no timeout at all).
Below Exception is coming in specific server, issue is not consistent.
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is java.sql.SQLException:
Connection is not associated with a managed connection.org.jboss.resource.adapter.jdbc.jdk6.WrappedConnectionJDK6#49828f4c
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:296)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:320)
at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:213)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:141)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:104)
at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:99)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:969)
at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1003)
at org.springframework.jdbc.object.StoredProcedure.execute(StoredProcedure.java:144)
at com.watsons.tfo.sp.dao.SPSaveTransactionDetail.execute(SupplierPromotionDaoImpl.java:3354)
at com.watsons.tfo.sp.dao.SupplierPromotionDaoImpl.savePromoTransaction(SupplierPromotionDaoImpl.java:422)
at com.watsons.tfo.sp.service.SupplierPromotionServiceImpl.savePromoTransaction(SupplierPromotionServiceImpl.java:158)
at sun.reflect.GeneratedMethodAccessor1010.invoke(Unknown Source)
--
org.springframework.dao.RecoverableDataAccessException: CallableStatementCallback; SQL [{call PKG_TFO_SUPPLIER_PROMOTION.PR_INS_SAVE_TRANSACTION_DET(?, ?)}]; No more data to read from socket; nested exception is java.sql.SQLRecoverableException: No more data to read from socket
at com.watsons.tfo.sp.service.SupplierPromotionServiceImpl.savePromoTransaction(SupplierPromotionServiceImpl.java:160)
at sun.reflect.GeneratedMethodAccessor1010.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy206.savePromoTransaction(Unknown Source)
at com.watsons.tfo.sp.controller.SupplierPromoTransactionController.saveSupplierPromoTransaction(SupplierPromoTransactionController.java:1077)
at com.watsons.tfo.sp.controller.SupplierPromoTransactionController$$FastClassByCGLIB$$8994f9f9.invoke(<generated>)
First i thought the is because of stale connection, So I have added stale connection checker in the oracle-ds.xml. But again the issue is came up
Please help me to resolve this.
Disabling the CachedConnection Manager
<Valve className="org.jboss.web.tomcat.tc5.jca.CachedConnectionValve"
cachedConnectionManagerObjectName="jboss.jca:service=CachedConnectionManager"
transactionManagerObjectName="jboss:service=TransactionManager" />
The above tag is mentioned in server.xml. this cachedconnectionvalve should be turned off in production server.
There are two main configuration parameters that have a direct effect on performance and scalability: cached connection manager and HTTP session replication.
The CachedConnectionManager is configured to be in the servlet container in debug mode. It's also configured in the production configuration but with debug mode off.
I commented the CachedConnectionValve at end of the server.xml file.
Also commented the CachedConnectionManager in META-INF/jboss-service.xml.
The "Connection is not associated with a managed connection" is gone and application is running fine.
Update:
Found one more log for the same error in the oracle log files...
Oracle version: Oracle Database 11.2.0.2
i.e XMLSEQUENC/EEXTRACT FAILS WITH ORA-7445[__INTEL_NEW_MEMCPY()+5395]
This is the bug with the XML Processing in oracle DB version, Then
After instructed by oracle people, Updated the patch (11666959) for oracle .
Same is mentioned in oracle web site.
11666959 is required for the release utility; XMLSEQUENC/EEXTRACT FAILS WITH ORA-7445[__INTEL_NEW_MEMCPY()+5395] IN 11.2.0.2