DB connection issue with Playframework 2.1 and Bonecp 0.8.0 : This connection has been closed - heroku

I was facing an issue with Bonecp 0.7.1 on a Playframework app using postgresql 9.2.4 on Heroku. It seems this version had a DB connection leak causing after several DB accesses the folllwing error :
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection Sleeping for 1000ms and trying again. Attempts left: 1. Exception: null.Message:FATAL: too many connections for role "eonqhnjenuislk" Database warning
[error] c.j.b.PoolWatchThread - Error in trying to obtain a connection. Retrying in 1000ms
org.postgresql.util.PSQLException: FATAL: too many connections for role "eonqhnjenuislk"
at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:469) ~[postgresql-9.1-901.jdbc4.jar:na]
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112) ~[postgresql-9.1-901.jdbc4.jar:na]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) ~[postgresql-9.1-901.jdbc4.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) ~[postgresql-9.1-901.jdbc4.jar:na]
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) ~[postgresql-9.1-901.jdbc4.jar:na]
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) ~[postgresql-9.1-901.jdbc4.jar:na]
As every threads of the connection pool was acquired and retained, the application was not reachable anymore until I restarted it.
Then I heard that this issue was corrected in Bonecp 0.8.0, so I upgraded the lib. But the issue seems not to be completely fixed. In fact, now connection threads are not retained anymore what make the application reachable at anytime but sometimes a DB connection close suddenly... The app throws the following error causing an 500 error to the end users :
javax.persistence.PersistenceException: org.postgresql.util.PSQLException: This connection has been closed.
at com.avaje.ebeaninternal.server.transaction.TransactionManager.createTransaction(TransactionManager.java:331)
at com.avaje.ebeaninternal.server.core.DefaultServer.createServerTransaction(DefaultServer.java:2056)
at com.avaje.ebeaninternal.server.core.BeanRequest.createImplicitTransIfRequired(BeanRequest.java:58)
at com.avaje.ebeaninternal.server.core.PersistRequest.initTransIfRequired(PersistRequest.java:81)
at com.avaje.ebeaninternal.server.persist.DefaultPersister.executeSqlUpdate(DefaultPersister.java:146)
at com.avaje.ebeaninternal.server.core.DefaultServer.execute(DefaultServer.java:1928)
at com.avaje.ebeaninternal.server.core.DefaultServer.execute(DefaultServer.java:1935)
at com.avaje.ebeaninternal.server.core.DefaultSqlUpdate.execute(DefaultSqlUpdate.java:148)
at actor.PublicParkingPlacesActor$1.apply(PublicParkingPlacesActor.java:41)
at actor.PublicParkingPlacesActor$1.apply(PublicParkingPlacesActor.java:26)
at play.libs.F$Promise$PromiseActor.onReceive(F.java:425)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:159)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:425)
at akka.actor.ActorCell.invoke(ActorCell.scala:386)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:230)
at akka.dispatch.Mailbox.run(Mailbox.scala:212)
at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:502)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:714)
at org.postgresql.jdbc2.AbstractJdbc2Connection.setAutoCommit(AbstractJdbc2Connection.java:661)
at com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
at play.api.db.BoneCPApi$$anon$1.onCheckOut(DB.scala:328)
at com.jolbox.bonecp.AbstractConnectionStrategy.postConnection(AbstractConnectionStrategy.java:75)
at com.jolbox.bonecp.AbstractConnectionStrategy.getConnection(AbstractConnectionStrategy.java:92)
at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:553)
at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:131)
at play.db.ebean.EbeanPlugin$WrappingDatasource.getConnection(EbeanPlugin.java:146)
at com.avaje.ebeaninternal.server.transaction.TransactionManager.createTransaction(TransactionManager.java:297)
... 20 more
Thanks a lot for your help!
EDIT :
DB configuration :
db.default.isolation=READ_COMMITTED
db.default.partitionCount=2
db.default.maxConnectionsPerPartition=10
db.default.minConnectionsPerPartition=5
db.default.acquireIncrement=1
db.default.acquireRetryAttempts=2
db.default.acquireRetryDelay=5 seconds
db.default.connectionTimeout=10 second
db.default.idleMaxAge=10 minute
db.default.idleConnectionTestPeriod=5 minutes
db.default.initSQL="SELECT 1"
db.default.maxConnectionAge=1 hour
EDIT 2:
Here is the DB config I set according to this post Heroku/Play/BoneCp connection issues
These changes reduce the number of "This connection has been closed" issues, but I still get 1 or 2 of them perdays what makes some HTTP requests to fail. So the issue is still not fixed:
db.default.isolation=READ_COMMITTED
db.default.partitionCount=2
db.default.maxConnectionsPerPartition=10
db.default.minConnectionsPerPartition=5
db.default.acquireIncrement=1
db.default.acquireRetryAttempts=2
db.default.acquireRetryDelay=5 seconds
db.default.connectionTimeout=10 seconds
db.default.idleMaxAge=10 minutes
db.default.idleConnectionTestPeriod=30 seconds
db.default.initSQL="SELECT 1"
db.default.maxConnectionAge=30 minutes

Related

Wrong atomikos state ABORTING after timeout in transaction

I am testing how atomikos solving timeout in transaction. I have 30s timeout in infinite read from db. After 30s I got this exception:
14:08:40.329 [Atomikos:4] WARN c.a.icatch.imp.ActiveStateHandler - Transaction rob-app-b1c7b95a3b0efb82dfb516b04620a213154159609015500001 has timed out - rolling back...
2018-11-07 14:08:40.354 [pool-3-thread-1] WARN c.a.jdbc.JdbcConnectionProxyHelper - Error enlisting in transaction - connection might be broken? Please check the logs for more information...
java.lang.IllegalStateException: wrong state: ABORTING
at com.atomikos.icatch.imp.CoordinatorImp.registerSynchronization(CoordinatorImp.java:420)
at com.atomikos.icatch.imp.TransactionStateHandler.registerSynchronization(TransactionStateHandler.java:129)
at com.atomikos.icatch.imp.CompositeTransactionImp.registerSynchronization(CompositeTransactionImp.java:177)
at com.atomikos.jdbc.AtomikosConnectionProxy.enlist(AtomikosConnectionProxy.java:211)
at com.atomikos.jdbc.AtomikosConnectionProxy.invoke(AtomikosConnectionProxy.java:122)
at com.sun.proxy.$Proxy133.prepareStatement(Unknown Source)
Why I got this error and no Atomikos Exception in AtomikosConnectionProxy during enlist method ?
AtomikosSQLException.throwAtomikosSQLException("The transaction has timed out - try increasing the timeout if needed");
Seems like there is a timeout because your application has been using the transaction for too long. Try increasing the transaction timeout?

Hibernate/c3pO doesn't work under Tomcat - Windows 10: apparent deadlock

My app uses Hibernate and c3po, and isn't starting under windows 10 + eclipse Oxygen + Tomcat 8, while under Linux works fine with the same configuration.
These are some lines the console shows when starting Tomcat
2017/11/27 18:21:03 WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#15cc07ce -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
This should a connection problem as reported here .
This seems confirmed by an exception I get:
2017/11/27 18:21:14 WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1e723184 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.postgresql.util.PSQLException: FATALE: the remaining connection slots are reserveed to non replica superusers connections
at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:469)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
From postgresql log I see there are many connection until the db says it cannot allocate more. The connections are far above 10, while I have this in my hibernate.cfg.xml
<property name="hibernate.c3p0.min_size">1</property>
<property name="hibernate.c3p0.max_size">3</property>
<property name="hibernate.c3p0.timeout">1800</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
psql -h localhost -U user db works fine
I used Wireshark and RawCap under Windows to capture the connections and it seems the connections are acquired, as the log says, but after seeing Postgres saying "ready for queries", they are closed, if I saw well.
What else can I look at to debug what's happening??
It turned out that I was using the wrong postgresql driver. I have java 8, and using PostgreSQL JDBC 4.2 Driver, 42.1.4 solved the issue.

Connection refused error in worker logs - apache storm

I see the below error in worker logs, it happens almost every milliseconds, but the cluster is running fine, I wanted to know what does these error mean and any idea on why this would occur.
This happens on all the worker nodes
2016-05-12T15:32:53.514-0500 b.s.m.n.Client [ERROR] connection attempt 3 to Netty-Client-xxxxx.hq.abc.com/xx.xx.xxx.xx:6700 failed: java.net.ConnectException: Connection refused: xxxxx.hq.abc.com/xx.xx.xxx.xxx:6700
And after some time i see this
2016-05-12T15:44:25.940-0500 b.s.m.n.Client [ERROR] discarding 1 messages because the Netty client to Netty-Client-xxxxx.hq.abc.com/xx.xx.xxx.xxx:6700 is being closed
After struggling forever with this problem, I found that setting storm.local.hostname property in the storm.yaml file solved the issue for me. On my laptop, I set storm.zookeeper.servers, nimbus.host and storm.local.hostname all to "localhost".
I am using version 0.10.2 of storm.

JasperReports Server "The job failed to execute"

I am new to JasperReports Server and have searched for a resolve to this issue. I have found nothing. I inherited this server and the reports are not running as scheduled.
Job: 3rd (ID: 61)
Report unit: /reports/Scheduled/00_Schedule_Primer
Quartz Job: ReportJobs.job_61
Quartz Trigger: ReportJobs.trigger_62_1
Exceptions:
An error occurred while executing the report.
com.jaspersoft.jasperserver.api.JSException: jsexception.error.creating.connection
at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.JdbcDataSourceService.createConnection(JdbcDataSourceService.java:62)
at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.BaseJdbcDataSource.setReportParameterValues(BaseJdbcDataSource.java:52)
at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.JdbcDataSourceService.setReportParameterValues(JdbcDataSourceService.java:67)
at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.fillReport(EngineServiceImpl.java:743)
at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.fillReport(EngineServiceImpl.java:367)
at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.executeReport(EngineServiceImpl.java:876)
at com.jaspersoft.jasperserver.api.engine.jasperreports.domain.impl.ReportUnitRequest.execute(ReportUnitRequest.java:60)
at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.execute(EngineServiceImpl.java:301)
at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJob.executeReport(ReportExecutionJob.java:444)
at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJob.executeAndSendReport(ReportExecutionJob.java:372)
at com.jaspersoft.jasperserver.api.engine.scheduling.quartz.ReportExecutionJob.execute(ReportExecutionJob.java:188)
at org.quartz.core.JobRunShell.run(JobRunShell.java:195)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: java.sql.SQLException: Middleware connect fail:No RPC Connection active.
at com.ibm.u2.jdbc.UniJDBCMsgFactory.createException(UniJDBCMsgFactory.java:113)
at com.ibm.u2.jdbc.UniJDBCExceptionSupport.addAndThrowException(UniJDBCExceptionSupport.java:62)
at com.ibm.u2.jdbc.UniJDBCProtocolU2Impl.connect(UniJDBCProtocolU2Impl.java:746)
at com.ibm.u2.jdbc.UniJDBCProtocolU2Impl.executeOpenDatabase(UniJDBCProtocolU2Impl.java:243)
at com.ibm.u2.jdbc.UniJDBCConnectionImpl.<init>(UniJDBCConnectionImpl.java:116)
at com.ibm.u2.jdbc.UniJDBCDriver.connect(UniJDBCDriver.java:111)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:48)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.JdbcDataSourceService.createConnection(JdbcDataSourceService.java:58)
... 12 more
I don't even have a clue where to start troubleshooting this. ANY help would be awesome!
It seems your connection url, username and password values are not proper.If these three things are proper then the connection will be created.Check in your Jasper server configuration.

WebSocket with Spring 4.0 with MessageBroker and WebServers

I've downloaded the latest version of the websocket with spring example from
https://github.com/rstoyanchev/spring-websocket-portfolio.
I'm running the application on Tomcat 8.0 RC5 and using the message broker RabbitMQ 3.1.5.
The simulation runs only 2-3 times after that the simulation doesn't work anymore, due to the below error. I tried also toi run the application on Glassfish 4.0, but the same error occurs. Can you please help what could be the reason?
15:47:35 [BrokerWebSocketChannel-1] ExceptionWebSocketHandlerDecorator - Unhandl
ed error for ExceptionWebSocketHandlerDecorator [delegate=LoggingWebSocketHandle
rDecorator [delegate=org.springframework.messaging.handler.websocket.SubProtocol
WebSocketHandler#1d2bde4]]
java.lang.IllegalStateException: The WebSocket session has been closed and no me
thod (apart from close()) may be called on a closed session
at org.apache.tomcat.websocket.WsSession.checkState(WsSession.java:642)
at org.apache.tomcat.websocket.WsSession.getNegotiatedSubprotocol(WsSess
ion.java:297)
at org.springframework.web.socket.adapter.StandardWebSocketSession.getAc
ceptedProtocol(StandardWebSocketSession.java:113)
at org.springframework.web.socket.sockjs.transport.session.WebSocketServ
erSockJsSession.getAcceptedProtocol(WebSocketServerSockJsSession.java:91)
at org.springframework.messaging.handler.websocket.SubProtocolWebSocketH
andler.findProtocolHandler(SubProtocolWebSocketHandler.java:149)
at org.springframework.messaging.handler.websocket.SubProtocolWebSocketH
andler.afterConnectionClosed(SubProtocolWebSocketHandler.java:224)
at org.springframework.web.socket.support.WebSocketHandlerDecorator.afte
rConnectionClosed(WebSocketHandlerDecorator.java:69)
at org.springframework.web.socket.support.LoggingWebSocketHandlerDecorat
or.afterConnectionClosed(LoggingWebSocketHandlerDecorator.java:74)
at org.springframework.web.socket.support.ExceptionWebSocketHandlerDecor
ator.afterConnectionClosed(ExceptionWebSocketHandlerDecorator.java:89)
at org.springframework.web.socket.sockjs.transport.session.AbstractSockJ
sSession.close(AbstractSockJsSession.java:237)
at org.springframework.messaging.simp.stomp.StompProtocolHandler.handleM
essageToClient(StompProtocolHandler.java:183)
at org.springframework.messaging.handler.websocket.SubProtocolWebSocketH
andler.handleMessage(SubProtocolWebSocketHandler.java:194)
at org.springframework.messaging.support.channel.ExecutorSubscribableCha
nnel$1.run(ExecutorSubscribableChannel.java:80)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:724)

Resources