JDBC pooling Oracle driver Socket read timed out - oracle

I run Java EE application on Glassfish server v3 together with Oracle 12 DB on the same machine under Windows Server 2012 64bit. I use latest ojdbc7 driver.
Connection pool config:
<jdbc-connection-pool validation-table-name="DUAL" steady-pool-size="20" statement-cache-size="100" associate-with-thread="true" statement-timeout-in-seconds="30" idle-timeout-in-seconds="60" max-wait-time-in-millis="2000" validate-atmost-once-period-in-seconds="20" datasource-classname="oracle.jdbc.pool.OracleDataSource" pool-resize-quantity="5" max-pool-size="60" res-type="javax.sql.DataSource" name="dbPool" is-connection-validation-required="true">
<property name="driverClass" value="oracle.jdbc.OracleDriver"></property>
<property name="user" value="xxx"></property>
<property name="url" value="jdbc:oracle:thin:#(DESCRIPTION=(ENABLE=BROKEN)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=orcl)))"></property>
<property name="password" value="xxx"></property>
<property name="portNumber" value="1521"></property>
<property name="databaseName" value="orcl"></property>
<property name="serverName" value="127.0.0.1"></property>
<property name="oracle.jdbc.ReadTimeout" value="300000"></property>
<property name="oracle.net.CONNECT_TIMEOUT" value="10000"></property>
</jdbc-connection-pool>
After 2 or 3 hours, when there is more than 1 user (3-5) using my application, it stops responding and I get this in glassfish logs
javax.enterprise.resource.resourceadapter.com.sun.enterprise.resource.allocator|_ThreadID=152;_ThreadName=Thread-2;|RAR5038:Unexpected exception while creating resource for pool dbPool. Exception : javax.resource.spi.ResourceAllocationException: Connection could not be allocated because: IO Error: Socket read timed out
[...]
Local Exception Stack:
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Error in allocating a connection. Cause: Connection could not be allocated because: IO Error: Socket read timed out
From the database side it looks like this
Fatal NI connect error 12560, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 64-bit Windows: Version 12.1.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for 64-bit Windows: Version 12.1.0.1.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 12.1.0.1.0 - Production
Time: 13-JUN-2014 03:14:49
Tracing not turned on.
Tns error struct:
ns main err code: 12560
TNS-12560: TNS:protocol adapter error
ns secondary err code: 0
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (3404) as a result of ORA-609
When I just reset db listener everything works ok for next 1-2 hours (depends on application load). So temporary solution is to run bat script from windows scheduler to reset the listener every 1h.
I tried everything I could find - applied these parameters:
- Sqlnet.ora:
SQLNET.INBOUND_CONNECT_TIMEOUT=180
SQLNET.EXPIRE_TIME=5
- Listener.ora:
INBOUND_CONNECT_TIMEOUT_LISTENER_IPC=120
But still without success

Related

Can't connect to Oracle database from application

after restarting operating system (aix) I am not able to connect to locally installed oracle db from my application.
connected to sqlplus and executed startup command.
Select statements working fine from sqlplus. Should we start listener separately?
But the application fails with:
Error:
IO Error: The Network Adapter could not establish the connection
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.1.v20150916-55dc7c3): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

Hibernate/c3pO doesn't work under Tomcat - Windows 10: apparent deadlock

My app uses Hibernate and c3po, and isn't starting under windows 10 + eclipse Oxygen + Tomcat 8, while under Linux works fine with the same configuration.
These are some lines the console shows when starting Tomcat
2017/11/27 18:21:03 WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#15cc07ce -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
This should a connection problem as reported here .
This seems confirmed by an exception I get:
2017/11/27 18:21:14 WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1e723184 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.postgresql.util.PSQLException: FATALE: the remaining connection slots are reserveed to non replica superusers connections
at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:469)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
From postgresql log I see there are many connection until the db says it cannot allocate more. The connections are far above 10, while I have this in my hibernate.cfg.xml
<property name="hibernate.c3p0.min_size">1</property>
<property name="hibernate.c3p0.max_size">3</property>
<property name="hibernate.c3p0.timeout">1800</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
psql -h localhost -U user db works fine
I used Wireshark and RawCap under Windows to capture the connections and it seems the connections are acquired, as the log says, but after seeing Postgres saying "ready for queries", they are closed, if I saw well.
What else can I look at to debug what's happening??
It turned out that I was using the wrong postgresql driver. I have java 8, and using PostgreSQL JDBC 4.2 Driver, 42.1.4 solved the issue.

Wildfly 10 for jms messaging

I tried a simple spring jms example and deployed in tomee worked fine (din't start activemq server). Then I migrated to wildfly 10. But here i need to start activemq server. Is it needed wildfly is already integrated with activemq right? On starting the server (standalone.bat) should start the activemq isn't?
EDIT: Actually some configuration has been changed (I was not aware of the change). I unzipped wildfly 10 again and tried to deploy using "standalone.bat -c standalone-full.xml". Server started successfully but got the below given error.
10:29:26,172 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 10.0.0.Final (WildFly Core 2.0.10.Final) started in 13367ms - Started 401 of 691 services (397 services are lazy, passive or on-demand)
10:29:29,101 INFO [org.springframework.jms.listener.DefaultMessageListenerContainer] (org.springframework.jms.listener.DefaultMessageListenerContainer#0-1) JMS message listener invoker needs to establish shared Connection
10:29:30,111 ERROR [org.springframework.jms.listener.DefaultMessageListenerContainer] (org.springframework.jms.listener.DefaultMessageListenerContainer#0-1) Could not refresh JMS Connection for destination 'myMessageQueue' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://localhost:61616. Reason: java.net.ConnectException: Connection refused: connect
I managed to fix this problem by adding an embedded broker to my springcontext.xml
<amq:broker id="broker" useJmx="false" persistent="false">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://localhost:61617" />
</amq:transportConnectors>
</amq:broker>
Refered : http://activemq.apache.org/how-do-i-embed-a-broker-inside-a-connection.html

Glassfish JDBC Connection Pool to Mysql DB mounted on openshift

First of all, English is not my native language, so I apologize in advance for any errors in my grammar.
What I'm trying to do is connect from a local hosted Glassfish Server, to a Mysql Database mounted in Openshift.
The server has already been mounted in Openshift, and I have been able to connect to it using MySQL Workbench over TCP/IP with SSH.
The problem comes with connecting from glassfish, when I try to ping the JDBC Connection Pool it gives me the following error:
Ping Connection Pool failed for jdbc/studygroup-pool. The connection could not be allocated: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. Please check the server.log for more details.
What I have in domain.xml
<jdbc-connection-pool driver-classname="com.mysql.jdbc.Driver" name="jdbc/studygroup-pool" res-type="java.sql.Driver">
<property name="password" value="MysqlPassword"></property>
<property name="user" value="MysqlAdmin"></property>
<property name="URL" value="jdbc:mysql://127.13.73.130:3306/servicesstudygroup"></property>
</jdbc-connection-pool>
Is there anyway to give the ssh key to glassfish to connect that way?
I'm lost in what I have to change to connect to it. If there is any information missing, please tell me what I am missing and I will provide it.
This is what i get with rhc about portforwarding
Service Local
OpenShift
------- --------------- ---- -------------------
httpd 127.0.0.1:8080 => 127.13.73.131:8080
httpd 127.0.0.1:8081 => 127.13.73.133:8080
java 127.0.0.1:7600 => 127.13.73.129:7600
java 127.0.0.1:8082 => 127.13.73.129:8080
mongodb 127.0.0.1:27017 => 127.13.73.132:27017
mysql 127.0.0.1:3307 => 127.13.73.130:3306
-----------------------------------------------------------------------------------------------
And this is from env | grep MYSQL
SQL
OPENSHIFT_MYSQL_DIR=/var/lib/openshift/sshuser/mysql/
OPENSHIFT_MYSQL_DB_PORT=3306
OPENSHIFT_MYSQL_DB_HOST=127.13.73.130
OPENSHIFT_MYSQL_DB_PASSWORD=mysqlpassword
OPENSHIFT_MYSQL_IDENT=redhat:mysql:5.5:0.2.20
OPENSHIFT_MYSQL_DB_USERNAME=adminmysql
OPENSHIFT_MYSQL_DB_SOCKET=/var/lib/openshift/sshuser/mysql//socket/mysql.sock
OPENSHIFT_MYSQL_DB_URL=mysql://adminmysql:passwordmysql#127.13.73.130:3306/
OPENSHIFT_MYSQL_VERSION=5.5
OPENSHIFT_MYSQL_DB_LOG_DIR=/var/lib/openshift/sshuser/app-root/logs/
OPENSHIFT_MYSQL_LD_LIBRARY_PATH_ELEMENT=/opt/rh/mysql55/root/usr/lib64
The only thing that i have changed are the sshuser, mysql user and password.

Spring transaction/entity manager don't compensate for stale connections (on websphere + openJPA)?

The goal is to have a J2EE application running on a WebsphereApplicationServer 7, which accesses a JDBC datasource (DB2) via OpenJPA 2.0. On most of our testservers, my code is working fine; however, we have one testserver where the EntityManager aborts / does not get initialized properly because of a stale connection ("java.net.SocketException: Broken pipe"):
<openjpa-2.1.2-SNAPSHOT-r422266:1384519 nonfatal user error> org.apache.openjpa.persistence.ArgumentException: Failed to execute query "select count(x.profSurname) from Surname x where x.profUsrstate = 0". Check the query syntax for correctness. See nested exception for details.
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:872)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794)
at org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542)
at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:315)
at org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:331)
at org.apache.openjpa.persistence.QueryImpl.getSingleResult(QueryImpl.java:359)
(...)
Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: [jcc][t4][2030][11211][4.13.127] A communication error occurred during operations on the connection's underlying socket, socket input stream,
or socket output stream. Error location: T4Agent.sendRequest() - flush (-1). Message: Broken pipe. ERRORCODE=-4499, SQLSTATE=08001 {prepstmnt 1826931080 SELECT COUNT(t0.PROF_SURNAME) FROM EMPINST.SURNAME t0 WHERE (t0.PROF_USRSTATE = CAST(? AS BIGINT)) optimize for 1 row [params=?]} [code=-4499, state=08001]
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:281)
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:265)
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:72)
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeQuery(LoggingConnectionDecorator.java:1183)
at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:284)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager$CancelPreparedStatement.executeQuery(JDBCStoreManager.java:1787)
at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:274)
at org.apache.openjpa.jdbc.sql.SelectImpl.executeQuery(SelectImpl.java:499)
at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:424)
at com.ibm.ws.persistence.jdbc.sql.SelectImpl.execute(SelectImpl.java:89)
at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:391)
at org.apache.openjpa.jdbc.sql.LogicalUnion$UnionSelect.execute(LogicalUnion.java:427)
at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:230)
at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:220)
at org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:94)
at org.apache.openjpa.kernel.QueryImpl$PackingResultObjectProvider.open(QueryImpl.java:2070)
at org.apache.openjpa.kernel.QueryImpl.singleResult(QueryImpl.java:1320)
at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1242)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1007)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863)
... 113 more
---- Begin backtrace for Nested Throwables
com.ibm.websphere.ce.cm.StaleConnectionException: [jcc][t4][2030][11211][4.13.127] A communication error occurred during operations on the connection's underlying socket, socket input stream,
or socket output stream. Error location: T4Agent.sendRequest() - flush (-1). Message: Broken pipe. ERRORCODE=-4499, SQLSTATE=08001
at sun.reflect.GeneratedConstructorAccessor91.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
at java.lang.reflect.Constructor.newInstance(Constructor.java:527)
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(GenericDataStoreHelper.java:607)
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericDataStoreHelper.java:666)
at com.ibm.ws.rsadapter.AdapterUtil.mapException(AdapterUtil.java:2271)
at com.ibm.ws.rsadapter.jdbc.WSJdbcUtil.mapException(WSJdbcUtil.java:1185)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.executeQuery(WSJdbcPreparedStatement.java:726)
at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:286)
at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeQuery(LoggingConnectionDecorator.java:1181)
at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:284)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager$CancelPreparedStatement.executeQuery(JDBCStoreManager.java:1787)
at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:274)
at org.apache.openjpa.jdbc.sql.SelectImpl.executeQuery(SelectImpl.java:499)
at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:424)
at com.ibm.ws.persistence.jdbc.sql.SelectImpl.execute(SelectImpl.java:89)
at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:391)
at org.apache.openjpa.jdbc.sql.LogicalUnion$UnionSelect.execute(LogicalUnion.java:427)
at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:230)
at org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:220)
at org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:94)
(...)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:103)
at java.net.SocketOutputStream.write(SocketOutputStream.java:147)
at com.ibm.db2.jcc.t4.fb.b(fb.java:1685)
at com.ibm.db2.jcc.t4.fb.a(fb.java:1633)
at com.ibm.db2.jcc.t4.a.D(a.java:421)
... 138 more
I am working with OpenJPA using the Springframework 3.0 JpaTransactionManager and LocalContainerEntityManagerFactory to get my persistence context:
<tx:annotation-driven transaction-manager="lctxManager" />
<bean id="lctxManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="lcentityManagerFactory"></property>
</bean>
<bean id="lcentityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="persistenceUnitName" value="activities"/>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.OpenJpaVendorAdapter">
<property name="showSql" value="false"></property>
</bean>
</property>
</bean>
The persistence.xml is as follows:
<persistence-unit name="activities" transaction-type="RESOURCE_LOCAL">
<non-jta-data-source>jdbc/activities</non-jta-data-source>
<!-- My classes -->
<exclude-unlisted-classes>true</exclude-unlisted-classes>
<properties>
<property name="openjpa.TransactionMode" value="local" />
</properties>
I need to restart the server on which the application runs for the Exception to vanish again -- before it (randomly?) pops up again.
On googling the problem, I found a site that mentioned it was faulty code (no commit on transactions) that causes the problem: http://mikeschubert.com/2006/08/03/javanetsocketex/
However, I am under the impression that the JPATransactionManager is supposed to take care of that.
Other websites mentioned that implementing a connection pool would help (when using Hibernate in communication with a Tomcat-server, e.g. elegantly handling stale database connections in Hibernate/Spring Transactions); however, the Websphere Application Server already manages a connections pool for the jdbc/activities datasource (minSize: 1; maxSize: 10; Connection timeout: 180 sec; Reap time: 180sec; Unused timeout: 1800 sec, Purge policy: FailingConnectionOnly).
Any hints on where I should start looking into the problem would be great.
There is a possibility that if you don't set DB2COMM registry variable then communication error might happen.
DB2COMM should be set to TCPIP.

Resources