I am getting the below error from jboss server when the load in the application is very high. When this error comes the application didnt process any request after a few times the connection resets and application resumes.
DEBUG Periodic recovery - first pass <Tue, 7 Feb 2017 13:01:59>
[Thread-12] [2017-02-07 13:01:59,029] DEBUG StatusModule: first pass
[Thread-12] [2017-02-07 13:01:59,029] DEBUG [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
[Thread-12] [2017-02-07 13:01:59,030] DEBUG [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
[Thread-12] [2017-02-07 13:02:09,030] DEBUG Periodic recovery - second pass <Tue, 7 Feb 2017 13:02:09>
[Thread-12] [2017-02-07 13:02:09,030] DEBUG AtomicActionRecoveryModule: Second pass
[Thread-12] [2017-02-07 13:02:09,030] DEBUG [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_6] - TORecoveryModule - second pass
[Thread-12] [2017-02-07 13:02:09,030] DEBUG [com.arjuna.ats.internal.jta.recovery`enter code here`.info.secondpass] Local XARecoveryModule - second pass
[Thread-12] [2017-02-07 13:02:09,030] DEBUG Recover java:/DefaultJMSProvider
[Thread-12] [2017-02-07 13:02:09,031] DEBUG ********************************Failed to connect to server
javax.naming.NameNotFoundException: XAConnectionFactory not bound
at org.jnp.server.NamingServer.getBinding(NamingServer.java:771)
at org.jnp.server.NamingServer.getBinding(NamingServer.java:779)
at org.jnp.server.NamingServer.getObject(NamingServer.java:785)
at org.jnp.server.NamingServer.lookup(NamingServer.java:443)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:726)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at org.jboss.util.naming.Util.lookup(Util.java:222)
at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.getConnectionFactory(MessagingXAResourceWrapper.java:393)
at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.connect(MessagingXAResourceWrapper.java:339)
at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.getDelegate(MessagingXAResourceWrapper.java:261)
at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.recover(MessagingXAResourceWrapper.java:108)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:773)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecovery(XARecoveryModule.java:665)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:200)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:799)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:412)
[Thread-12] [2017-02-07 13:02:09,031] DEBUG Cannot get delegate XAResource
javax.transaction.xa.XAException: Error trying to connect to provider java:/DefaultJMSProvider
at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.getDelegate(MessagingXAResourceWrapper.java:271)
at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.recover(MessagingXAResourceWrapper.java:108)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:773)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecovery(XARecoveryModule.java:665)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:200)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:799)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:412)
Caused by: javax.naming.NameNotFoundException: XAConnectionFactory not bound
at org.jnp.server.NamingServer.getBinding(NamingServer.java:771)
at org.jnp.server.NamingServer.getBinding(NamingServer.java:779)
at org.jnp.server.NamingServer.getObject(NamingServer.java:785)
at org.jnp.server.NamingServer.lookup(NamingServer.java:443)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:726)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at org.jboss.util.naming.Util.lookup(Util.java:222)
at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.getConnectionFactory(MessagingXAResourceWrapper.java:393)
at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.connect(MessagingXAResourceWrapper.java:339)
at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.getDelegate(MessagingXAResourceWrapper.java:261)
after this below warning is also observed
WARN Destroying connection that is not valid, due to the following exception: oracle.jdbc.driver.T4CConnection#43017c6e
java.sql.SQLException: pingDatabase failed status=-2
at org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker.isValidConnection(OracleValidConnectionChecker.java:74)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:560)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:323)
at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.matchManagedConnections(LocalManagedConnectionFactory.java:402)
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:225)
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:690)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)
at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:424)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)
at com.infosys.fdc.util.ConnectionManager.getConnectionFromPool(ConnectionManager.java:382)
at com.infosys.fdc.util.ConnectionManager.getConnection(ConnectionManager.java:507)
at com.infosys.fdc.ejb.CommonEJBBean.execute(CommonEJBBean.java:165)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:386)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:233)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:173)
at org.jboss.ejb.plugins.TxInterceptorBMT.invoke(TxInterceptorBMT.java:77)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:173)
at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:228)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)
at org.jboss.ejb.plugins.security.PreSecurityInterceptor.process(PreSecurityInterceptor.java:100)
at org.jboss.ejb.plugins.security.PreSecurityInterceptor.invoke(PreSecurityInterceptor.java:84)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)
at org.jboss.ejb.Container.invoke(Container.java:1092)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
at org.jboss.proxy.ejb.SecurityContextInterceptor.invoke(SecurityContextInterceptor.java:64)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:68)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)
at $Proxy125.execute(Unknown Source)
java.sql.SQLException: pingDatabase failed status=-2
This means there is "something" wrong with either the Oracle database itself or with the network.
pingDatabase is a method of oracle.jdbc.OracleConnection and is used by the JBoss OracleValidConnectionChecker to check if a connection to the database is still valid and working.
According to Oracle Database JBDC Driver API doc - Constant Field Values, the return codes are:
DATABASE_OK= 0; // The physical database connection is not closed and the database is reachable. (will not be logged)
DATABASE_CLOSED= -1; // The physical database connection is closed.
DATABASE_NOTOK= -2; // The physical database connection is not closed but the database is not reachable.
DATABASE_TIMEOUT= -3 // The call timed out before any positive or negative acknowledgement was received.
Test the network with the usual means like ping, telnet, ssh... connection tests.
Check if your DNS name resolution is working properly; use an IP address instead of a hostname when in doubt.
Test if connections to the Oracle database are working using the Oracle tnsping utility or with SQL*plus.
Note: requires a native Oracle client install.
JBoss is only reporting this error which is coming from the Oracle JDBC driver. The root cause will either be the Oracle database itself, or the network.
Upgrade the Oracle JDBC driver. We have seen many reports where this fixed the issue. It would seem that the older driver(s) had a bug which could cause return code "-2".
To get to the root cause, need to work with Oracle since this error is coming from the Oracle driver, and is only being reported by JBoss.
The error indicates that connectivity was not possible between the JBoss application server and the database
This is most likely due to a problem with the network or the database server itself. Sometimes, such issues are temporary, however, you will want to engage your network and/or DBA team to review the issue if this issue persists.
It is highly recommend reaching out to Oracle with the TRACE logs and getting input from them.
Computer science 101. Reduce the load on your server. My team has developed some demos to show the impact of large number of connections to your database, and you can see a video of it here: https://www.youtube.com/watch?v=XzN8Rp6glEo
Related
I connected Talend to Autonomous Data Warehouse following the instructions supplied by Oracle, https://www.oracle.com/a/ocom/docs/database/talend-connection-adw.pdf
The connection seems to be successful, the studio builder is able to connect and run queries. But when we try to create a new job and add the connection, running the job returns the following error:
java.sql.SQLRecoverableException: IO Error: General SSLEngine problem, connect lapse 22 ms., Authentication lapse 0 ms.
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:794)
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:688)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:39)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:691)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at adw.test_0_1.test.tDBInput_1Process(test.java:337)
at adw.test_0_1.test.runJobInTOS(test.java:674) I ...
Anyone has ever seen this before?
This is a known problem with older jdbc drivers.
Download the latest ojdbc8.jar from https://www.oracle.com/database/technologies/appdev/jdbc-downloads.html is the best way to go.
(I cannot tell you offhand which version it started to work with and where exactly the problem was coming from, but it works with 19.10 and onwards for sure.)
Almost all the client connections are directed to one server and the application fails to connect after some time with IOException. Seems load balancing is not working properly. We are forced to restart the listener at this point.
jdbc URL : jdbc:oracle:thin:#(DESCRIPTION=(FAILOVER=ON)(LOAD_BALANCE=yes)(ADDRESS=(PROTOCOL=TCP)(HOST=scan-ip)(PORT=1538))(CONNECT_DATA=(service_name=production)(SERVER=DEDICATED)))
Exception:
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Io exception: Connection reset)
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 60 more
Caused by: java.sql.SQLException: Io exception: Connection reset
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:414)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
... 64 more
Version:
11.2.0.4.0 - 64bit
Just to make sure - when you say "all connection are directed to one node" - do you mean that the session are created on one node or that the scan listener taking care of the connection is only being used on one node?
Check and see if you SCAN address contain 3 IP addresses (nslookup scan-ip), make sure that your SCAN listeners are being distributed across nodes (srvctl status scan and srvctl status scan_listener). You can also check to see that the database service you're trying to use (production) is being registered on all appropriate listeners (check database parameter "remote_listener" to make sure where the service is being registered to).
I am trying to establish a jdbc-oci connection to an oracle database on a coldfusion v 10 std server.
The reason for employing jdbc oci is the necessity to encrypt data flowing between the app server and the database (by using common settings in sqlnet.ora). However, no matter what syntax we try, neither a service
based nor a tnsnames based jdbc url will work. Immediately after saving (and automatically testing) the connection, coldfusion errors out with "java.sql.SQLException: Timed out trying to establish connection".
It looks weird that a timeout may have happened since the error will always be thrown immediately. Therefor I suppose there is another root cause for failing to establish the connection.
Any ideas out there?
Please note that we simulated connection establishment with a console application aside coldfusion but using the same jvm and jdbc driver successfully. it does therefore not seem to be a jvm or driver or oci.dll
related issue.
data source settings :
Driver Class : oracle.jdbc.driver.OracleDriver
JDBC URL (using tnsnames.ora) : jdbc:oracle:oci:#<theName>
JDBC URL (using ip, port, service) : jdbc:oracle:oci:#//<IP>:1521/<SID>
error message :
Connection verification failed for data source: theName
java.sql.SQLException: Timed out trying to establish connection
The root cause was that: java.sql.SQLException: Timed out trying to establish connection
environment :
ojdbc6 driver lives in a directory known to coldfusion
oci.dll lives in a directory known to the system path variable, verified by sysinternals processexplorer on the coldfusion service
for the JDBC URL (using tnsnames.ora), the directory containing tnsnames.ora is known as a system variable TNS_ADMIN, verified by sysinternals processexplorer on the coldfusion service
along with tnsnames.ora there is a sqlnet.ora file comprising settings to switch on data encryption
Stacktrace:
A non-SQL error occurred while requesting a connection from "datasource-name here".
Timed out trying to establish connection
Exception in thread "Thread-2120" java.lang.UnsatisfiedLinkError: no ocijdbc11 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
at java.lang.Runtime.loadLibrary0(Runtime.java:845)
at java.lang.System.loadLibrary(System.java:1084)
at oracle.jdbc.driver.T2CConnection$1.run(T2CConnection.java:3560)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.jdbc.driver.T2CConnection.loadNativeLibrary(T2CConnection.java:3556)
at oracle.jdbc.driver.T2CConnection.logon(T2CConnection.java:269)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:553)
at oracle.jdbc.driver.T2CConnection.<init>(T2CConnection.java:165)
at oracle.jdbc.driver.T2CDriverExtension.getConnection(T2CDriverExtension.java:53)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:528)
at coldfusion.server.j2ee.sql.pool.JDBCPool.createPhysicalConnection(JDBCPool.java:614)
at coldfusion.server.j2ee.sql.pool.ConnectionRunner$RunnableConnection.run(ConnectionRunner.java:67)
at java.lang.Thread.run(Thread.java:722)
A non-SQL error occurred while requesting a connection from "datasource-name here".
Timed out trying to establish connection
[Fatal Error] :2662:4: The element type "view" must be terminated by the matching end-tag "</view>".
Adding of the Path to the oracle instant client at the configuration file cf-directory\cfusion\bin\jvm.config has solved the problem.
I am facing a very strange problem.
I am using my application with RMI for 8 month in production but since a new (big) update I dont know what is wrong.
It is a server where 3 clients connect.
And when I am using only client with a certain method I can not find anything wrong especially because all other clients work perfectly.
Is there maybe with RMI any local "cache" which should be deleted otherwise I can't tell.
Here is the output of the client's console:
C:\Users\Claudia\Desktop>java -jar CLIENT_erp_auer_client_v0_8ASASADSD.jar
java.rmi.MarshalException: error marshalling arguments; nested exception is:
java.net.SocketException: Software caused connection abort: socket write error
at sun.rmi.server.UnicastRef.invoke(Unknown Source)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(Unknown Sou
rce)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(Unknown Source)
at $Proxy7.update(Unknown Source)
at client.entity_types.LieferscheinpositionService.aendereLieferscheinposition(
LieferscheinpositionService.java:60)
at client.lieferscheinverwaltung.TreeTableNEULieferschein.addLieferscheinpositi
onen(TreeTableNEULieferschein.java:912)
at client.lieferscheinverwaltung.TreeTableNEULieferschein.<init>(TreeTableNEULi
eferschein.java:204)
at client.lieferscheinverwaltung.LieferscheinErstellung.getTreeTableNeuLiefersc
hein(LieferscheinErstellung.java:916)
at client.lieferscheinverwaltung.LieferscheinErstellung.<init>(LieferscheinErst
ellung.java:773)
at client.lieferscheinverwaltung.LieferscheinManagement.openLieferschein(Liefer
scheinManagement.java:1055)
at client.lieferscheinverwaltung.LieferscheinManagement$9.mouseClicked(Liefersc
Thank you.
For testing purposes I try to run Netty 3.9.0 echo server example (can be found in Netty examples) inside Oracle database.
It starts correctly and I can see service running on port (with nmap), but whenever try to send message I get below exception:
java.net.SocketException: recv() failed, errno = 104
Connection reset by peer
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:33)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
at sun.nio.ch.IOUtil.read(IOUtil.java:166)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
kgskupdbalance: time_used[58236668] greater than threshold[30000000] current_time[12380314910] time_start[12322078242]
Remarks:
Code works when run outside (in shell)
privileges are granted (and no other exception found)
My 1st suspect would be java.nio in database
Somebody has explanation ?
NOTE: Just to repeat, it's just for testing purposes.
Do not ask for motivation :-)
Thank you
It goes in zone which is not supported (native methods).