Long running Stand-alone Spring app on a weekly schedule (not on any App server) usually takes few days to complete but failed due to java.net.SocketException: Broken pipe.
Here's stack trace:
Caused by: java.sql.SQLRecoverableException: IO Error: Broken pipe
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:876)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3620)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:93)
at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:703)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:644)
... 28 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at oracle.net.ns.DataPacket.send(DataPacket.java:199)
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:211)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:227)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1122)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:863)
The java.net.SocketException: Broken pipe exception suggests that, you are writing while the other end has already closed the connection. As the other end isn't reading it. As I also said, if this is happening there is something wrong with your application protocol specification or implementation, most probably that you don't even have one.
If application is dealing with heavy amount of data, think about JVM memory issues. Another possibility is that other end of connection does not wait long enough then it closes the connection, because application is taking too long to prepare the data and send. And connection times out.
Read this article to know more about it.
Related
I'm trying to access a queue.
def mqProps = new Hashtable<String, Object>()
mqProps.put(MQConstants.CHANNEL_PROPERTY, 'CHANNEL')
mqProps.put(MQConstants.PORT_PROPERTY, PORT)
mqProps.put(MQConstants.HOST_NAME_PROPERTY, 'HOST')
mqProps.put(MQConstants.USER_ID_PROPERTY, 'myuser') // is it the correct property for the user?
mqProps.put(MQConstants.PASSWORD_PROPERTY, 'mypassword') // is it the correct property for the password?
def qMgr = new MQQueueManager('QM', mqProps)
However I'm facing the following error
javax.script.ScriptException: com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2538'
...
Caused by: com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2538'.
...
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host 'HOST(PORT)' rejected.
...
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host 'HOST/address:PORT' rejected.
...
Caused by: java.net.ConnectException: Connection timed out: connect
...
The error happened on the line:
def qMgr = new MQQueueManager('QM', mqProps)
Can you please explain me reason of this issue? Thank you a lot.
Reason Code 2538 is MQRC_HOST_NOT_AVAILABLE.
You can quickly discover this by using the mqrc command line tool that comes with IBM MQ. Type:
mqrc 2538
and you will be told:
2538 0x000009ea MQRC_HOST_NOT_AVAILABLE
Alternatively you can look it up in the IBM MQ Knowledge Center.
Reading the explanation in Knowledge Center will show you a number of common possibilities for the problem.
The listener has not been started on the remote system.
The connection name in the client channel definition is incorrect.
The network is currently unavailable.
A firewall blocking the port, or protocol-specific traffic
Perhaps the most common of errors is that the listener running at the queue manager is not using the same port number that you have put in the client application connection details.
You haven't shown us in your question any details about the listener running on the queue manager, so we will have to leave that for you to check yourself.
We have a web application with the stack Spring, Hibernate, C3P0, Oracle DB Driver (habing an Oracle DB behind).
From time to time we experience blocking locks over a longer period of time which then get killed on the DB end. (we know this is caused by bad application design and we will fix it, but it's not the point of this quesion).
After the DB session was killed by DB it seems that the connection pool reuses the now broken connection which results in the error:
A PooledConnection that has already signalled a Connection error is still in use!
Another error has occurred [ java.sql.SQLRecoverableException: Closed Connection ] which will not be reported to listeners!
On the DataSource we configured
dataSource.setTestConnectionOnCheckin(true);
dataSource.setTestConnectionOnCheckout(true);
But it did not help. We expected that the connections fail these tests and then get renewed. But this does not happen.
Any hints for us how to recreate the broken connections?
This warning is given when a Connection that is already checked out experiences an Exception that causes c3p0 to treat it as invalid (so it will not be reincorporated back into the pool on close()), but the Connection continues to be used and experiences an Exception again. These are not broken Connections in the pool. They are broken Connections in-use by the application. So testing them on checkout (or checkin) doesn't do anything about them.
To get rid of this, you need to examine the Exception handling within your application code. Are there circumstances where an invalid Connection might have thrown an Exception, but that Exception was caught and the Connection reused?
The warning itself is harmless. It's just saying c3p0 already knows the Connection is bad, it won't emit an event to signal that again.
This is the error I am getting when I am using springframework for creating a client, Let me know why I am getting this error,I think it is related to ssl connection, Please let me know your suggestion to stop this error
Exception in thread "main" org.springframework.ws.client.WebServiceIOException: I/O error: Connection reset; nested exception is java.net.SocketException: Connection reset
at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:545)
at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386)
at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380)
at com.eoi.oraclecrm.inbound.census.service.OracleCRMCensusDataService.QueryCrm(OracleCRMCensusDataService.java:40)
at com.selerix.enrollment.service.QueryHandler.QueryRecord(QueryHandler.java:43)
at com.selerix.enrollment.service.SelerixServiceTest2.main(SelerixServiceTest2.java:57)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.springframework.ws.transport.http.CommonsHttpConnection.onSendAfterWrite(CommonsHttpConnection.java:114)
at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:47)
at org.springframework.ws.client.core.WebServiceTemplate.sendRequest(WebServiceTemplate.java:624)
at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:587)
at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539)
... 5 more
As #Steffen Ullrich already commented, it seems like a server restriction to avoid attacks. You will have to contact the team in charge of the server and negotiate a solution. E.g. whitelist your IP for a couple days, use another massive loading method, etc.
This production issue is now taking over my head. I have configured to use Bitronix as transaction manager with the jBPM5 tool. It works fine from a long time but sudden started breaking with below exception.
Exception stacktrace
Caused by: java.sql.SQLException: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 1 connection(s) (1 still available)
at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:262)
at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:71)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
... 78 more
Caused by: bitronix.tm.internal.BitronixRuntimeException: cannot get valid connection from an XAPool of resource jdbc/jbpm-ds with 1 connection(s) (1 still available) after trying for 30s
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:160)
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:91)
at bitronix.tm.resource.jdbc.PoolingDataSource.getConnection(PoolingDataSource.java:258)
... 80 more
Caused by: java.sql.SQLRecoverableException: IO Error: Connection timed out
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:899)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1175)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1296)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3613)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3657)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1495)
at bitronix.tm.resource.jdbc.JdbcPooledConnection.testConnection(JdbcPooledConnection.java:215)
at bitronix.tm.resource.jdbc.JdbcPooledConnection.getConnectionHandle(JdbcPooledConnection.java:299)
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:130)
... 82 more
Caused by: java.net.SocketException: Connection timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at oracle.net.ns.Packet.receive(Packet.java:308)
at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:324)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:268)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:190)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:107)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:350)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:208)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:886)
... 90 more
Strange thing is that application has sufficient number of XA connections are available in XAPool. It takes the connection, tried to connect, unable to connect and just discard the connection and take new connection to process.
SEVERE: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 9 connection(s) (9 still available)
SEVERE: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 8 connection(s) (8 still available)
SEVERE: unable to get a connection from pool of a PoolingDataSource containing an XAPool of resource jdbc/jbpm-ds with 7 connection(s) (7 still available)
Could anybody please shed some light to recover from this issue?
Update1:
After some try I am able to reproduce the same issue in local. Once database connection is resetted or network connectivity lost between my application & database, pool connections there in PoolingDataSource (provided by Bitronix) pool becomes invalid. My query is Why it is unable auto-recover? Is there any why through which we can auto-recover or establish the new connections?
I am using btm version 2.1.4.
Thanks,
Mayank
I was having the same issue but it was due to user permissions. My user does not have enough permissions so I was getting this issue. After that, I ran these 3 below-mentioned commands to create a new user and it works for me. I hope, it helps someone.
CREATE USER 'newuser'#'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON * . * TO 'newuser'#'localhost';
FLUSH PRIVILEGES;
I am facing a very strange problem.
I am using my application with RMI for 8 month in production but since a new (big) update I dont know what is wrong.
It is a server where 3 clients connect.
And when I am using only client with a certain method I can not find anything wrong especially because all other clients work perfectly.
Is there maybe with RMI any local "cache" which should be deleted otherwise I can't tell.
Here is the output of the client's console:
C:\Users\Claudia\Desktop>java -jar CLIENT_erp_auer_client_v0_8ASASADSD.jar
java.rmi.MarshalException: error marshalling arguments; nested exception is:
java.net.SocketException: Software caused connection abort: socket write error
at sun.rmi.server.UnicastRef.invoke(Unknown Source)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(Unknown Sou
rce)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(Unknown Source)
at $Proxy7.update(Unknown Source)
at client.entity_types.LieferscheinpositionService.aendereLieferscheinposition(
LieferscheinpositionService.java:60)
at client.lieferscheinverwaltung.TreeTableNEULieferschein.addLieferscheinpositi
onen(TreeTableNEULieferschein.java:912)
at client.lieferscheinverwaltung.TreeTableNEULieferschein.<init>(TreeTableNEULi
eferschein.java:204)
at client.lieferscheinverwaltung.LieferscheinErstellung.getTreeTableNeuLiefersc
hein(LieferscheinErstellung.java:916)
at client.lieferscheinverwaltung.LieferscheinErstellung.<init>(LieferscheinErst
ellung.java:773)
at client.lieferscheinverwaltung.LieferscheinManagement.openLieferschein(Liefer
scheinManagement.java:1055)
at client.lieferscheinverwaltung.LieferscheinManagement$9.mouseClicked(Liefersc
Thank you.