While initiating flow from spring web server by passing required values from PartyA to PartyB in corda, I am getting following exception in my initiating node PartyA, kindly do the needfull.
entered verifysend method
E 12:01:47+0530 [Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3#6cb2b947)] core.client.createConnection - AMQ214016: Failed to create netty connection
javax.net.ssl.SSLException: handshake timed out
at io.netty.handler.ssl.SslHandler.handshake(...)(Unknown Source) ~[netty-all-4.1.9.Final.jar:4.1.9.Final]
=========collecting ended=========
Even through my flow is running successfully by giving response as Transaction id:.... commited to ledger in web server, but my flow is taking around 5 minutes to create a unconsumed state.
I think you are run into low-memory issue, so one of your nodes was crashed. Hence lead to the handshake error.
The current minimal requirement for starting a node is 1GB of JVM Heap and 2GB minimal host RAM.
Related
In my production environment I got the following error in my server:
Cannot forward to error page for request [/api/validation] as the response has already been committed. As a result, the response may have the wrong status code. If your application is running on WebSphere Application Server you may be able to resolve this problem by setting com.ibm.ws.webcontainer.invokeFlushAfterService to false
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer
Now I created a client and produced 1000 thread every second to call this [/api/validation].
The error I got was
Exception in thread "Thread-9954" org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:7080/v1/name/validate": Timeout waiting for connection from pool; nested exception is org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool.
Now I want to know is what is the cause of Connection reset by peer .
According to what I know is this error occurs when the client aborts the connection by sending the RST packet.
I set the socket Timeout of my client's rest template to 9000. I make the server sleep for about 15000 MS. Now shouldn't the server get Connection reset by peer as the server tries to send the response after 15 seconds and my client just waits for about 9 seconds. Shouldn't I get the error?
Also in the production environment the wait time (Rest template socket time out) for the client is set to about a 90 seconds ( more than the time the server requires to response). Why is the error being produced in the production?
In spring Documentation --> 32.6 TCP Adapters it is mentioned that we use clientMode = "true" then the inbound adapter is responsible for the connection with external server.
I have created a flow in which the TCP Adapter with client connection factory makes connection with external server the code for the flow is :
IntegrationFlow flow = IntegrationFlows.from(Tcp.inboundAdapter(Tcp.nioClient(hostConnection.getIpAddress(),Integer.parseInt(hostConnection.getPort()))
.serializer(customSerializer)
.deserializer(customSerializer)
.id(hostConnection.getConnectionNumber())).clientMode(true).retryInterval(1000).errorChannel("testChannel").id(hostConnection.getConnectionNumber()+"adapter"))
.enrichHeaders(f->f.header("CustomerCode",hostConnection.getConnectionNumber()))
.channel(directChannel())
.handle(Jms.outboundAdapter(ConnectionFactory())
.destination(hostConnection.getConnectionNumber()))
.get();
theFlow = this.flowContext.registration(flow).id(hostConnection.getConnectionNumber()+"outflow").register();
I have created multiple flow by iterating over the list of connections and
iterate the above code in for loop and register them in flowcontext with unique ID.
My clients are created successfully with no issue and then establish there connection as supported by topology.
Issue :
I have counted the number of client connection created successfully so I have counted that 7 client connection (7 Integration flow) made successfully and they initiate connection from themselves.
when I create 8th client connection (8th flow created and registered successfully) but the .clientMode(true) is not working means the client don't initiate connection itself after first failure means it try for the first time to make connection if connected successfully then no issue but in case of failure it don't retry again.
Also my other created clients i.e 7 clients connection which are created successfully they also stopped initiating connection from itself when they got disconnected.
Note: There is no issue with flow only the TCP Adapters they stop initiating the connection
The flow is created and registered successfully as there is no issue it is because when I run a control bus command #adapter_id.retryConnection() it got connected with the server.
I don't understand that what is the issue with my flow that i couldn't initiate a connection after a particular count i.e seven or is there limitation in creating number of clients.
One possibility is the taskScheduler's thread pool is exhausted - that shouldn't happen with the above configuration, but it depends on what else is in the application. Take a thread dump (e.g. jstack) to see what the taskScheduler threads are doing.
See the documentation for information about how to configure the threads in the scheduler. However, if it solves it, you should really figure out what task(s) are using scheduler threads for long tasks.
Also turn on DEBUG logging to see if it provides any clues.
I am getting socket exception for broken pipe in my client side.
[write] I/O error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Broken pipe (Write failed)
[LoggingManagedHttpClientConnection::shutdown] http-outgoing-278: Shutdown connection
1520546494584[20180308 23:01:34] [ConnectionHolder::abortConnection] Connection discarded
1520546494584[20180308 23:01:34] [BasicHttpClientConnectionManager::releaseConnection] Releasing connection [Not bound]
It seems that the upgradation of httpclient jar is causing issue.
Issue is not coming with httpclient-4.3.2
Exception is coming in every 2 minutes. Issue is intermittent at times.
after , send expect:100-continue ,conn.flush is throwing exception
client and server are Linux machine
client uses http jar to make request to server REST.
Please help me in debugging the issue
can httpjar cause such issue?
The persistent connections that are kept alive by the connection manager become stale. That is, the target server shuts down the connection on its end without HttpClient being able to react to that event, while the connection is being idle, thus rendering the connection half-closed or 'stale'
This is a general limitation of the blocking I/O in Java. There is simply no way of finding out whether or not the opposite endpoint has closed connection other than by attempting to read from the socket.
If a stale connection is used to transmit a request message the request execution usually fails in the write operation with SocketException and gets automatically retried.
Apache HttpClient works this problem around by employing the so stale connection check which is essentially a very brief read operation. However, the check can and often is disabled. In fact it is often advisable to have it disabled due to extra latency the check introduces.
The handling of stale connections was changed in version 4.4. Previously, the code would check every connection by default before re-using it. The code now only checks the connection if the elapsed time since the last use of the connection exceeds the timeout that has been set. The default timeout is set to 2000ms
Suppose that after 30s (default client-failure-check-period) the client did not receive any packets from the server as a result of net connection problems.
Will the client now be disconnect from session/connection?
Suppose now I add this configration :
<retry-interval>1000</retry-interval>
<retry-interval-multiplier>1.5</retry-interval-multiplier>
<max-retry-interval>60000</max-retry-interval>
<reconnect-attempts>1000</reconnect-attempts>
What will happen now?
Will the client still get disconnected from session/connection but only after trying to reconnect 1000 times (until net is available again)? Or will it ignore the need to do disconnect?
Regarding your first question, and according to HornetQ documentation, that can be found under 17.2. Detecting failure from the client side:
As long as the client is receiving data from the server it will consider the connection to be still alive.
If the client does not receive any packets for client-failure-check-period milliseconds then it will consider the connection failed and will either initiate failover, or call any FailureListener instances (or ExceptionListener instances if you are using JMS) depending on how it has been configured.
Therefore the client will assume that the connection was in fact lost and start its failure processes.
For your second question, also according to the HornetQ documentation, that can be found under 34.3. Configuring reconnection/reattachment attributes:
reconnect-attempts. This optional parameter determines the total number of reconnect attempts to make before giving up and shutting down. A value of -1 signifies an unlimited number of attempts. The default value is 0.
So, yes, the connection will be dropped after 1000 attempts.
We are trying to write a message to a broker queue. But the whole request fails when it tries to commit the JMS transaction & then it tries to rollback each subsequent time. We use oracle XA drivers. Not sure where to post this issue: MQ forums or Oracle forum. So thought would give a try here. Can someone help resolve this please.
Error:
[9/25/12 17:10:06:871 EDT] 0000003e XATransaction E J2CA0027E: An exception occurred while invoking commit on an XA Resource Adapter from dataSource JMS$QCF$JMSManagedConnection#23, within transaction ID {XidImpl: formatId(57415344), gtrid_length(36), bqual_length(54), data(00000139ff43ef2500000001000043106c82332ef6bc723402e84f341fb357080ddd4d1b00000139ff43ef2500000001000043106c82332ef6bc723402e84f341fb357080ddd4d1b000000010000000000000000000000000001)}: javax.transaction.xa.XAException: The method 'xa_commit' has failed with errorCode '-7'.
at com.ibm.mq.jmqi.JmqiXAResource.commit(JmqiXAResource.java:407)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.commit(JMSManagedSession.java:1702)
at com.ibm.ejs.j2c.XATransactionWrapper.commit(XATransactionWrapper.java:463)
at com.ibm.ws.Transaction.JTA.JTAXAResourceImpl.commit_one_phase(JTAXAResourceImpl.java:305)
at com.ibm.ws.Transaction.JTA.RegisteredResources.flowCommitOnePhase(RegisteredResources.java:2916)
at com.ibm.ws.Transaction.JTA.TransactionImpl.commitXAResources(TransactionImpl.java:2533)
at com.ibm.ws.Transaction.JTA.TransactionImpl.stage1CommitProcessing(TransactionImpl.java:1687)
at com.ibm.ws.Transaction.JTA.TransactionImpl.processCommit(TransactionImpl.java:1647)
at com.ibm.ws.Transaction.JTA.TransactionImpl.commit(TransactionImpl.java:1582)
at com.ibm.ws.Transaction.JTA.TranManagerImpl.commit(TranManagerImpl.java:247)
at com.ibm.ws.Transaction.JTA.TranManagerSet.commit(TranManagerSet.java:168)
at com.ibm.ws.Transaction.JTA.UserTransactionImpl.commit(UserTransactionImpl.java:293)
at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1009)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1002)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:901)
at org.springframework.scheduling.commonj.DelegatingWork.run(DelegatingWork.java:61)
at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:264)
at java.security.AccessController.doPrivileged(Native Method)
at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:1137)
at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:195)
at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:187)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1560)
.
[9/25/12 17:10:06:880 EDT] 0000003e RegisteredRes W WTRN0052E: An attempt by the transaction manager to call one phase commit on a transactional resource has resulted in an XAER_RMFAIL error. The resource was com.ibm.ws.Transaction.JTA.JTAXAResourceImpl#1d07bf1#{XidImpl: formatId(57415344), gtrid_length(36), bqual_length(54), data(00000139ff43ef2500000001000043106c82332ef6bc723402e84f341fb357080ddd4d1b00000139ff43ef2500000001000043106c82332ef6bc723402e84f341fb357080ddd4d1b000000010000000000000000000000000001)}
[9/25/12 17:10:06:887 EDT] 0000003e DefaultMessag W org.springframework.jms.listener.DefaultMessageListenerContainer handleListenerSetupFailure Setup of JMS message listener invoker failed for destination 'queue:///RANDOM QUEUE?targetClient=1' - trying to recover. Cause: Heuristic completion: outcome state is mixed; nested exception is javax.transaction.HeuristicMixedException
Here's the cause and resolution
Quote-
The cause of these errors is usually the result of a WebSphere MQ
messaging provider JMS Connection being closed off by WebSphere
Application Server because the Aged timeout for the Connection has
expired.
Resolution-
To resolve this issue, ensure that the JMS Connection Factory being
used by the application has the Connection Pool property Aged timeout
set to zero. This will prevent JMS Connections being closed when they
are returned to the Free Pool, and so ensures that any outstanding
transactional work can be completed
It is sometimes also caused by the faulty DataDirect Driver and is reported and fixed by IBM, see this.
Earlier we had multiple JMS sessions. That was probably the cause of the issue in one of the environments. So we had to change to 2 diff. sessions & now it works!
Julian:
My scenario is slightly different from yours. Earlier we had:
A request message was put on the queue. This queue was picked up & processed. Then we saved to the DB & then generated another message & put it on another broker queue & then sent a response to the first message. All this was 1 flow.
Now we changed that to 2 diff. flows : Request --> Process --> Save to DB --> Reply
and then another flow to put on Broker queue.
Hope this helps