I use Spring Integration 3.0.0 with Active MQ 5.11.1 it works without any problem. but I noticed that when I stop Active MQ, i get error in my logs every 5 seconds.
if you have an idea for this problem?
Error :
ERROR [org.springframework.jms.listener.DefaultMessageListenerContainer#0-1] [DefaultMessageListenerContainer] Could not refresh JMS Connection for destination 'topic' - retrying in 5000 ms. Cause: Could not connect to broker URL: localhost. Reason: java.net.ConnectException: Connection refused: connect
When the listener container loses the connection, it tries to reconnect every 5 seconds by default until the broker is running again.
You can configure the time and/or add an exponential back off. See setRecoveryInterval and setBackOff.
Or, call stop() on the container to stop the attempts.
Call start() to start again.
Related
We are using Hikari Connection pool to connect to Oracle on Azure. We tried increasing the timeout value to 12000000 after consecutive connection time out error with a value 600000.
Error is intermittent
2022-04-06 08:32:18 java.lang.Exception: Error while getting connection
from connection pool for:#### Connection is not available, request
timed out after 300001ms.
What ever be the time we get this error intermittently.
In my production environment I got the following error in my server:
Cannot forward to error page for request [/api/validation] as the response has already been committed. As a result, the response may have the wrong status code. If your application is running on WebSphere Application Server you may be able to resolve this problem by setting com.ibm.ws.webcontainer.invokeFlushAfterService to false
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer
Now I created a client and produced 1000 thread every second to call this [/api/validation].
The error I got was
Exception in thread "Thread-9954" org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:7080/v1/name/validate": Timeout waiting for connection from pool; nested exception is org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool.
Now I want to know is what is the cause of Connection reset by peer .
According to what I know is this error occurs when the client aborts the connection by sending the RST packet.
I set the socket Timeout of my client's rest template to 9000. I make the server sleep for about 15000 MS. Now shouldn't the server get Connection reset by peer as the server tries to send the response after 15 seconds and my client just waits for about 9 seconds. Shouldn't I get the error?
Also in the production environment the wait time (Rest template socket time out) for the client is set to about a 90 seconds ( more than the time the server requires to response). Why is the error being produced in the production?
I deployed a long running Storm topology. After several hours running, the whole topology went down. I checked worker logs, and found these logs . As it says, zookeeper client session timed out and it caused reconnection. I suspect it was relate to my broken topology. Now I try to find out what can cause clients timeout.
2016-02-29T10:34:12.386+0800 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 23789ms for sessionid 0x252f862028c0083, closing socket connection and attempting reconnect
2016-02-29T10:34:12.986+0800 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED
2016-02-29T10:34:13.059+0800 b.s.cluster [WARN] Received event :disconnected::none: with disconnected Zookeeper.
2016-02-29T10:34:13.197+0800 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server zk-3.cloud.mos/172.16.13.147:2181. Will not attempt to authenticate using SASL (unknown error)
2016-02-29T10:34:13.241+0800 o.a.s.z.ClientCnxn [WARN] Session 0x252f862028c0083 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_31]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[na:1.8.0_31]
at org.apache.storm.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[storm-core-0.9.6.jar:0.9.6]
at org.apache.storm.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[storm-core-0.9.6.jar:0.9.6]
Your client can no longer talk to the ZooKeeper server. The first thing that happened was there was no answer to the heartbeats within the negotiated session timeout:
2016-02-29T10:34:12.386+0800 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 23789ms for sessionid 0x252f862028c0083, closing socket connection and attempting reconnect
Then when it tried to reconnect, it got a connection refused:
2016-02-29T10:34:13.241+0800 o.a.s.z.ClientCnxn [WARN] Session 0x252f862028c0083 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
This means either your ZooKeeper server:
Is not reachable (network connection down)
Is dead (so nothing is listening on the socket)
Is GCing itself to death and cannot communicate (although that might have issued a connection timeout error, I'm not sure)
To tell more you will need to check the ZooKeeper server logs on your (Hadoop?) cluster.
Its worked for me by increasing the connection timeout in server.properties:
zookeeper.connection.timeout.ms=60000
One way that this can happen is if you start zookeeper, then break in the terminal, then try to start kafka.
In order to use kafka, you really should use 3 terminal windows (or 3 PuTTY sessions if you are SSHing into your instance from Windows)
First Session for Zookeeper server.
Second Session for Kafka server.
Third Session for running Kafka commands to do things like create topics.
I have started Kafka in cluster mode with 3 zookeeper server and 3 Kafka server. All zookeeper server started successfully but while starting Kafka server its get disconnected stating "fatal error during Kafka server startup. prepare to shutdown (kafka.server.kafkaserver)". while investigation, I found that Kafka server get disconnected every time after 18 seconds[which is zookeeper.connection.timeout.ms = 18000 default value] so I updated the same and issue get resolved.
always use 2181 as port number for zookeeper connection until you haven't configured your zookeeper !!!
I do use failover transport feature by using the following pattern in the broker URL
failover:(tcp://host:port)
Init code goes as follow:
factory = new PooledConnectionFactory(BROKER_URL);
connection = factory.createConnection();
connection.start();
the put message code looks more or less like this:
session = connection.createSession( false, Session.AUTO_ACKNOWLEDGE );
Destination destQueue = new ActiveMQQueue(queue);
MessageProducer producer = session.createProducer(destQueue);
TextMessage msg = session.createTextMessage(message);
producer.send(msg);
When a failover occurs -
[org.apache.activemq.transport.failover.FailoverTransport] Transport (broker) failed, reason: , attempting to automatically reconnect: java.net.SocketException: recv failed: Connection aborted by peer
and got reconnected after
[org.apache.activemq.transport.failover.FailoverTransport] Failed to connect to [broker] after: 10 attempt(s) continuing to retry.
08:55:29,596 INFO [org.apache.activemq.transport.failover.FailoverTransport] Successfully reconnected to broker
do I have to reinitiate a connection? Or to be more specific, do I have to do anything with the connection object to be able to produce/consume message after the failover?
thanks
The whole point of the failover transport is to handle the reconnection for you. The logs you've shown indicate a successful reconnect cycle where the transport has continued to retry to connect to a broker and eventually did so.
When the Oracle 10 databases are up and running fine, OCILogon2() will connect immediately. When the databases are turned off or inaccessible due to network issues - it will fail immediately.
However when our DBAs go into emergency maintenance and block incomming connections, it can take 5 to 10 minutes to timeout.
This is problematic for me since I've found that OCILogin2 isn't thread safe and we can only use it serially - and I connect to quite a few Oracle DBs. 3 blocked servers X 5-10 minutes = 15 to 30 minutes of lockup time
Does anyone know how to set the OCILogon2 connection timeout?
Thanks.
I'm currenty playing with OCI and it seems to me that it's impossible.
The only way I can think of is to use non-blocking mode. You'll need OCIServerAttach() and OCISessionBegin() instead of OCILogon() in this case. But when I tried this, OCISessionBegin() constantly returns OCI_ERROR with the following error code:
ORA-03123 operation would block
Cause: The attempted operation cannot complete now.
Action: Retry the operation later.
It looks strange and I don't yet know how to deal with it.
Possible workaround is to run your logon in another process, which you can kill after timeout...
We think we found the right file setting - but it's one of those problems where we have to wait until something rare and horrible occurs before we can verify it :-/
[sqlnet.ora]
SQLNET.OUTBOUND_CONNECT_TIMEOUT=60
From the Oracle docs..
http://download.oracle.com/docs/cd/B28359_01/network.111/b28317/sqlnet.htm#BIIFGFHI
5.2.35 SQLNET.OUTBOUND_ CONNECT _TIMEOUT
Purpose
Use the SQLNET.OUTBOUND_ CONNECT _TIMEOUT parameter to specify the time, in seconds, for a client to establish an Oracle Net connection to the database instance.
If an Oracle Net connection is not established in the time specified, the connect attempt is terminated. The client receives an ORA-12170: TNS:Connect timeout occurred error.
The outbound connect timeout interval is a superset of the TCP connect timeout interval, which specifies a limit on the time taken to establish a TCP connection. Additionally, the outbound connect timeout interval includes the time taken to be connected to an Oracle instance providing the requested service.
Without this parameter, a client connection request to the database server may block for the default TCP connect timeout duration (approximately 8 minutes on Linux) when the database server host system is unreachable.
The outbound connect timeout interval is only applicable for TCP, TCP with SSL, and IPC transport connections.
Default
None
Example
SQLNET.OUTBOUND_ CONNECT _TIMEOUT=10