Getting Rabbitmq HeartBeat Exception - amazon-ec2

Question

I am facing one issue with rabbitmq, that it terminate the connection with below exception
{"message":"com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; cause: com.rabbitmq.client.MissedHeartbeatException: Heartbeat missing with heartbeat = 60 seconds"}
so I am using following configuration for it:-
Rabbitmq docker on aws ec2-instance1 with t2_small server.
Services which are facing this issue are running on aws ec2-instance2 with t2-small server.
Set following configuration for rabbitmq
factory.setAutomaticRecoveryEnabled(true);
factory.setNetworkRecoveryInterval(1000);
factory.setRequestedHeartbeat(60);
I just not able to understand what can be the reason for getting this type of error, how can we resolve this issue.
Also I found below logs from the service which was trying to connect with rabbitmq
{"log":"Caught an exception during connection recovery!\n","stream":"stderr","time":"2018-03-22T00:00:00.632851865Z"}
{"log":"java.net.NoRouteToHostException: No route to host\n","stream":"stderr","time":"2018-03-22T00:00:00.633374123Z"}
{"log":"\u0009at java.net.PlainSocketImpl.socketConnect(Native Method)\n","stream":"stderr","time":"2018-03-22T00:00:00.633666158Z"}
{"log":"\u0009at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)\n","stream":"stderr","time":"2018-03-22T00:00:00.633935828Z"}
{"log":"\u0009at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)\n","stream":"stderr","time":"2018-03-22T00:00:00.634170787Z"}
{"log":"\u0009at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)\n","stream":"stderr","time":"2018-03-22T00:00:00.63440824Z"}
{"log":"\u0009at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)\n","stream":"stderr","time":"2018-03-22T00:00:00.634625637Z"}
{"log":"\u0009at java.net.Socket.connect(Socket.java:589)\n","stream":"stderr","time":"2018-03-22T00:00:00.635038038Z"}
{"log":"\u0009at com.rabbitmq.client.impl.FrameHandlerFactory.create(FrameHandlerFactory.java:32)\n","stream":"stderr","time":"2018-03-22T00:00:00.635172903Z"}
{"log":"\u0009at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:34)\n","stream":"stderr","time":"2018-03-22T00:00:00.635369445Z"}
{"log":"\u0009at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverConnection(AutorecoveringConnection.java:435)\n","stream":"stderr","time":"2018-03-22T00:00:00.635639932Z"}
{"log":"\u0009at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.beginAutomaticRecovery(AutorecoveringConnection.java:407)\n","stream":"stderr","time":"2018-03-22T00:00:00.63584649Z"}
{"log":"\u0009at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.access$000(AutorecoveringConnection.java:53)\n","stream":"stderr","time":"2018-03-22T00:00:00.636051142Z"}
{"log":"\u0009at com.rabbitmq.client.impl.recovery.AutorecoveringConnection$1.shutdownCompleted(AutorecoveringConnection.java:352)\n","stream":"stderr","time":"2018-03-22T00:00:00.636233667Z"}
{"log":"\u0009at com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(ShutdownNotifierComponent.java:75)\n","stream":"stderr","time":"2018-03-22T00:00:00.636899252Z"}
{"log":"\u0009at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:576)\n","stream":"stderr","time":"2018-03-22T00:00:00.637183801Z"}

Answer 1

From what I could find searching around, seems the connection could have been closed by the AWS load-balancers.
Could you try modifying your configuration as follows:
factory.setRequestedHeartbeat(30);
And see if it resolves your issue?