How to handle AmqpConnectException on pod restart of a spring boot app? - spring-boot

Question

We're using a (reactive) Spring Boot app with RabbitMQ (Spring AMQP), running on a Kubernetes cluster, and lately we noticed some weird behavior in the logs of the application.
When we make a new deploy to the kubernetes cluster we keep getting the following error repeatedly with the correspond exception stack trace until the new pods are up and old ones destroyed:
Failed to check/redeclare auto-delete queue(s).
org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:61)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:602)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:724)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.createConnection(ConnectionFactoryUtils.java:252)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2163)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2136)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2116)
at org.springframework.amqp.rabbit.core.RabbitAdmin.getQueueInfo(RabbitAdmin.java:407)
at org.springframework.amqp.rabbit.core.RabbitAdmin.getQueueProperties(RabbitAdmin.java:391)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.attemptDeclarations(AbstractMessageListenerContainer.java:1914)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.redeclareElementsIfNecessary(AbstractMessageListenerContainer.java:1895)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.initialize(SimpleMessageListenerContainer.java:1347)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1193)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:660)
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333)
at java.base/java.net.Socket.connect(Socket.java:648)
at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:60)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1220)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1170)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.connectAddresses(AbstractConnectionFactory.java:640)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.connect(AbstractConnectionFactory.java:615)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:565)
... 12 common frames omitted
We can't seem to find the source of the issue.
I'd love if anyone have any ideas. Thanks in advance!

Answer 1

Caused by: java.net.ConnectException: Connection refused - means the broker can't be reached.
That is normal behavior - the containers are trying to reconnect. You can control the frequency by setting the container's recoveryBackOff property. By default, the container will attempt to connect every 5 seconds, using a FixedBackOff - you can use a fixed backoff with a longer retry interval, or an ExponentialBackOff that will increase the delay each time.
/**
* Specify the {#link BackOff} for interval between recovery attempts.
* The default is 5000 ms, that is, 5 seconds.
* With the {#link BackOff} you can supply the {#code maxAttempts} for recovery before
* the {#link #stop()} will be performed.
* #param recoveryBackOff The BackOff to recover.
* #since 1.5
*/
public void setRecoveryBackOff(BackOff recoveryBackOff) {