Consumers from Rabbit were stopped and never recover - spring-boot

Question

My spring boot application has been running with no erros during months using RabbitMq. In the last weeks, my consumer stopped working and I have to restart my application. Everyday I have to restart the application.
I noticed that it happens after this error in my log:
24-07-2021 07:20:08.138 [SimpleAsyncTaskExecutor-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 17002, SQLState: 08006
24-07-2021 07:20:08.295 [SimpleAsyncTaskExecutor-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - IO Error: Socket read interrupted, Authentication lapse 682 ms.
24-07-2021 07:20:10.608 [SimpleAsyncTaskExecutor-1] WARN o.s.a.r.l.ConditionalRejectingErrorHandler - Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method 'protected void br.jus.tjmg.certidaoEmissao.application.queues.EmissaoCriminalHistoricaConsumer.processaMensagem(br.jus.tjmg.certidao.webservice.ConsultaCertidoesRequest)' threw exception
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:198)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:127)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1477)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1400)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1387)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1366)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:848)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:832)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:78)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1073)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:450)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:378)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:474)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:289)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at br.jus.tjmg.certidaoEmissao.application.services.EmissaoCertidaoService$$EnhancerBySpringCGLIB$$46f4e1c7.atualizarExecucaoCertidao(<generated>)
at br.jus.tjmg.certidaoEmissao.application.queues.EmissaoCriminalHistoricaConsumer.processaMensagem(EmissaoCriminalHistoricaConsumer.java:35)
at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:181)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:114)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:51)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:190)
... 10 common frames omitted
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:115)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:109)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:136)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:254)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:262)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:56)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:162)
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:400)
... 25 common frames omitted
Caused by: java.sql.SQLRecoverableException: IO Error: Socket read interrupted, Authentication lapse 682 ms.
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:794)
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:688)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:39)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:691)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:384)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:273)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:198)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:176)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:106)
... 32 common frames omitted
Caused by: java.io.IOException: Socket read interrupted, Authentication lapse 682 ms.
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:790)
... 42 common frames omitted
Caused by: java.net.SocketTimeoutException: Socket read interrupted
at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:152)
at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101)
at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:499)
at oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:1279)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:663)
... 42 common frames omitted
24-07-2021 07:20:20.230 [SimpleAsyncTaskExecutor-1] ERROR o.s.a.r.l.SimpleMessageListenerContainer - Stopping container from aborted consumer
24-07-2021 07:20:20.525 [SimpleAsyncTaskExecutor-1] INFO o.s.a.r.l.SimpleMessageListenerContainer - Waiting for workers to finish.
24-07-2021 07:20:20.584 [SimpleAsyncTaskExecutor-1] INFO o.s.a.r.l.SimpleMessageListenerContainer - Successfully waited for workers to finish.
I would like to avoid my consumer gets down. I have tried many different solutions, for example to restart my application after some events, but they didn't work.
public static void main(String[] args) {
context = SpringApplication.run(CertidaoApplication.class, args);
}
public static void restart() {
ApplicationArguments args = context.getBean(ApplicationArguments.class);
Thread thread = new Thread(() -> {
context.close();
context = SpringApplication.run(CertidaoApplication.class, args.getSourceArgs());
});
thread.setDaemon(false);
thread.start();
}
#EventListener
public void handleEvent(ListenerContainerConsumerTerminatedEvent e) {
logger.info("Restart spring boot. ListenerContainerConsumerTerminatedEvent");
CertidaoApplication.restart();
}
#EventListener
public void handleEvent(ListenerContainerConsumerFailedEvent e) {
logger.info("Restart spring boot. ListenerContainerConsumerFailedEvent");
CertidaoApplication.restart();
}
#EventListener
public void handleEvent(ContextStoppedEvent e) {
logger.info("Restart spring boot. ContextStoppedEvent");
CertidaoApplication.restart();
}
#EventListener
public void handleEvent(ContextClosedEvent e) {
logger.info("Restart spring boot. ContextClosedEvent");
CertidaoApplication.restart();
}
Here my configuration:
#Bean
public RabbitListenerContainerFactory<SimpleMessageListenerContainer> prefetchRabbitListenerContainerFactory(ConnectionFactory rabbitConnectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(rabbitConnectionFactory);
factory.setPrefetchCount(2);
factory.setDefaultRequeueRejected(Boolean.FALSE);
factory.setMessageConverter(jsonMessageConverter());
factory.setMissingQueuesFatal(Boolean.FALSE);
return factory;
}
My consumer:
#RabbitListener(queues = "${queue}", concurrency = "1", containerFactory = "prefetchRabbitListenerContainerFactory")
protected void processMessage(ConsultaRequest message) {
logger.info("Test" + message.getNumber());
Thanks a lot.