Error Handling with Apache Camel and ActiveMQ - so breaking out of pipeline for exchange - spring-boot

I've been back and forth with an issue on our system that even with some research around the forums and several tests, we can't seem to be able to fix.
I'll try to be as clear as I can with what we are dealing with
We have a main service with a route that reads from an activemq queue ( spring boot with embedded broker ) sends it to a Route(B) and then ships everything to a final Route(C) . Route(B) is on a dependency of the service.
Camel Version: 3.3.0
Spring-boot version: 2.3.3.RELEASE
Route A:
onException(Exception::class.java)
.handled(true)
.bean("foo.ErrorProcessor", "processError")
from("activemq:queue:myqueue")
.routeId("myroute")
.to("direct:my_external_route")
.to(ExchangePattern.InOnly,"direct:myroute_result")
Route B:
onException(Exception::class.java)
.handled(true)
.bean("foo.ErrorProcessor", "processError")
from("direct:my_external_route")
.routeId("my_external_route")
.process {something()} //This processor can throw exceptions that are treated in our processor
Route C:
from("direct:myroute_result")
.process(someProcess())
.to(ExchangePattern.InOnly,"activemq:queue:results_queue")
Spring Boot activemq configs
spring:
jmx:
enabled: true
activemq:
broker-url: vm://localhost?broker.persistent=false,useShutdownHook=false
in-memory: true
non-blocking-redelivery: true
packages:
trust-all: false
trusted: com.mypackage
pool:
block-if-full: true
block-if-full-timeout: -1
enabled: false
idle-timeout: 30000
max-connections: 10
time-between-expiration-check: -1
use-anonymous-producers: true
Everything runs very well and smoothly when B's processors do not throw exceptions. When it does, even though they are being treated and a normal object is being returned in the message body, all we have on the logs is
2021-04-10 15:33:32.354 DEBUG [#1 - JmsConsumer[consumerName]] o.a.c.p.Pipeline
: Message exchange has failed: so breaking out of pipeline for exchange: Exchange[ID-1234] Handled by the error handler. {}
We even added a default error handler to our activemq connection factory but nothing happens there as well. We have a DLQ consumer who also does not seems to get anything. The error processor on routeA also does not catches anything which is expected since the exception was handled previously.
Has anyone ever had this issue or similar ? I know that some issues between Camel and the JMS component regarding error handling were raised in the past but we are struggling to understand what is the root of this issue.
Thanks in advance,
Pedro

Probably what you are looking for is the continued option on your Route B exception clause. This option allows you to continue routing to the original route as if the exception did not occur. Do not use the handled option as it will not allow routing to the original route but break out.
So your Route B should be defined as something like this:
onException(Exception::class.java) .continued(true)
.bean("foo.ErrorProcessor", "processError")
from("direct:my_external_route")
.routeId("my_external_route")
.process {something()}
Refer the camel documentation for more details: CAMEL EXCEPTION CLAUSE

Related

WrongTargetException in Hazlecast

I have a multinode (2 nodes) setup of Weblogic 12c. In that I have deployed a spring boot application having microservices. The spring boot application is having the structure like Controller <--> Service <--> DAO. The #Cacheable annotation is used for few methods in "Service" classes/ layer. When endpoint it called, it reaches to controller and then it calls service class method. The issue is some of the calls are failing with below exception.
2023-01-15T14:55:31,122 ERROR [tuning)'] iServiceExceptionHandlerAdvice [, , ]
| Exception is encountered, message - WrongTarget! local: Member [XX.XX.XX.XX]:5981 - 9bXXX334-bf12-4XXb-afb6-XXXXXXX05442 this, expected-target: null, partitionId: 121, replicaIndex: 0, operation: com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
com.hazelcast.spi.exception.WrongTargetException: WrongTarget! local: Member [XX.XX.XX.XX]:5981 - 9bXXX334-bf12-4XXb-afb6-XXXXXXX05442 this, expected-target: null, partitionId: 121, replicaIndex: 0, operation: com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
at com.hazelcast.spi.impl.operationservice.impl.Invocation.newTargetNullException(Invocation.java:336) ~[hazelcast-3.12.10.jar:3.12.10]
at com.hazelcast.spi.impl.operationservice.impl.PartitionInvocation.newTargetNullException(PartitionInvocation.java:94) ~[hazelcast-3.12.10.jar:3.12.10]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.initInvocationTarget(Invocation.java:289) ~[hazelcast-3.12.10.jar:3.12.10]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:614) ~[hazelcast-3.12.10.jar:3.12.10]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.access$400(Invocation.java:96) ~[hazelcast-3.12.10.jar:3.12.10]
at com.hazelcast.spi.impl.operationservice.impl.Invocation$InvocationRetryTask.run(Invocation.java:827) ~[hazelcast-3.12.10.jar:3.12.10]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_141]
The above exception is from one node. Other node also fails intermittently with exact same exception, only the local: Member [XX.XX.XX.XX]:5982 will be different port like incremented by 1. This behavior goes away when I restart the deployed application (the spring boot application). It does not need to restart the complete weblogic server. Moreover this sometimes starts working without doing anything.
As per documentation of Hazelcast, https://docs.hazelcast.org/docs/3.7.8/manual/html-single/index.html says
WrongTargetException: Thrown when an operation is executed on the wrong machine, usually because the partition that operation belongs to has been moved to some other member.
There is no clear pattern to replicate this issue on local environment.
The alternatives I am thinking are
I can add try-catch in controller and log the details and throw some graceful exception. But still the data will be empty.
In try-catch in controller, I can call another method performing same operation but will not have #Cachable annotation. So that it will do everything same and fetch the data. put it in response and send it back.
Get rid of Hazelcast cache and use some other cache.
Is there any setting in Hazelcast which can do this (something similar to option 2) with some configuration. Instead of failing completely, it will do the invocation of method?

Best way to disable #KafkaListeners

My application contains several #KafkaListeners.
When I run the app locally or run some #SpringBootTests (without kafka), I got my log polluted with
[org.apache.kafka.clients.NetworkClient] - [Consumer clientId=consumer-app-1, groupId=app] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
For such cases, I would like to disable listeners, as they are useless anyways.
I know I can do it with
#KafkaListener(... autoStartup = "${consumer.auto.start:false}")
but adding this property to all of the consumers feels cumbersome.
Unfortunately, there is no general property like spring.kafka.consumer.group-id that would affect all the consumers.
Is there a better way to achieve the same?
Thanks
If you are not interested in Spring for Apache Kafka activity in some of your Spring Boot tests, just consider to exclude its auto-configuration:
#SpringBootTest
#EnableAutoConfiguration(exclude=KafkaAutoConfiguration.class)
This way the #EnableKafka won't be applied to your application context and no KafkaListenerContainers are going to be registered to start on context refresh.
UPDATE
Another way, for the reason you explained, a respective configuration property, which you can add to application.properties, which could be a test scope-specific, or profile-based. See more info in Spring Boot docs: https://docs.spring.io/spring-boot/docs/2.1.9.RELEASE/reference/html/using-boot-auto-configuration.html#using-boot-disabling-specific-auto-configuration
The property name is spring.autoconfigure.exclude - you have to specify fully-qualified class name. Therefore for your use-case it is:
spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.kafka.org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration
See also how to have test-specific application.properties or divided by the profile: https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.profiles

Recovering Kafka clients (consumers/producers) after they went down

At the company i work with we use Spring for Kafka without authentication and lately we did some experiments to setup the security in Kafka and we enabled authentication for a brief moment which cause a crush in all our consumers/producers within our microservices ! (the microservices stayed up)
The exception :
Authorization Exception and no authorizationExceptionRetryInterval set
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: foo-group
after some researchs we found out that this is the expected behavior by kafka clients and we needed to set the authorizationExceptionRetryInterval property
public void setAuthorizationExceptionRetryInterval​(java.time.Duration authorizationExceptionRetryInterval)
Set the interval between retries after AuthorizationException is
thrown by KafkaConsumer. By default the field is null and retries are
disabled. In such case the container will be stopped. The interval
must be less than max.poll.interval.ms consumer property.
Here is some other useful links
Setting authorizationExceptionRetryInterval for Spring Kafka
Why does the spring KafkaConsumer suspend all consumption from n topics when one fails to authorize
What i want to know is :
Is a failed authentication the only case when
consumers/producers goes down ?
If there are some other cases, how to make sure that our
consumers/producers recover without human intervention (restarting
the microservices) ? In other word how to check if the
consumers/producers are up and restart them otherwise ?
Containers are stopped only under the following circumstances:
AuthorizationException with no authorizationExceptionRetryInterval
NoOffsetForPartitionException - thrown when ConsumerConfig.AUTO_OFFSET_RESET_CONFIG is not earliest or latest and there is no existing offset for a partition with this consumer group.
FencedInstanceIdException - using transactions and static group members (meaning some other instance is using this instance id).
StopAfterFenceException - when stopContainerWhenFenced is true (default false) - only applies with transactions
Any Error (such as OOME)

Starting Spring Boot application without check for Kafka Server

I have got an application that uses SpringBoot 2.10.0.Release and kafka in the version 2.10.0. The application has got a simple producer and consumer: The sender works with KafkaTemplate and the consumer with KafkaListener.
What I try to achieve is to be able to start the SpringBoot application even if the KafkaServer is not running.
Currently without a running KafkaBroker the application cannot be started with this error message:
org.springframework.context.ApplicationContextException:
Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry';
nested exception is org.apache.kafka.common.errors.TimeoutException
Is there a way to achieve this and if yes could anybody give me hint or a keyword how to manage this?
When running the Spring-Boot application with a KafkaListener, the listener will per default try to listen to Kafka. If the KafkaBroker is invalid or missing, then you will get a org.apache.kafka.common.KafkaException.
You can change the default behaviour of the container factory by setting the autoStartup property to false. One way to do this is by adding autoStartup = "false" element to your KafkaListener annotation:
#KafkaListener(topics = "some_topic", autoStartup = "false")
public void fooEventListener(){
Now your spring boot application will start. You will still get an error when trying to use the KafkaListener if the broker is still down or invalid, but you will now be able to handle the error within your Java code instead of a Spring Boot server crash.
Documentation about KafkaListner autoStartup element.
It have to be mentioned that the error you are receiving (TimeoutException) is not because the broker is down, it is what Kafka will throw if the buffer is full.
The batch records will then be removed from the queue and will not be delivered to the broker. This error will not be the reason for you application using Kafka not to start.

Queue listener fails if queue not present Spring-Rabbit

We are trying to use spring-rabbit to create a message listener, in our spring boot application. But, we are getting an issue during receiving message if the queue doesn't exist, the application throws an error, which is called as passive declaration.
Channel error on connection (172.13.1.3:49352 -> 172.13.1.7:5672, vhost: '/', user: 'guest'), channel 1:
message_queue_1_1 | operation queue.declare caused a channel exception not_found: no queue 'add' in vhost '/'
What we want is to declare a queue actively, i.e. if the queue doesn't exist, it should be declared inside the mq.
To declare AMQP objects on the Broker automatically from the application, you really need to configure them as beans and also have a special AmqpAdmin bean to perform the hard declaration logic. The documentation has all the required explanation: https://docs.spring.io/spring-amqp/docs/2.0.4.RELEASE/reference/html/_reference.html#broker-configuration

Resources