How to add default behavior to #KafkaListener - spring

I would like to add custom behavior before my method annoted with #KafkaListener is called.
Actually I'm using an abstract class and the final class is using
#KafkaListener(topics = ....)
public void onMessages(List<ConsumerRecord> records) {
super.onMessages(records);
}
#Override
public void process(ConsumerRecord record) {
// Called by abstract class to really process the message (one by one)
}
But I also need to configure the abstract class in a #PostConstruct.
What would be the best approach ?
I would prefer just decorate the default container and use it with something like that :
#MyCustomKafkaLister(topics = ....)
public void onMessage(ConsumerRecord record) {
// Just handle the message
}
Or by customizing the MessageListenerFactory to create a CustomMessageListener inherited from the default one which will call my method annoted by #KafkaListener after some custom behavior.
But I don't know how.
Edit 1
I want my abstract processing to do the following :
public void process(List<ConsumerRecord> records) {
for (ConsumerRecord<K, V> record : records) {
// Check message
try {
if (record.value() == null) {
checkDeser(record, ErrorHandlingDeserializer2.VALUE_DESERIALIZER_EXCEPTION_HEADER);
}
if (record.key() == null) {
checkDeser(record, ErrorHandlingDeserializer2.KEY_DESERIALIZER_EXCEPTION_HEADER);
}
} catch (DeserializationException ex) {
this.deadLetterPublishingRecoverer.accept(record, ex);
LOGGER.error("Deserialization error recovered to DLT.", ex);
}
// Process message
try {
// Here I'm calling the original #KafkaListener aka the subclass
myRealListenerObject.processOneByOne(record);
} catch (Exception ex) {
LOGGER.warn("Exception while processing record. Key : {}", record.key(), ex);
handleException(record, ex);
}
}
}
This is calling "myRealListenerObject.processOneByOne(record);" which should be my listener implementation using #KafkaListener (or #CustomKafkaListener)
Edit 2
I would like my listeners to be like
#CustomKafkaListeners(topics = "myTopic", ...)
public void process(ConsumerRecord record) {
// Do my stuff
}
rather than having something like that for every listeners :
#KafkaListeners(topics = "myTopic", ...)
public void process(List<ConsumerRecord> records) {
for (ConsumerRecord record : records) {
try {
if (record.value() == null) {
checkDeser(record, ErrorHandlingDeserializer2.VALUE_DESERIALIZER_EXCEPTION_HEADER);
}
if (record.key() == null) {
checkDeser(record, ErrorHandlingDeserializer2.KEY_DESERIALIZER_EXCEPTION_HEADER);
}
} catch (DeserializationException ex) {
this.deadLetterPublishingRecoverer.accept(record, ex);
LOGGER.error("Deserialization error recovered to DLT.", ex);
}
// Process message
try {
// Do my stuff
} catch (Exception ex) {
LOGGER.warn("Exception while processing record. Key : {}", record.key(), ex);
MyExceptionHandler.handleException(record, ex);
}
}
}

You can perform that logic using a FilteringBatchMessageListenerAdapter with a custom RecordFilterStrategy to check for the deserialization exceptions.
Simply add the adapter to the listener container factory.

Related

Springboot Kafka #Listener consumer pause/resume not working

I have a springboot Kafka Consumer & Producer. The consumer is expected to read data from topic 1 by 1, process(time consuming) it & write it to another topic and then manually commit the offset.
In order to avoid rebalancing, I have tried to call pause() and resume() on KafkaContainer but the consumer is always running & never responds to pause() call, tried it even with a while loop and faced no success(unable to pause the consumer). KafkaListenerEndpointRegistry is Autowired.
Springboot version = 2.6.9, spring-kafka version = 2.8.7
#KafkaListener(id = "c1", topics = "${app.topics.topic1}", containerFactory = "listenerContainerFactory1")
public void poll(ConsumerRecord<String, String> record, Acknowledgment ack) {
log.info("Received Message by consumer of topic1: " + value);
String result = process(record.value());
producer.sendMessage(result + " topic2");
log.info("Message sent from " + topicIn + " to " + topicOut);
ack.acknowledge();
log.info("Offset committed by consumer 1");
}
private String process(String value) {
try {
pauseConsumer();
// Perform time intensive network IO operations
resumeConsumer();
} catch (InterruptedException e) {
log.error(e.getMessage());
}
return value;
}
private void pauseConsumer() throws InterruptedException {
if (registry.getListenerContainer("c1").isRunning()) {
log.info("Attempting to pause consumer");
Objects.requireNonNull(registry.getListenerContainer("c1")).pause();
Thread.sleep(5000);
log.info("kafkalistener container state - " + registry.getListenerContainer("c1").isRunning());
}
}
private void resumeConsumer() throws InterruptedException {
if (registry.getListenerContainer("c1").isContainerPaused() || registry.getListenerContainer("c1").isPauseRequested()) {
log.info("Attempting to resume consumer");
Objects.requireNonNull(registry.getListenerContainer("c1")).resume();
Thread.sleep(5000);
log.info("kafkalistener container state - " + registry.getListenerContainer("c1").isRunning());
}
}
Am I missing something? Could someone please guide me with the right way of achieving the required behaviour?
You are running the process() method on the listener thread so pause/resume will not have any effect; the pause only takes place when the listener thread exits the listener method (and after it has processed all the records received by the previous poll).
The next version (2.9), due later this month, has a new property pauseImmediate, which causes the pause to take effect after the current record is processed.
You can try like this. This work for me
public class kafkaConsumer {
public void run(String topicName) {
try {
Consumer<String, String> consumer = new KafkaConsumer<>(config);
consumer.subscribe(Collections.singleton(topicName));
while (true) {
try {
ConsumerRecords<String, String> consumerRecords = consumer.poll(Duration.ofMillis(80000));
for (TopicPartition partition : consumerRecords.partitions()) {
List<ConsumerRecord<String, String>> partitionRecords = consumerRecords.records(partition);
for (ConsumerRecord<String, String> record : partitionRecords) {
kafkaEvent = record.value();
consumer.pause(consumer.assignment());
/** Implement Your Business Logic Here **/
Once your processing done
consumer.resume(consumer.assignment());
try {
consumer.commitSync();
} catch (CommitFailedException e) {
}
}
}
} catch (Exception e) {
continue;
}
}
} catch (Exception e) {
}
}

How to accept http requests after shutdown signal in Quarkus?

I tried this:
void onShutdown(#Observes final ShutdownEvent event) throws InterruptedException {
log.infof("ShutdownEvent received, waiting for %s seconds before shutting down", shutdownWaitSeconds);
TimeUnit.SECONDS.sleep(shutdownWaitSeconds);
log.info("Continue shutting down");
}
But after receiving ShutdownEvent Quarkus already responds with 503 to http requests. Looks like this could be done with ShutdownListener in preShutdown method. I have implemented this listener but it does not get called yet. How do I register ShutdownListener?
Use case here is OpenShift sending requests to terminating pod.
Option 1: Create Quarkus extension
Instructions are here. ShutdownController is my own class implementing ShutdownListener where I have a sleep in preShutdown method.
class ShutdownControllerProcessor {
#BuildStep
FeatureBuildItem feature() {
return new FeatureBuildItem("shutdown-controller");
}
#BuildStep
ShutdownListenerBuildItem shutdownListener() {
// Called at build time. Default constructor will be called at runtime.
// Getting MethodNotFoundException when calling default constructor here.
return new ShutdownListenerBuildItem(new ShutdownController(10));
}
}
Option 2: Modify ShutdownRecorder private static final field
New shutdown listener can be added using reflection. This is a bit ugly solution.
registerIfNeeded() need to be called after Quarkus startup, for example with timer 1 second after #PostConstruct.
#ApplicationScoped
public class ListenerRegisterer {
public void registerIfNeeded() {
try {
tryToRegister();
} catch (NoSuchFieldException | IllegalAccessException e) {
throw new IllegalStateException(e);
}
}
private void tryToRegister() throws NoSuchFieldException, IllegalAccessException {
final var field = ShutdownRecorder.class.getDeclaredField("shutdownListeners");
field.setAccessible(true);
final var listeners = (List<ShutdownListener>) field.get(null);
if (listeners != null && !listeners.toString().contains("ShutdownController")) {
listeners.add(new ShutdownController(10));
setFinalStatic(field, listeners);
}
}
private static void setFinalStatic(final Field field, final Object newValue) throws NoSuchFieldException, IllegalAccessException {
field.setAccessible(true);
final var modifiersField = Field.class.getDeclaredField("modifiers");
modifiersField.setAccessible(true);
modifiersField.setInt(field, field.getModifiers() & ~Modifier.FINAL);
field.set(null, newValue);
}
}

Spring retry exceptionexpression

So I'm trying to retry for specific exceptions and created a bean which has shouldRetry(Throwable t) function. The function returns true if exception has to be retried, otherwise false.
But What I'm observing is shouldRetry(Throwable t) is executing twice(log is printing twice) for one retry attempt, however serviceImpl from where exception is being thrown is executing only once for one retry attempt.
Could someone please let me know if I'm doing something wrong here, or is it the default behavior/bug with spring retry itself.
#Component("dbRecoverableExceptionHandler")
#RequiredArgsConstructor(onConstructor = #__(#Autowired))
#Slf4j
public class DBRecoverableExceptionHandler {
private final Environment environment;
private final MultiTaggedCounter exceptionRetryCounter;
public Boolean isRetryable(Throwable t) {
String[] recoverableExceptionClasses = environment
.getRequiredProperty("db-recoverable-exception-classes", String[].class);
for (String s1 : recoverableExceptionClasses) {
if (t.getClass().getSimpleName().contains(s1)) {
exceptionRetryCounter.increment(1, s1);
log.warn("Retrying for exception " + t.toString());
return true;
}
}
return false;
}
}
#Retryable(exceptionExpression = "#{#dbRecoverableExceptionHandler.isRetryable(#root)}",
maxAttemptsExpression = "#{${max-attempts}}",
backoff = #Backoff(delayExpression = "#{${retry-backoff-delay-time}}",
multiplierExpression = "#{${retry-backoff-multiplier}}"))
It is as expected.
The method will be called by the RetryTemplate twice for each execution...
while (canRetry(retryPolicy, context) && !context.isExhaustedOnly()) {
try {
if (this.logger.isDebugEnabled()) {
this.logger.debug("Retry: count=" + context.getRetryCount());
}
// Reset the last exception, so if we are successful
// the close interceptors will not think we failed...
lastException = null;
return retryCallback.doWithRetry(context);
}
catch (Throwable e) {
lastException = e;
try {
registerThrowable(retryPolicy, state, context, e);
}
catch (Exception ex) {
throw new TerminatedRetryException("Could not register throwable",
ex);
}
finally {
doOnErrorInterceptors(retryCallback, context, e);
}
if (canRetry(retryPolicy, context) && !context.isExhaustedOnly()) {
...
The first call to canRetry() (in the while loop) is skipped on the very first call since there is no exception yet, on subsequent iterations, when the method throws an exception, it is called twice.

EJB3.0 transaction fail inside a method in successive transaction taking place

I am processing three transaction inside a single method in stateless container managed bean .i want to persist three transaction while if one throws exception other two should complete their respective transaction ,error is that if first or any one is throwing exception other two are to executing please give some helpful suggestion
public void allocateSubjectToStudent(SubjectAllocatedToStudentDto dto)throws Exception {
logger.info("allocateSubjectToStudent method entry :");
List<Subject> coreList=dto.getCoreList();
Iterator<Subject> iterator=coreList.iterator();
while(iterator.hasNext()){
logger.info("inside while :");
SubjectAllocatedToStudentBo bo=new SubjectAllocatedToStudentBo();
bo.setBacthId(dto.getBacthId());
bo.setSemester(dto.getSemester());
bo.setStudentId(dto.getStudentId());
Subject subject=iterator.next();
bo.setSubjectName(subject.getSubjectName());
bo.setSubjectType(subject.getAbbreviation());
try{
manager.persist(bo);
}
catch(javax.persistence.PersistenceException e){
Throwable t = e.getCause();
while ((t != null) && !(t instanceof org.hibernate.exception.ConstraintViolationException)) {
t = t.getCause();
}//while
if (t instanceof org.hibernate.exception.ConstraintViolationException) {
throw new Exception("Core subject already allocated to student");
} //end of if
}//end of catch
}//end of while
List<Subject> departmentallist=dto.getDepartmentList();
Iterator<Subject> iterator1=departmentallist.iterator();
while(iterator1.hasNext()){
logger.info("inside while :");
SubjectAllocatedToStudentBo bo=new SubjectAllocatedToStudentBo();
bo.setBacthId(dto.getBacthId());
bo.setSemester(dto.getSemester());
bo.setStudentId(dto.getStudentId());
Subject subject=iterator1.next();
bo.setSubjectName(subject.getSubjectName());
bo.setSubjectType(subject.getAbbreviation());
try{
manager.persist(bo);
}
catch(javax.persistence.PersistenceException e){
Throwable t = e.getCause();
while ((t != null) && !(t instanceof org.hibernate.exception.ConstraintViolationException)) {
t = t.getCause();
}//while
if (t instanceof org.hibernate.exception.ConstraintViolationException) {
throw new Exception("InterDepartmental subject already allocated to student");
} //end of if
}//end of catch
}//end of while
List<Subject> electiveList=dto.getElectiveList();
Iterator<Subject> iterator2=electiveList.iterator();
while(iterator2.hasNext()){
logger.info("inside while :");
SubjectAllocatedToStudentBo bo=new SubjectAllocatedToStudentBo();
bo.setBacthId(dto.getBacthId());
bo.setSemester(dto.getSemester());
bo.setStudentId(dto.getStudentId());
Subject subject=iterator2.next();
bo.setSubjectName(subject.getSubjectName());
bo.setSubjectType(subject.getAbbreviation());
try{
manager.persist(bo);
}
catch(javax.persistence.PersistenceException e){
Throwable t = e.getCause();
while ((t != null) && !(t instanceof org.hibernate.exception.ConstraintViolationException)) {
t = t.getCause();
}//while
if (t instanceof org.hibernate.exception.ConstraintViolationException) {
throw new Exception("Elective subject already allocated to student");
} //end of if
}//end of catch
}//end of while
logger.info("allocateSubjectToStudent method exit :");
} //end of method
create three different method all with TranscationAttributeType REQUIRES_NEW
Please find below code snippet for EJB3 Bean
public void doYourWork()
{
a();
b();
c();
}
#TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public void a()
{
try
{
//Do the first transaction here
}catch(Exception e)
{
}
}
#TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public void b()
{
try
{
//Do the second transaction here
}catch(Exception e)
{
}
}
#TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public void c()
{
try
{
//Do the third transaction here
}catch(Exception e)
{
}
}
Within a single method invocation there is only a single transaction active. To achieve what you want, you must perform the three operations in different transactions. This would require one more level of abstracttion.
public class MyFreshTransaction {
#TransactionAttribute(REQUIRES_NEW)
public void updateO() {
//do something
}
#TransactionAttribute(REQUIRES_NEW)
public void update1() {
//do something
}
#TransactionAttribute(REQUIRES_NEW)
public void update2() {
//do something
}
}
#Stateless
public class MyTransactionProcessor {
#EJB
private MyFreshTransaction freshTransaction;
public void processTransaction() {
try {
//The current transaction context will be suspended, and a new one invoked
//if the new one fails and is rollback, the current one is not affected.
//you can then handle the exception, by rethrowing the exception,in which case
//the current transaction will also be rolled back, or continue based on your logic.
freshTransaction.update0();
} catch (Exception ex ) {//handle}
try {
//The current transaction context will be suspended, and a new one invoked
//if the new one fails and is rollback, the current one is not affected.
//you can then handle the exception, by rethrowing the exception,in which case
//the current transaction will also be rolled back, or continue based on your logic.
freshTransaction.update1();
} catch (Exception ex ) {//handle}
try {
//The current transaction context will be suspended, and a new one invoked
//if the new one fails and is rollback, the current one is not affected.
//you can then handle the exception, by rethrowing the exception,in which case
//the current transaction will also be rolled back, or continue based on your logic.
freshTransaction.update2();
} catch (Exception ex ) {//handle}
}
}
Note that if any of the update transaction was successful, and the the parent transaction is rolled back, it will not affect the status of the 'child' transactions, as they had already been committed and their effects (if DB effects) will be committed too.
Read on Java EE Transactions Java EE Transactions

how to populate cvc-complex-type error information in mule

I have mule message filters to validate SOAP Request against schema,
my question is:
how to log error details of 'which element is have parse exception (cvc-complex-type). because catch block is always trowing exception details: "Message has been rejected by filter". how can i show exact exception?. so that user will enter correct data on request.
write a customfilterclass as a child of SchemaValidationfilter.
override accept method and throw soap fault.
public class CustomSchemaValidationFilter extends SchemaValidationFilter {
#Override
public boolean accept(MuleMessage muleMessage) {
return validateBody(muleMessage);
}
public boolean validateBody(MuleMessage message) throws Fault {
Source source = null;
try {
source = loadSource(message);
} catch (Exception e) {
throw getFaultObj(e);
}
try {
createValidator().validate(source);
} catch (SAXException e) {
throw getFaultObj(e);
} catch (IOException e) {
throw getFaultObj(e);
}
return true;
}
private Fault getFaultObj(Exception e) {
return new SoapFault(e.getMessage(), e, new QName(
DEFAULT_SCHEMA_LANGUAGE));
}
}
//and add in mule flow following tags to validate filter.
<custom-filter class="org.mule.module.xml.filters.CustomSchemaValidationFilter">
<spring:property name="schemaLocations" value="${approvalHistorySchemaLocation}" />
</custom-filter>**

Resources