I have the below configuration. When the message is invalid I want to send an email and for errors I want to save it in database. How can I handle this in errorHandler() ?
#Configuration
#EnableKafka
public class KafkaConsumerConfig implements KafkaListenerConfigurer{
#Bean
ErrorHandler errorHandler() {
return new SeekToCurrentErrorHandler((rec, ex) ->
{ dbService.saveErrorMsg(rec); }
,new FixedBackOff(5000, 3)) ;
}
#Override
public void configureKafkaListeners(KafkaListenerEndpointRegistrar registrar) {
registrar.setValidator(this.validator);
}
#KafkaListener(topics = "mytopic", concurrency = "3", groupId = "mytopic-1-groupid")
public void consumeFromTopic1(#Payload #Valid ValidatedClass val, ConsumerRecordMetadata meta) throws Exception
{
dbservice.callDB(val,"t");
}
I presume your emai code is in dbService.saveErrorMsg.
Spring Boot should automatically detect the ErrorHandler #Bean and wire it into the container factory.
See Boot's KafkaAnnotationDrivenConfiguration class and ConcurrentKafkaListenerContainerFactoryConfigurer.
Related
I am trying to consume a JSON message using spring kafka. The message which is consumed by the consumer is like this.
{
"EventHeader": {
"entityName": "Account",
"changedFields": ["Id", "Name"]
},
"NewFields": {
"Id": "001",
"Name": "Test Account",
},
"OldFields": {}
}
So far I have created classes for "EventHeader", "NewFields","OldFields" ,and for "KafkaPayload". And also I have created a custom deserializer to deserialize this JSON payload.Here is my custom deserializer.
public class CustomDeserailizer <T extends Serializable> implements Deserializer<T> {
private ObjectMapper objectMapper = new ObjectMapper();
public static final String VALUE_CLASS_NAME_CONFIG = "value.class.name";
#Override
public void configure(Map<String, ?> configs, boolean isKey) {
Deserializer.super.configure(configs, isKey);
}
#Override
public T deserialize(String topic, byte[] objectData) {
return (objectData == null) ? null : (T) SerializationUtils.deserialize(objectData);
}
#Override
public T deserialize(String topic, Headers headers, byte[] data) {
return Deserializer.super.deserialize(topic, headers, data);
}
#Override
public void close() {
Deserializer.super.close();
}
}
I have set the consumer configurations as below.
public class KafkaConfig {
#Bean
public KafkaConsumer<String, KafkaPayload> consumerFactory(){
Properties config = new Properties();
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
config.put(ConsumerConfig.GROUP_ID_CONFIG, "groupId");
config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, CustomDeserializer.class);
return new KafkaConsumer<>(config);
}
}
Now I need to show the consumed message through a #KafkaListener setting the consumer into ConsumerFactory. But I don't understand how to do that. This is my first time using kafka.So could anyone give me some idea about this?
This is how I am trying to do that.
#Bean
public ConcurrentKafkaListenerContainerFactory<String, KafkaPayload> kafkaListener(){
ConcurrentKafkaListenerContainerFactory factory = new ConcurrentKafkaListenerContainerFactory();
factory.setConsumerFactory(consumerFactory());
return factory;
}
This is my KafkaListener
public class ConsumerService {
#KafkaListener(topics = "Topic", groupId = "sample-group",containerFactory = "kafkaListener")
public void consume(KafkaPayload kafkaPayload){
System.out.println("Consumed Message :"+ kafkaPayload);
}
}
Since you are using Spring Boot, just set the value deserializer class name as a property and Boot will automatically wire it into the container factory for your #KafkaListener. No need to define your own consumer factory or container factory.
spring.kafka.consumer.value-deserializer=com.acme.CustomDeserializer
https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#application-properties.integration.spring.kafka.consumer.value-deserializer
I try to convert the "Hello World example" from Spring Integration samples (https://github.com/spring-projects/spring-integration-samples/tree/master/basic/helloworld) from XML, to Java Configuration, (so with the #Configuration annotation).
The configuration class looks like this :
#Configuration
#EnableIntegration
public class BasicIntegrationConfig{
#Bean
public DirectChannel inputCHannel() {
return new DirectChannel();
}
#Bean
public QueueChannel outputChannel() {
return new QueueChannel();
}
#Bean
#ServiceActivator(inputChannel= "inputChannel", outputChannel= "outputChannel" )
public MessageHandler fileWritingMessageHandler() {
MessageHandler mh = new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println("Message payload: " + message.getPayload());
}
};
return mh;
}
}
To test it, I use the main() supplied from sample project :
DirectChannel fileChannel = applicationContext.getBean("inputChannel", DirectChannel.class);
QueueChannel outputChannel = applicationContext.getBean("outputChannel", QueueChannel.class);
System.out.println("********** SENDING MESSAGE");
fileChannel.send(new GenericMessage<>("test"));
System.out.println(outputChannel.receive(0).getPayload());
I see in the console "Message payload: test", but unfortunately, I don't receive the message on the outputchannel (I have a NullPointerException on outputChannel.receive(0).
Do you have an idea why the Service Activator does not send the message to the output channel?
Your MessageHandler returns void.
You need to subclass AbstractReplyProducingMessageHandler instead.
Thank you Gary, it works perfectly after switching to :
#Bean
#ServiceActivator(inputChannel= "inputChannel")
public AbstractReplyProducingMessageHandler fileWritingMessageHandler() {
AbstractReplyProducingMessageHandler mh = new AbstractReplyProducingMessageHandler() {
#Override
protected Object handleRequestMessage(Message<?> message) {
String payload= (String)message.getPayload();
return "Message Payload : ".concat(payload);
}
};
mh.setOutputChannelName("outputChannel");
return mh;
}
As a side note, I had to remove the output channel attribute in #ServiceActivator annotation, and put it in method body instead (Bean Validation Exception if not).
From this documentation:
Messages in a message group share the same group id, i.e. they have same group identifier property (JMSXGroupID for JMS, _AMQ_GROUP_ID for Apache ActiveMQ Artemis Core API).
I can see why the property originally set via JMSXGroupID becomes _AMQ_GROUP_ID when I browse the messages in the broker with a value of product=paper. However, In my #JmsListener annotated method I can see the _AMQ_GROUP_ID property is missing and the JMSXGroupID is coming through as null in the Message's headers hashmap.
#JmsListener(destination = "${artemis.destination}", subscription = "${artemis.subscriptionName}",
containerFactory = "containerFactory", concurrency = "15-15")
public void consumeMessage(Message<StatefulSpineEvent<?>> eventMessage)
So
My Producer application sends the message to the queue after setting the string property JMSXGroupID to 'product=paper'
I can see _AMQ_GROUP_ID has a value of 'product=paper' when I browse that message's headers in the Artemis UI
When I debug my listener application and look at the map of headers, _AMQ_GROUP_ID is absent and JMSXGroupID has a value of null instead of 'product=paper'.
Is the character '=' invalid or is there something else that can cause this? I'm running out of things to try.
Edit, with new code:
HeaderMapper:
#Component
public class GroupIdMessageMapper extends SimpleJmsHeaderMapper {
#Override
public MessageHeaders toHeaders(Message jmsMessage) {
MessageHeaders messageHeaders = super.toHeaders(jmsMessage);
Map<String, Object> messageHeadersMap = new HashMap<>(messageHeaders);
try {
messageHeadersMap.put("JMSXGroupID", jmsMessage.getStringProperty("_AMQ_GROUP_ID"));
} catch (JMSException e) {
e.printStackTrace();
}
// can see while debugging that this returns the correct headers
return new MessageHeaders(messageHeadersMap);
}
}
Listener:
#Component
public class CustomSpringJmsListener {
protected final Logger LOG = LoggerFactory.getLogger(getClass());
#JmsListener(destination = "local-queue", subscription = "groupid-example",
containerFactory = "myContainerFactory", concurrency = "15-15")
public void receive(Message message) throws JMSException {
LOG.info("Received message: " + message);
}
}
Application code:
#SpringBootApplication
#EnableJms
public class GroupidApplication implements CommandLineRunner {
private static Logger LOG = LoggerFactory
.getLogger(GroupidApplication.class);
#Autowired
private JmsTemplate jmsTemplate;
#Autowired MessageConverter messageConverter;
public static void main(String[] args) {
LOG.info("STARTING THE APPLICATION");
SpringApplication.run(GroupidApplication.class, args);
LOG.info("APPLICATION FINISHED");
}
#Override
public void run(String... args) {
LOG.info("EXECUTING : command line runner");
jmsTemplate.setPubSubDomain(true);
createAndSendObjectMessage("Message1");
createAndSendTextMessage("Message2");
createAndSendTextMessage("Message3");
createAndSendTextMessage("Message4");
createAndSendTextMessage("Message5");
createAndSendTextMessage("Message6");
}
private void createAndSendTextMessage(String messageBody) {
jmsTemplate.send("local-queue", session -> {
Message message = session.createTextMessage(messageBody);
message.setStringProperty("JMSXGroupID", "product=paper");
return message;
});
}
// BEANS
#Bean
public JmsListenerContainerFactory<?> myContainerFactory(ConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
// This provides all boot's default to this factory, including the message converter
configurer.configure(factory, connectionFactory);
// You could still override some of Boot's default if necessary.
factory.setSubscriptionDurable(true);
factory.setSubscriptionShared(true);
factory.setMessageConverter(messagingMessageConverter());
return factory;
}
#Bean
public MessagingMessageConverter messagingMessageConverter() {
return new MessagingMessageConverter(messageConverter, new GroupIdMessageMapper());
}
}
Stack trace of where SimpleJmsHeaderMapper is being called:
toHeaders:130, SimpleJmsHeaderMapper (org.springframework.jms.support)
toHeaders:57, SimpleJmsHeaderMapper (org.springframework.jms.support)
extractHeaders:148, MessagingMessageConverter
(org.springframework.jms.support.converter) access$100:466,
AbstractAdaptableMessageListener$MessagingMessageConverterAdapter
(org.springframework.jms.listener.adapter) getHeaders:552,
AbstractAdaptableMessageListener$MessagingMessageConverterAdapter$LazyResolutionMessage
(org.springframework.jms.listener.adapter) resolveArgumentInternal:68,
HeaderMethodArgumentResolver
(org.springframework.messaging.handler.annotation.support)
resolveArgument:100, AbstractNamedValueMethodArgumentResolver
(org.springframework.messaging.handler.annotation.support)
resolveArgument:117, HandlerMethodArgumentResolverComposite
(org.springframework.messaging.handler.invocation)
getMethodArgumentValues:148, InvocableHandlerMethod
(org.springframework.messaging.handler.invocation) invoke:116,
InvocableHandlerMethod
(org.springframework.messaging.handler.invocation) invokeHandler:114,
MessagingMessageListenerAdapter
(org.springframework.jms.listener.adapter) onMessage:77,
MessagingMessageListenerAdapter
(org.springframework.jms.listener.adapter) doInvokeListener:736,
AbstractMessageListenerContainer (org.springframework.jms.listener)
invokeListener:696, AbstractMessageListenerContainer
(org.springframework.jms.listener) doExecuteListener:674,
AbstractMessageListenerContainer (org.springframework.jms.listener)
doReceiveAndExecute:318, AbstractPollingMessageListenerContainer
(org.springframework.jms.listener) receiveAndExecute:257,
AbstractPollingMessageListenerContainer
(org.springframework.jms.listener) invokeListener:1190,
DefaultMessageListenerContainer$AsyncMessageListenerInvoker
(org.springframework.jms.listener) executeOngoingLoop:1180,
DefaultMessageListenerContainer$AsyncMessageListenerInvoker
(org.springframework.jms.listener) run:1077,
DefaultMessageListenerContainer$AsyncMessageListenerInvoker
(org.springframework.jms.listener) run:748, Thread (java.lang)
Try subclassing the SimpleJmsHeaderMapper and override toHeaders(). Call super.toHeaders(), create a new Map<> from the result; put() any additional headers you want into the map and return a new MessageHeaders from the map.
Pass the custom mapper into a new MessagingMessageConverter and pass that into the container factory.
If you are using Spring Boot, simply add the converter as a #Bean and boot will auto wire it into the factory.
EDIT
After all this; I just wrote an app and it works just fine for me without any customization at all...
#SpringBootApplication
public class So58399905Application {
public static void main(String[] args) {
SpringApplication.run(So58399905Application.class, args);
}
#JmsListener(destination = "foo")
public void listen(String in, MessageHeaders headers) {
System.out.println(in + headers);
}
#Bean
public ApplicationRunner runner(JmsTemplate template) {
return args -> template.convertAndSend("foo", "bar", msg -> {
msg.setStringProperty("JMSXGroupID", "product=x");
return msg;
});
}
}
and
bar{jms_redelivered=false, JMSXGroupID=product=x, jms_deliveryMode=2, JMSXDeliveryCount=1, ...
EDIT2
It's a bug in the artemis client - with 2.6.4 (Boot 2.1.9) only getStringProperty() returns the value of the _AMQ_GROUP_ID property when getting JMSXGroupID.
The mapper uses getObjectProperty() which returned null. With the 2.10.1 client; the message properly returns the value of the _AMQ_GROUP_ID property from getObjectProperty().
Using spring boot and IBM MQ i need to send message to MQ.
In My Spring boot application i have registered MQQueueConnectionFactory as below.
#SpringBootApplication
#EnableJms
public class MainApplication {
public static void main(String[] args) {
new SpringApplicationBuilder(MainApplication.class).web(WebApplicationType.NONE).run(args);
logger.info("init completed...");
}
#Bean
public MQQueueConnectionFactory queueConnectionFactory() {
MQQueueConnectionFactory queueConnectionFactory = new MQQueueConnectionFactory();
try {
queueConnectionFactory.setTransportType(WMQConstants.WMQ_CM_CLIENT);
queueConnectionFactory.setHostName(host);
queueConnectionFactory.setChannel(channel);
queueConnectionFactory.setPort(port);
queueConnectionFactory.setQueueManager(queueManager);
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
return queueConnectionFactory;
}
}
And i have destination resolver as below.
#Component
public class IBMWebSphereMqDestinationResolver extends
DynamicDestinationResolver implements DestinationResolver {
#Override
public Destination resolveDestinationName(Session session, String destinationName, boolean pubSubDomain) throws JMSException {
Destination destination = super.resolveDestinationName(session, destinationName, pubSubDomain);
if (destination instanceof MQDestination) {
MQDestination mqDestination = (MQDestination) destination;
}
return destination;
}
}
I am using JmsTemplate to send message to MQ.
#Service
public class MqServiceImpl implements MqService {
private final Logger logger = LoggerFactory.getLogger(this.getClass());
#Autowired
private JmsTemplate jmsTemplate;
#Autowired
private MQDestination destination;
#Handler
#Override
public void sendMessage(String textMessage) {
logger.info("textMessage {} ", textMessage);
logger.info("destination {} ", destination);
jmsTemplate.convertAndSend(destination, textMessage);
}
}
However when i try to start the application i am getting MQDestination' that could not be found.
service.impl.MqServiceImpl required a bean of type 'com.ibm.mq.jms.MQDestination' that could not be found.
The injection point has the following annotations:
- #org.springframework.beans.factory.annotation.Autowired(required=true)
Action:
Consider defining a bean of type 'com.ibm.mq.jms.MQDestination' in your configuration.
Shouldn't spring try to call DestinationResolver ?
If not how do i register destination for IBM MQ?
You are defining an DestinationResolver bean, but injecting an MQDestination bean. This is why you get the error. You should inject the DestinationResolver and call setDestinationResolver on the JmsTemplate.
But the JmsTemplate resolves the destinations dynamically. This works as well:
public void sendMessage(String textMessage) {
String destination = "MY.QUEUE";
jmsTemplate.convertAndSend(destination, textMessage);
}
I have a Spring Boot application which listens to messages on a Kafka queue. To filter those messages, have the following two classs
#Component
public class Listener implements MessageListener {
private final CountDownLatch latch1 = new CountDownLatch(1);
#Override
#KafkaListener(topics = "${spring.kafka.topic.boot}")
public void onMessage(Object o) {
System.out.println("LISTENER received payload *****");
this.latch1.countDown();
}
}
#Configuration
#EnableKafka
public class KafkaConfig {
#Autowired
private Listener listener;
#Bean
public FilteringMessageListenerAdapter filteringReceiver() {
return new FilteringMessageListenerAdapter(listener, recordFilterStrategy() );
}
public RecordFilterStrategy recordFilterStrategy() {
return new RecordFilterStrategy() {
#Override
public boolean filter(ConsumerRecord consumerRecord) {
System.out.println("IN FILTER");
return false;
}
};
}
}
While messages are being processed by the Listener class, the RecordFilterStrategy implementation is not being invoked. What is the correct way to use FilteringMessageListenerAdapter?
Thanks
The solution was as follows:
No need for the FilteringMessageListenerAdapter class.
Rather, create a ConcurrentKafkaListenerContainerFactory, rather than relying on what Spring Boot provides out of the box. Then, set the RecordFilterStrategy implementation on this class.
#Bean
ConcurrentKafkaListenerContainerFactory<Integer, String>
kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setRecordFilterStrategy(recordFilterStrategy());
return factory;
}