I use spring boot 2.2 application who receive mq message via jms.
Data is json
In my code I have this config
#Bean
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
When I send data via IBM MQ Explorer, I get this error
org.springframework.jms.support.converter.MessageConversionException: Could not find type id property [_type] on message
Is there a way to set this with IBM MQ explorer?
If your message doesn't have a property containing type information, you can subclass the MappingJackson2MessageConverter and override getJavaTypeForMessage
/**
* Determine a Jackson JavaType for the given JMS Message,
* typically parsing a type id message property.
* <p>The default implementation parses the configured type id property name
* and consults the configured type id mapping. This can be overridden with
* a different strategy, e.g. doing some heuristics based on message origin.
* #param message the JMS Message to set the type id on
* #throws JMSException if thrown by JMS methods
* #see #setTypeIdOnMessage(Object, javax.jms.Message)
* #see #setTypeIdPropertyName(String)
* #see #setTypeIdMappings(java.util.Map)
*/
protected JavaType getJavaTypeForMessage(Message message) throws JMSException {
Related
How can I filter a message before is processed by a #RabbitListener annotated method ?
If the message is for. ex. is "duplicated" because contains an header with a determinate value I would like to return "ack" and skip processing. (skip the body of #RabbitListener method)
I tried to do it in a MessagePostProcessor (with addAfterReceivePostProcessors) but cannot skip execution for ex. based on a message property (header).
This is the signature of MessageProcessor :
Message postProcessMessage(Message message) throws AmqpException;
I would like to return an "ack" here so the message processing is skipped.
thank you for your support.
I think an AmqpRejectAndDontRequeueException is what you need to throw from your MessagePostProcessor impl.
See its javadocs:
/**
* Exception for listener implementations used to indicate the
* basic.reject will be sent with requeue=false in order to enable
* features such as DLQ.
* #author Gary Russell
* #since 1.0.1
*
*/
#SuppressWarnings("serial")
public class AmqpRejectAndDontRequeueException extends AmqpException {
And respective docs: https://docs.spring.io/spring-amqp/docs/current/reference/html/#exception-handling
When using #KafkaListener with batches, the error handler logs the content of the full batch (all messages) in case of an exception.
How can I make this less verbose? I'd like to avoid spamming the log files with all the messages and only see the actual exception.
Here is a minimal example of how my consumer currently looks like:
#Component
class TestConsumer {
#Bean
fun kafkaBatchListenerContainerFactory(kafkaProperties: KafkaProperties): ConcurrentKafkaListenerContainerFactory<String, String> {
val configs = kafkaProperties.buildConsumerProperties()
configs[ConsumerConfig.MAX_POLL_RECORDS_CONFIG] = 10000
val factory = ConcurrentKafkaListenerContainerFactory<String, String>()
factory.consumerFactory = DefaultKafkaConsumerFactory(configs)
factory.isBatchListener = true
return factory
}
#KafkaListener(
topics = ["myTopic"],
containerFactory = "kafkaBatchListenerContainerFactory"
)
fun batchListen(values: List<ConsumerRecord<String, String>>) {
// Something that might throw an exception in rare cases.
}
}
What version are you using?
This container property was added in 2.2.14.
/**
* Set to false to log {#code record.toString()} in log messages instead
* of {#code topic-partition#offset}.
* #param onlyLogRecordMetadata false to log the entire record.
* #since 2.2.14
*/
public void setOnlyLogRecordMetadata(boolean onlyLogRecordMetadata) {
this.onlyLogRecordMetadata = onlyLogRecordMetadata;
}
It has been true by default since version 2.7 (which is why the javadocs now read that way).
This was the previous javadoc:
/**
* Set to true to only log {#code topic-partition#offset} in log messages instead
* of {#code record.toString()}.
* #param onlyLogRecordMetadata true to only log the topic/parrtition/offset.
* #since 2.2.14
*/
Also, starting with version 2.5, you can set the log level on the error handler:
/**
* Set the level at which the exception thrown by this handler is logged.
* #param logLevel the level (default ERROR).
*/
public void setLogLevel(KafkaException.Level logLevel) {
Assert.notNull(logLevel, "'logLevel' cannot be null");
this.logLevel = logLevel;
}
I am trying to write a Kafka consumer application in spring-kafka. I can think of 2 scenarios in which error can occur :
While processing records, an exception can occur in Service layer ( while updating records through API in a table)
Deserialization error
I had already explored an option to handle scenario 1, I can just throw an exception in my code and handle it using SeekToCurrentErrorHandler.
#Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory();
factory.setConsumerFactory(consumerFactory());
factory.getContainerProperties().setAckOnError(false);
factory.getContainerProperties().setAckMode(AckMode.RECORD);
factory.setErrorHandler(new SeekToCurrentErrorHandler(new FixedBackOff(1000L, 2L)));
return factory;
}
For scenario 2, I have got an option of ErrorHandlingDeserializer but I am not sure how to implement it with SeekToCurrentErrorHandler. Is there a way to include ErrorHandler for both scenarios using SeekToCurrentErrorHandler.
My property class is as below :
#Bean
public ConsumerFactory<String, String> consumerFactory(){
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,KAFKA_BROKERS);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, OFFSET_RESET);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class.getName());
props.put(ConsumerConfig.GROUP_ID_CONFIG, GROUP_ID_CONFIG);
props.put(KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG, SCHEMA_REGISTRY_URL);
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, SSL_PROTOCOL);
props.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG,SSL_TRUSTSTORE_LOCATION_FILE_NAME);
props.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, SSL_TRUSTSTORE_SECURE);
props.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG,SSL_KEYSTORE_LOCATION_FILE_NAME);
props.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, SSL_KEYSTORE_SECURE);
props.put(SslConfigs.SSL_KEY_PASSWORD_CONFIG, SSL_KEY_SECURE);
return new DefaultKafkaConsumerFactory<>(props);
}
Publishing error records :
I am also thinking to publish error records to Dead Letter queue. For scenario 1, it should retry and publish on dead letter queue and for scenario 2, it should directly publish as there is no benefit of re-trying. I may not have access to create a topic on my own and would need to ask my producers to create one topic for error records as well. How can I implement logic to publish records on custom error topic.
As I have no control over name if I use DeadLetterPublishingRecoverer. Based on my understanding, it creates topic with <original_topic_name>.DLT.
The SeekToCurrentErrorHandler treats certain exceptions (such as DeserializationException) as fatal and are not retried - that failed record is immediately sent to the recoverer.
For retryable exceptions, the recoverer is called after retrieds are exhausted.
/**
* Add exception types to the default list. By default, the following exceptions will
* not be retried:
* <ul>
* <li>{#link DeserializationException}</li>
* <li>{#link MessageConversionException}</li>
* <li>{#link ConversionException}</li>
* <li>{#link MethodArgumentResolutionException}</li>
* <li>{#link NoSuchMethodException}</li>
* <li>{#link ClassCastException}</li>
* </ul>
* All others will be retried.
* #param exceptionTypes the exception types.
* #see #removeNotRetryableException(Class)
* #see #setClassifications(Map, boolean)
*/
public final void addNotRetryableExceptions(Class<? extends Exception>... exceptionTypes) {
Based on my understanding, it creates topic with <original_topic_name>.DLT.
That is the default behavior; you can provide your own DLT topic name strategy (destination resolver).
See the documentation.
The following example shows how to wire a custom destination resolver:
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template,
(r, e) -> {
if (e instanceof FooException) {
return new TopicPartition(r.topic() + ".Foo.failures", r.partition());
}
else {
return new TopicPartition(r.topic() + ".other.failures", r.partition());
}
});
ErrorHandler errorHandler = new SeekToCurrentErrorHandler(recoverer, new FixedBackOff(0L, 2L));
Producer of the message is not sending message as persistent and when i am trying to consume the message through MessageListener, and any exception(runtime) occurs, it retries for specific number of times (default is 6 from AMQ side) and message get lost.
Reason is that since producer is not setting the Delivery mode as Persistent, after certain number of retry attempt, DLQ is not being created and message does not move to DLQ. Due to this , i lost the message.
My Code is like this :-
#Configuration
#PropertySource("classpath:application.properties")
public class ActiveMqJmsConfig {
#Autowired
private AbcMessageListener abcMessageListener;
public DefaultMessageListenerContainer purchaseMsgListenerforAMQ(
#Qualifier("AMQConnectionFactory") ConnectionFactory amqConFactory) {
LOG.info("Message listener for purchases from AMQ : Starting");
DefaultMessageListenerContainer defaultMessageListenerContainer =
new DefaultMessageListenerContainer();
defaultMessageListenerContainer.setConnectionFactory(amqConFactory);
defaultMessageListenerContainer.setMaxConcurrentConsumers(4);
defaultMessageListenerContainer
.setDestinationName(purchaseReceivingQueueName);
defaultMessageListenerContainer
.setMessageListener(abcMessageListener);
defaultMessageListenerContainer.setSessionTransacted(true);
return defaultMessageListenerContainer;
}
#Bean
#Qualifier(value = "AMQConnectionFactory")
public ConnectionFactory activeMQConnectionFactory() {
ActiveMQConnectionFactory amqConnectionFactory =
new ActiveMQConnectionFactory();
amqConnectionFactory
.setBrokerURL(System.getProperty(tcp://localhost:61616));
amqConnectionFactory
.setUserName(System.getProperty(admin));
amqConnectionFactory
.setPassword(System.getProperty(admin));
return amqConnectionFactory;
}
}
#Component
public class AbcMessageListener implements MessageListener {
#Override
public void onMessage(Message msg) {
//CODE implementation
}
}
Problem :- By setting the client-id at connection level (Connection.setclientid("String")), we can subscribe as durable subscriber even though message is not persistent. By doing this, if application throws runtime exception , after a certain number of retry attempt, DLQ will be created for the Queue and message be moved to DLQ.
But in DefaultMessageListenerContainer, connection is not exposed to client. it is maintained by Class itself as a pool, i guess.
How can i achieve the durable subscription in DefaultMessageListenerContainer?
You can set the client id on the container instead:
/**
* Specify the JMS client ID for a shared Connection created and used
* by this container.
* <p>Note that client IDs need to be unique among all active Connections
* of the underlying JMS provider. Furthermore, a client ID can only be
* assigned if the original ConnectionFactory hasn't already assigned one.
* #see javax.jms.Connection#setClientID
* #see #setConnectionFactory
*/
public void setClientId(#Nullable String clientId) {
this.clientId = clientId;
}
and
/**
* Set the name of a durable subscription to create. This method switches
* to pub-sub domain mode and activates subscription durability as well.
* <p>The durable subscription name needs to be unique within this client's
* JMS client id. Default is the class name of the specified message listener.
* <p>Note: Only 1 concurrent consumer (which is the default of this
* message listener container) is allowed for each durable subscription,
* except for a shared durable subscription (which requires JMS 2.0).
* #see #setPubSubDomain
* #see #setSubscriptionDurable
* #see #setSubscriptionShared
* #see #setClientId
* #see #setMessageListener
*/
public void setDurableSubscriptionName(#Nullable String durableSubscriptionName) {
this.subscriptionName = durableSubscriptionName;
this.subscriptionDurable = (durableSubscriptionName != null);
}
I try to send a message based on a flatten MAP using Spring Boot and AMQP. The message should then be received using #RabbitListener and transfer it back to a MAP.
First I have nested json String and flat it and send it using the following code:
// Flatten the JSON String returned into a map
Map<String,Object> jsonMap = JsonFlattener.flattenAsMap(result);
rabbitTemplate.convertAndSend(ApplicationProperties.rmqExchange, ApplicationProperties.rmqTopic, jsonMap, new MessagePostProcessor() {
#Override
public Message postProcessMessage(Message message) throws AmqpException {
message.getMessageProperties().setHeader("amqp_Key1", "wert1");
message.getMessageProperties().setHeader("amqp_Key2", "Wert2");
message.getMessageProperties().setDeliveryMode(MessageDeliveryMode.PERSISTENT);
return message;
}
});
So far so good.
On the receiving site I try to use a Listener and convert the message payload back to the Map as it was send before.
The problem ist that I have no idea how to do it.
I receive the message with the following code:
#RabbitListener(queues = "temparea")
public void receiveMessage(Message message) {
log.info("Receiving data from RabbitMQ:");
log.info("Message is of type: " + message.getClass().getName());
log.info("Message: " + message.toString());
}
As I mentioned before I have no idea how I can convert the message to my old MAP. The __ TypeId __ of the Message is: com.github.wnameless.json.flattener.JsonifyLinkedHashMap
I would be more than glad if somebody could assist me how I get this message back to an Java Map.
BR
Update after answer from Artem Bilan:
I added the following code to my configuration file:
#Bean
public SimpleRabbitListenerContainerFactory myRabbitListenerContainerFactory() {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setMessageConverter(new Jackson2JsonMessageConverter());
factory.setConnectionFactory(connectionFactory());
factory.setMaxConcurrentConsumers(5);
return factory;
}
But still I have no idea how to get the Map out of my message.
The new code block does not change anything.
You have to configure Jackson2JsonMessageConverter bean and Spring Boot will pick it up for the SimpleRabbitListenerContainerFactory bean definition which is used to build listener containers for the #RabbitListener methods.
UPDATE
Pay attention to the Spring AMQP JSON Sample.
There is a bean like jsonConverter(). According Spring Boot auto-configuration this bean is injected to the default:
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(SimpleRabbitListenerContainerFactoryConfigurer configurer, ConnectionFactory connectionFactory) {
Which is really used for the #RabbitListener by default, when the containerFactory attribute is empty.
So, you need just configure that bean and don't need any custom SimpleRabbitListenerContainerFactory. Or if you do that you should specify its bean name in that containerFactory attribute of your #RabbitListener definitions.
Another option to consider is like Jackson2JsonMessageConverter.setTypePrecedence():
/**
* Set the precedence for evaluating type information in message properties.
* When using {#code #RabbitListener} at the method level, the framework attempts
* to determine the target type for payload conversion from the method signature.
* If so, this type is provided in the
* {#link MessageProperties#getInferredArgumentType() inferredArgumentType}
* message property.
* <p> By default, if the type is concrete (not abstract, not an interface), this will
* be used ahead of type information provided in the {#code __TypeId__} and
* associated headers provided by the sender.
* <p> If you wish to force the use of the {#code __TypeId__} and associated headers
* (such as when the actual type is a subclass of the method argument type),
* set the precedence to {#link TypePrecedence#TYPE_ID}.
* #param typePrecedence the precedence.
* #since 1.6
* #see DefaultJackson2JavaTypeMapper#setTypePrecedence(Jackson2JavaTypeMapper.TypePrecedence)
*/
public void setTypePrecedence(Jackson2JavaTypeMapper.TypePrecedence typePrecedence) {
So, if you want still to have a Message as a method argument but get a gain of the JSON conversion based on the __TypeId__ header, you should consider to configure Jackson2JsonMessageConverter to be based on the Jackson2JavaTypeMapper.TypePrecedence.TYPE_ID.