#KafkaListener separate filtering logic for each listener - spring

I need to define a custom filtering strategy for each listener produced by the listener factory.
Currently, I'm using RecordFilterStrategy to do that:
#Bean
ConcurrentKafkaListenerContainerFactory<String, GenericRecord> kafkaListenerContainerFactoryProject() {
ConcurrentKafkaListenerContainerFactory<String, GenericRecord> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setRecordFilterStrategy(new RecordFilterStrategy<String, GenericRecord>() {
#Override
public boolean filter(ConsumerRecord<String, GenericRecord> consumerRecord) {
return true;
}
});
return factory;
}
But such filtering applies to all listeners produced by this factory. What I need is something like to define the different logic for each listener:
#Component
#SendTo("out")
#KafkaListener(topics = "incoming")
public class TestListener {
#Filter
public boolean filter(){
return true;
}
#KafkaHandler
public TestObject listener(TestObject testObject) {
log.debug("Received Message: " + testObject);
return testObject;
}
}
Does spring-kafka have some tools to do that? Or I need to write such logic on my own?
Thanks in advance!

No, you don't. What you just need is a set of ConcurrentKafkaListenerContainerFactory beans with particular RecordFilterStrategy. Then your #KafkaListener should just specify which factory they are based on:
/**
* The bean name of the {#link org.springframework.kafka.config.KafkaListenerContainerFactory}
* to use to create the message listener container responsible to serve this endpoint.
* <p>If not specified, the default container factory is used, if any.
* #return the container factory bean name.
*/
String containerFactory() default "";

Related

SpEL KafkaListener. How can i inject custom deserializer through properties?

I am using spring.
I have a configured ObjectMapper for the entire project and I use it to set up a kafka deserializer.
And then I need a custom kafka deserializer to be used in KafkaListener.
I'm configuring KafkaListener via autoconfiguration, not via #Configuration class.
#Component
#RequiredArgsConstructor
public class CustomMessageDeserializer implements Deserializer<MyMessage> {
private final ObjectMapper objectMapper;
#SneakyThrows
#Override
public MyMessage deserialize(String topic, byte[] data) {
return objectMapper.readValue(data, MyMessage.class);
}
}
If i do like this
#KafkaListener(
topics = {"${topics.invite-user-topic}"},
properties = {"value.deserializer=com.service.deserializer.CustomMessageDeserializer"}
)
public void receiveInviteUserMessages(MyMessage myMessage) {}
I received KafkaException: Could not find a public no-argument constructor
But with public no-argument constructor in CustomMessageDeserializer class i am getting NPE because ObjectMapper = null. It creates and uses a new class, not a spring component.
#KafkaListener supports SpEL expressions.
And I think that this problem can be solved using SpEL.
Do you have any idea how to inject spring bean CustomMessageDeserializer with SpEL?
There are no easy ways to do it with SPeL.
Analysis
To get started, see the JavaDoc for #KafkaListener#properties:
/**
*
* SpEL expressions must resolve to a String ...
*/
The value of value.deserializer is used to instantiate the specified deserializer class. Let's follow the call chain:
You specify this value in the #KafkaListener annotation, then you are probably not creating a bean of the ConsumerFactory.class. So Spring creates this bean class itself - see KafkaAutoConfiguration#kafkaConsumerFactory.
Next is the creation of the returned object new DefaultKafkaConsumerFactory(...) as ConsumerFactory<?,?> using the constructor for default delivery expressions keyDeserializer/valueDeserializer = () -> null
This factory is used to create a Kafka consumer (The entry point is the constructor KafkaMessageListenerContainer#ListenerConsumer, then KafkaMessageListenerContainer.this.consumerFactory.createConsumer...)
In the KafkaConsumer constructor, the valueDeserializer object is being created, because it is null (for the default factory of point 2 above):
if (valueDeserializer == null) {
this.valueDeserializer = config.getConfiguredInstance(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, Deserializer.class);
The implementation of config.getConfiguredInstance involves instantiating your deserializer class via a parameterless constructor using reflection and your String "com.service.deserializer.CustomMessageDeserializer" class name
Solutions
To use value.deserializer with your customized ObjectMapper, you must create the ConsumerFactory bean yourself using the setValueDeserializer(...) method. This is also mentioned in the second Important part of the JSON.Mapping_Types.Important documentation
If you don't want to create a ConsumerFactory bean, and also don't have complicated logic in your deserializer (you only have return objectMapper.readValue(data, MyMessage.class);), then register DefaultKafkaConsumerFactoryCustomizer:
#Bean
// inject your custom objectMapper
public DefaultKafkaConsumerFactoryCustomizer customizeJsonDeserializer(ObjectMapper objectMapper) {
return consumerFactory ->
consumerFactory.setValueDeserializerSupplier(() ->
new org.springframework.kafka.support.serializer.JsonDeserializer<>(objectMapper));
}
In this case, you don't need to create your own CustomMessageDeserializer class (remove it) and Spring will automatically parse the message into your MyMessage.
#KafkaListener annotation should also not contains the property properties = {"value.deserializer=com.my.kafka_test.component.CustomMessageDeserializer"}. This DefaultKafkaConsumerFactoryCustomizer bean will automatically be used to configure the default ConsumerFactory<?, ?> (see the implementation of the KafkaAutoConfiguration#kafkaConsumerFactory method)
Here how it works for me:
#KafkaListener(topics = "${solr.kafka.topic}", containerFactory = "batchFactory")
public void listen(List<SolrInputDocument> docs, #Header(KafkaHeaders.BATCH_CONVERTED_HEADERS) List<Map<String, Object>> headers, Acknowledgment ack) throws IOException {...}
And then I have 2 beans defined in my Configuration
#Profile("!test")
#Bean
#Autowired
public ConsumerFactory<String, SolrInputDocument> consumerFactory(KafkaProperties properties) {
Map<String, Object> props = properties.buildConsumerProperties();
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
DefaultKafkaConsumerFactory<String, SolrInputDocument> result = new DefaultKafkaConsumerFactory<>(props);
String validatedKeyDeserializerName = KafkaMessageType.valueOf(keyDeserializerName).toString();
ZiDeserializer<SolrInputDocument> deserializer = ZiDeserializerFactory.getInstance(validatedKeyDeserializerName);
result.setValueDeserializer(deserializer);
return result;
}
#Profile("!test")
#Bean
#Autowired
public ConcurrentKafkaListenerContainerFactory<String, SolrInputDocument> batchFactory(ConsumerFactory<String, SolrInputDocument> consumerFactory) {
ConcurrentKafkaListenerContainerFactory<String, SolrInputDocument> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory);
factory.setBatchListener(true);
factory.setConcurrency(2);
ExponentialBackOffWithMaxRetries backoff = new ExponentialBackOffWithMaxRetries(10);
backoff.setMultiplier(3); // Default is 1.5 but this seems more reasonable
factory.setCommonErrorHandler(new DefaultErrorHandler(null, backoff));
// Needed for manual commits
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
return factory;
}
Note that the interface ZiDeserializer<SolrInputDocument> deserializeris my interface and ZiDeserializerFactory.getInstance(validatedKeyDeserializerName); returns my custom implementation of ZiDeserializer. And ZiDeserializer extends org.apache.kafka.common.serialization.Deserializer. This works for me

How to create JMS MessageListenerContainer on a list of ConnectionFactory

The following configuration creates a MessageListenerContainer on one ConnectionFactory.
#Configuration
public class MyConfig {
#Bean
public MessageListenerContainer myListenerContainer() {
DefaultMessageListenerContainer messageListenerContainer = new DefaultMessageListenerContainer();
messageListenerContainer.setConnectionFactory(myConnectionFactory1);
[...]
return messageListenerContainer;
}
}
I want to create the same configuration of MessageListenerContainer but with differents ConnectionFactory (pointing on differents queues managers).
I've tried to return a list of MessageListenerContainer (MessageListenerContainer[] or List<MessageListenerContainer> with or without a #Qualifier) but new messages are ignored.
How could I manage the MessageListenerContainer creation on a list of ConnectionFactory?
There's another way to create multiple MessageListenerContainer, instead of creating MessageListenerContainer you create multiple JmsListenerContainerFactory and set the bean name in JmsListener.
#EnableJms
class JmsConfiguration{
Bean
public JmsListenerContainerFactory jmsListenerContainerFactory(){
// create jms listener
}
#Bean
public JmsListenerContainerFactory jmsListenerContainerFactory2(){
// create jms listener
}
}
Two factory beans are created here, jmsListenerContainerFactory and jmsListenerContainerFactory2, now you can use these factory beans in JmsListener to denote which bean to be used.
#Component
class JmsListeners {
#JmsListener(containerFactory="jmsListenerContainerFactory")
public void onMessage(...){
}
#JmsListener(containerFactory="jmsListenerContainerFactory2")
public void onMessage2(...){
}
}
You can register the beans dynamically with the application context.
#Component
class Configurer {
Configurer (GenericApplicationContext context) {
for (i = 0; ...) {
ConnectionFactory cf = ...
context.registerBean("cf" + i, ConnectionFactory.class, () -> cf);
context.getBean("cf" + i); // to initialize it
DefaultMessageListenerContainer container = ...
context.registerBean("container" + i, ...);
context.getBean("container" + i, ...
}
}
}

Spring Integration: connection to multiple MQ servers by config

I do have a Spring Boot 5 application and I also have it running against one IBM MQ server.
Now we want it to connect to three or more MQ servers. My intention is now to just add XY connection infos to the environment and then I get XY MQConnectionFactory beans and al the other beans that are needed for processing.
At the moment this is what I have:
#Bean
#Qualifier(value="MQConnection")
public MQConnectionFactory getIbmConnectionFactory() throws JMSException {
MQConnectionFactory factory = new MQConnectionFactory();
// seeting all the parameters here
return factory;
}
But this is quite static. Is there an elegant way of doing this?
I stumbled about IntegrationFlow. Is this a possibly working solution?
Thanks for all your tipps!
KR
Solution
Based on Artem Bilan's response I built this class.
#Configuration
public class ConnectionWithIntegrationFlowMulti {
protected static final Logger LOG = Logger.create();
#Value("${mq.queue.jms.sources.queue.queue-manager}")
private String queueManager;
#Autowired
private ConnectionConfig connectionConfig;
#Autowired
private SSLSocketFactory sslSocketFactory;
#Bean
public MessageChannel queureader() {
return new DirectChannel();
}
#Autowired
private IntegrationFlowContext flowContext;
#PostConstruct
public void processBeanDefinitionRegistry() throws BeansException {
Assert.notEmpty(connectionConfig.getTab().getLocations(), "At least one CCDT file locations must be provided.");
for (String tabLocation : connectionConfig.getTab().getLocations()) {
try {
IntegrationFlowRegistration theFlow = this.flowContext.registration(createFlow(tabLocation)).register();
LOG.info("Registered bean flow for %s with id = %s", queueManager, theFlow.getId());
} catch (JMSException e) {
LOG.error(e);
}
}
}
public IntegrationFlow createFlow(String tabLocation) throws JMSException {
LOG.info("creating ibmInbound");
return IntegrationFlows.from(Jms.messageDrivenChannelAdapter(getConnection(tabLocation)).destination(createDestinationBean()))
.handle(m -> LOG.info("received payload: " + m.getPayload().toString()))
.get();
}
public MQConnectionFactory getConnection(String tabLocation) throws JMSException {
MQConnectionFactory factory = new MQConnectionFactory();
// doing stuff
return factory;
}
#Bean
public MQQueue createDestinationBean() {
LOG.info("creating destination bean");
MQQueue queue = new MQQueue();
try {
queue.setBaseQueueManagerName(queueManager);
queue.setBaseQueueName(queueName);
} catch (Exception e) {
LOG.error(e, "destination bean: Error for integration flow");
}
return queue;
}
}
With Spring Integration you can create IntegrationFlow instances dynamically at runtime. For that purpose there is an IntegrationFlowContext with its registration() API. The returned IntegrationFlowRegistrationBuilder as a callback like:
/**
* Add an object which will be registered as an {#link IntegrationFlow} dependant bean in the
* application context. Usually it is some support component, which needs an application context.
* For example dynamically created connection factories or header mappers for AMQP, JMS, TCP etc.
* #param bean an additional arbitrary bean to register into the application context.
* #return the current builder instance
*/
IntegrationFlowRegistrationBuilder addBean(Object bean);
So, your MQConnectionFactory instances can be populated alongside with the other flow, used as references in the particular JMS components and registered as beans, too.
See more info in docs: https://docs.spring.io/spring-integration/docs/5.2.3.RELEASE/reference/html/dsl.html#java-dsl-runtime-flows
If you are fine with creating them statically, you can create the beans as you are now (each having a unique qualifier), but you can access them all dynamically in your services / components by having an #Autowired List<MQConnectionFactory> field or #Autowired Map<String, MQConnectionFactory> field. Spring will automatically populate the fields with all of the beans of type MQConnectionFactory
In the the Map implementation, the String will be the qualifier value.
If you also want to create the beans dynamically based on some properties, etc, it gets a little more complicated. You will need to look into something along the lines of instantiating beans at runtime

AWS SQS (queue) with Spring Boot - performance issues

I have a service that reads all messages from AWS SQS.
#Slf4j
#Configuration
#EnableJms
public class JmsConfig {
private SQSConnectionFactory connectionFactory;
public JmsConfig(
#Value("${amazon.sqs.accessKey}") String awsAccessKey,
#Value("${amazon.sqs.secretKey}") String awsSecretKey,
#Value("${amazon.sqs.region}") String awsRegion,
#Value("${amazon.sqs.endpoint}") String awsEndpoint) {
connectionFactory = new SQSConnectionFactory(
new ProviderConfiguration(),
AmazonSQSClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(
new BasicAWSCredentials(awsAccessKey, awsSecretKey)))
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(awsEndpoint, awsRegion))
.build());
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory =
new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(this.connectionFactory);
factory.setDestinationResolver(new DynamicDestinationResolver());
factory.setConcurrency("3-10");
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
factory.setReceiveTimeout(2000L); //??????????
return factory;
}
#Bean
public JmsTemplate defaultJmsTemplate() {
return new JmsTemplate(this.connectionFactory);
}
I've heard about long polling so I wonder how I could use it in my case. I wonder how this listener works - I do not want to create unnecessary calls to the AWS SQS.
My listener that reads messages and converts them to the Object and saves on Redis db:
#JmsListener(destination = "${amazon.sqs.destination}")
public void receive(String requestJSON) throws JMSException {
log.info("Received");
try {
Trace trace = Trace.fromJSON(requestJSON);
traceRepository.save(trace);
(...)
I'd like to know your opinions - what is the best approach to minimalize unnecessary calls to SQS to get messages.
Maybe shoud I use for example
factory.setReceiveTimeout(2000L);
Unfortunately there is too little information in Internet about it
Thanks,
Matthew

RabbitListener annotation queue name by ConfigurationProperties

I have configured my rabbit properties via application.yaml and spring configurationProperties.
Thus, when I configure exchanges, queues and bindings, I can use the getters of my properties
#Bean Binding binding(Queue queue, TopicExchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(properties.getQueue());
}
#Bean Queue queue() {
return new Queue(properties.getQueue(), true);
}
#Bean TopicExchange exchange() {
return new TopicExchange(properties.getExchange());
}
However, when I configure a #RabbitListener to log the messages on from the queue, I have to use the full properties name like
#RabbitListener(queues = "${some.long.path.to.the.queue.name}")
public void onMessage(
final Message message, final Channel channel) throws Exception {
log.info("receiving message: {}#{}", message, channel);
}
I want to avoid this error prone hard coded String and refer to the configurationProperties bean like:
#RabbitListener(queues = "${properties.getQueue()}")
I had a similar issue once with #EventListener where using a bean reference "#bean.method()" helped, but it does not work here, the bean expression is just interpreted as queue name, which fails because a queue namde "#bean...." does not exist.
Is it possible to use ConfigurationProperty-Beans for RabbitListener queue configuration?
Something like this worked for me where I just used the Bean and SpEL.
#Autowired
Queue queue;
#RabbitListener(queues = "#{queue.getName()}")
I was finally able to accomplish what we both desired to do by taking what #David Diehl suggested, using the bean and SpEL; however, using MyRabbitProperties itself instead. I removed the #EnableConfigurationProperties(MyRabbitProperties.class) in the config class, and registered the bean the standard way:
#Configuration
//#EnableConfigurationProperties(RabbitProperties.class)
#EnableRabbit
public class RabbitConfig {
//private final MyRabbitProperties myRabbitProperties;
//#Autowired
//public RabbitConfig(MyRabbitProperties myRabbitProperties) {
//this.myRabbitProperties = myRabbitProperties;
//}
#Bean
public TopicExchange myExchange(MyRabbitProperties myRabbitProperties) {
return new TopicExchange(myRabbitProperties.getExchange());
}
#Bean
public Queue myQueueBean(MyRabbitProperties myRabbitProperties) {
return new Queue(myRabbitProperties.getQueue(), true);
}
#Bean
public Binding binding(Queue myQueueBean, TopicExchange myExchange, MyRabbitProperties myRabbitProperties) {
return BindingBuilder.bind(myQueueBean).to(myExchange).with(myRabbitProperties.getRoutingKey());
}
#Bean
public MyRabbitProperties myRabbitProperties() {
return new MyRabbitProperties();
}
}
From there, you can access the get method for that field:
#Component
public class RabbitQueueListenerClass {
#RabbitListener(queues = "#{myRabbitProperties.getQueue()}")
public void processMessage(Message message) {
}
}
#RabbitListener(queues = "#{myQueue.name}")
Listener:
#RabbitListener(queues = "${queueName}")
application.properties:
queueName=myQueue

Resources