Spring Boot: how to use FilteringMessageListenerAdapter - spring-boot

I have a Spring Boot application which listens to messages on a Kafka queue. To filter those messages, have the following two classs
#Component
public class Listener implements MessageListener {
private final CountDownLatch latch1 = new CountDownLatch(1);
#Override
#KafkaListener(topics = "${spring.kafka.topic.boot}")
public void onMessage(Object o) {
System.out.println("LISTENER received payload *****");
this.latch1.countDown();
}
}
#Configuration
#EnableKafka
public class KafkaConfig {
#Autowired
private Listener listener;
#Bean
public FilteringMessageListenerAdapter filteringReceiver() {
return new FilteringMessageListenerAdapter(listener, recordFilterStrategy() );
}
public RecordFilterStrategy recordFilterStrategy() {
return new RecordFilterStrategy() {
#Override
public boolean filter(ConsumerRecord consumerRecord) {
System.out.println("IN FILTER");
return false;
}
};
}
}
While messages are being processed by the Listener class, the RecordFilterStrategy implementation is not being invoked. What is the correct way to use FilteringMessageListenerAdapter?
Thanks

The solution was as follows:
No need for the FilteringMessageListenerAdapter class.
Rather, create a ConcurrentKafkaListenerContainerFactory, rather than relying on what Spring Boot provides out of the box. Then, set the RecordFilterStrategy implementation on this class.
#Bean
ConcurrentKafkaListenerContainerFactory<Integer, String>
kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setRecordFilterStrategy(recordFilterStrategy());
return factory;
}

Related

How to use error-channel for catching exception in Spring Integration?

What I am trying to do? : I am new to Spring Integration and already have read many similar questions regarding error handling but I don't understand how to catch exceptions using error-channel?
What I have done so far:
#EnableIntegration
#IntegrationComponentScan
#Configuration
public class TcpClientConfig implements ApplicationEventPublisherAware {
private ApplicationEventPublisher applicationEventPublisher;
private final ConnectionProperty connectionProperty;
#Override
public void setApplicationEventPublisher(ApplicationEventPublisher applicationEventPublisher) {
this.applicationEventPublisher = applicationEventPublisher;
}
TcpClientConfig(ConnectionProperty connectionProperty) {
this.connectionProperty = connectionProperty;
}
#Bean
public AbstractClientConnectionFactory clientConnectionFactory() {
TcpNioClientConnectionFactory tcpNioClientConnectionFactory =
getTcpNioClientConnectionFactoryOf(
connectionProperty.getPrimaryHSMServerIpAddress(),
connectionProperty.getPrimaryHSMServerPort());
final List<AbstractClientConnectionFactory> fallBackConnections = getFallBackConnections();
fallBackConnections.add(tcpNioClientConnectionFactory);
final FailoverClientConnectionFactory failoverClientConnectionFactory =
new FailoverClientConnectionFactory(fallBackConnections);
return new CachingClientConnectionFactory(
failoverClientConnectionFactory, connectionProperty.getConnectionPoolSize());
}
#Bean
DefaultTcpNioSSLConnectionSupport connectionSupport() {
final DefaultTcpSSLContextSupport defaultTcpSSLContextSupport =
new DefaultTcpSSLContextSupport(
connectionProperty.getKeystorePath(),
connectionProperty.getTrustStorePath(),
connectionProperty.getKeystorePassword(),
connectionProperty.getTruststorePassword());
final String protocol = "TLSv1.2";
defaultTcpSSLContextSupport.setProtocol(protocol);
return new DefaultTcpNioSSLConnectionSupport(defaultTcpSSLContextSupport, false);
}
#Bean
public MessageChannel outboundChannel() {
return new DirectChannel();
}
#Bean
#ServiceActivator(inputChannel = "outboundChannel")
public MessageHandler outboundGateway(AbstractClientConnectionFactory clientConnectionFactory) {
TcpOutboundGateway tcpOutboundGateway = new TcpOutboundGateway();
tcpOutboundGateway.setConnectionFactory(clientConnectionFactory);
return tcpOutboundGateway;
}
#Bean
#ServiceActivator(inputChannel = "error-channel")
public void handleError(ErrorMessage em) {
throw new RuntimeException(String.valueOf(em));
}
private List<AbstractClientConnectionFactory> getFallBackConnections() {
final int size = connectionProperty.getAdditionalHSMServersConfig().size();
List<AbstractClientConnectionFactory> collector = new ArrayList<>(size);
for (final Map.Entry<String, Integer> server :
connectionProperty.getAdditionalHSMServersConfig().entrySet()) {
collector.add(getTcpNioClientConnectionFactoryOf(server.getKey(), server.getValue()));
}
return collector;
}
private TcpNioClientConnectionFactory getTcpNioClientConnectionFactoryOf(
final String ipAddress, final int port) {
TcpNioClientConnectionFactory tcpNioClientConnectionFactory =
new TcpNioClientConnectionFactory(ipAddress, port);
tcpNioClientConnectionFactory.setUsingDirectBuffers(true);
tcpNioClientConnectionFactory.setDeserializer(new CustomDeserializer());
tcpNioClientConnectionFactory.setApplicationEventPublisher(applicationEventPublisher);
tcpNioClientConnectionFactory.setSoKeepAlive(true);
tcpNioClientConnectionFactory.setConnectTimeout(connectionProperty.getConnectionTimeout());
tcpNioClientConnectionFactory.setSoTcpNoDelay(true);
tcpNioClientConnectionFactory.setTcpNioConnectionSupport(connectionSupport());
return tcpNioClientConnectionFactory;
}
}
Gateway
#Component
#MessagingGateway(defaultRequestChannel = "outboundChannel",errorChannel ="error-channel" )
public interface TcpClientGateway {
String send(String message);
}
Also currently, I am facing
required a bean of type org.springframework.messaging.support.ErrorMessage that could not be found
I need some assistance!
Thanking you in advance,
EDIT
#AllArgsConstructor
#Service
public class AsyncNonBlockingClient implements Connector {
TcpClientGateway tcpClientGateway;
#Override
public String send(final String payload) {
return tcpClientGateway.send(payload);
}
}
See documentation about messaging annotation:
Your problem is here: https://docs.spring.io/spring-integration/docs/current/reference/html/configuration.html#annotations_on_beans
#Bean
#ServiceActivator(inputChannel = "error-channel")
public void handleError(ErrorMessage em) {
This is a plain POJO method, therefore it cannot be marked with a #Bean. You use a #Bean really for beans to expose. Then you decide if that has to be a #ServiceActivator or not. So, just remove #Bean from this method and your error-channel consumer should be OK.

How to configure a custom Kafka deserializer and get the consumed JSON data using a KafkaListener

I am trying to consume a JSON message using spring kafka. The message which is consumed by the consumer is like this.
{
"EventHeader": {
"entityName": "Account",
"changedFields": ["Id", "Name"]
},
"NewFields": {
"Id": "001",
"Name": "Test Account",
},
"OldFields": {}
}
So far I have created classes for "EventHeader", "NewFields","OldFields" ,and for "KafkaPayload". And also I have created a custom deserializer to deserialize this JSON payload.Here is my custom deserializer.
public class CustomDeserailizer <T extends Serializable> implements Deserializer<T> {
private ObjectMapper objectMapper = new ObjectMapper();
public static final String VALUE_CLASS_NAME_CONFIG = "value.class.name";
#Override
public void configure(Map<String, ?> configs, boolean isKey) {
Deserializer.super.configure(configs, isKey);
}
#Override
public T deserialize(String topic, byte[] objectData) {
return (objectData == null) ? null : (T) SerializationUtils.deserialize(objectData);
}
#Override
public T deserialize(String topic, Headers headers, byte[] data) {
return Deserializer.super.deserialize(topic, headers, data);
}
#Override
public void close() {
Deserializer.super.close();
}
}
I have set the consumer configurations as below.
public class KafkaConfig {
#Bean
public KafkaConsumer<String, KafkaPayload> consumerFactory(){
Properties config = new Properties();
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
config.put(ConsumerConfig.GROUP_ID_CONFIG, "groupId");
config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, CustomDeserializer.class);
return new KafkaConsumer<>(config);
}
}
Now I need to show the consumed message through a #KafkaListener setting the consumer into ConsumerFactory. But I don't understand how to do that. This is my first time using kafka.So could anyone give me some idea about this?
This is how I am trying to do that.
#Bean
public ConcurrentKafkaListenerContainerFactory<String, KafkaPayload> kafkaListener(){
ConcurrentKafkaListenerContainerFactory factory = new ConcurrentKafkaListenerContainerFactory();
factory.setConsumerFactory(consumerFactory());
return factory;
}
This is my KafkaListener
public class ConsumerService {
#KafkaListener(topics = "Topic", groupId = "sample-group",containerFactory = "kafkaListener")
public void consume(KafkaPayload kafkaPayload){
System.out.println("Consumed Message :"+ kafkaPayload);
}
}
Since you are using Spring Boot, just set the value deserializer class name as a property and Boot will automatically wire it into the container factory for your #KafkaListener. No need to define your own consumer factory or container factory.
spring.kafka.consumer.value-deserializer=com.acme.CustomDeserializer
https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#application-properties.integration.spring.kafka.consumer.value-deserializer

Kafka Streams programmatically configuration not working

I'm doing a spring boot application and I'm trying to configure kafka programmatically, but for some reason is still getting the properties from application.yaml instead of the ones I set programmatically
#Configuration
public class KafkaConfiguration {
#Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(kafkaConsumerFactory());
factory.setConcurrency(1);
factory.getContainerProperties().setPollTimeout(30000);
return factory;
}
public ConsumerFactory<String, String> kafkaConsumerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "aaa"); // should crash since is not valid
props.put(ConsumerConfig.GROUP_ID_CONFIG, "app1");
return new DefaultKafkaConsumerFactory<>(props);
}
}
#Component
public class StreamListener {
#StreamListener(TestStreams.TEST_STREAM_IN)
public void testStream(#Payload GenericCustomEvent response, #Headers MessageHeaders headers) throws Exception {
log.debug("Received generic event {} with headers {}", response, headers);
}
}
public interface TestStreams {
String TEST_STREAM_IN = "test-stream-in";
#Input(TEST_STREAM_IN)
SubscribableChannel inputTestStream();
}
#EnableBinding({TestStreams.class})
#SpringBootApplication
public class KafkaApplication {
public static void main(String[] args) {
SpringApplication.run(KafkaApplication .class, args);
}
}
The binder doesn't use a KafkaListenerContainerFactory, it creates the container itself from the yaml.
You can modify the container(s) by adding a ListenerContainerCustomizer bean.
Example here Can I apply graceful shutdown when using Spring Cloud Stream Kafka 3.0.3.RELEASE?

Explicitly Start Kafka Consumer In Main After method runs

I have a spring boot service that consumes from a kafka topic. When i consume i perform certain tasks on the kafka message. Before i can perform these operations i need to wait for the service to load some data into caches that i have set up. My issue is if i set kafka consumer to autostart it starts consuming before the cache loads and it errors out.
I am trying to explicitly start the consumer after i load the cache however i get null pointer exceptions.
#Configuration
public class KafkaConfig {
#Value("${kafka.server}")
String server;
#Value("${kafka.port}")
String port;
#Value("${kafka.group.id}")
String groupid;
#Bean
public ConsumerFactory<String, String> consumerFactory() {
Map<String, Object> config = new HashMap<>();
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, server+":"+port);
config.put(ConsumerConfig.GROUP_ID_CONFIG, groupid);
config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
// config.put("security.protocol","SASL_PLAINTEXT");
// config.put("sasl.kerberos.service.name","kafka");
return new DefaultKafkaConsumerFactory<>(config);
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory();
factory.setConsumerFactory(consumerFactory());
factory.setAutoStartup(false);
return factory;
}
}
KafkaListener
#Service
public class KafkaConsumer {
#Autowired
AggregationService aggregationService;
#Autowired
private KafkaListenerEndpointRegistry registry;
private final CounterService counterService;
public KafkaConsumer(CounterService counterService) {
this.counterService = counterService;
}
#KafkaListener(topics = "gliTransactionTopic", group = "gliDecoupling", id = "gliKafkaListener")
public boolean consume(String message,
#Header(KafkaHeaders.RECEIVED_PARTITION_ID) Integer partition,
#Header(KafkaHeaders.OFFSET) Long offset) throws ParseException {
System.out.println("Inside kafka listener :" + message+" partition :"+partition.toString()+" offset :"+offset.toString());
aggregationService.run();
return true;
}
}
service To start stop
#Service
public class DecouplingController {
#Autowired
private KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;
public void stop() {
MessageListenerContainer listenerContainer = kafkaListenerEndpointRegistry
.getListenerContainer("gliKafkaListener");
listenerContainer.stop();
}
public void start() {
MessageListenerContainer listenerContainer = kafkaListenerEndpointRegistry
.getListenerContainer("gliKafkaListener");
listenerContainer.start();
}
}
main method
#SpringBootApplication
public class DecouplingApplication {
Ignite ignite;
static IgniteCache<Long, MappingsEntity> mappingsCache;
public static void main(String[] args) {
SpringApplication.run(DecouplingApplication.class, args);
Ignition.setClientMode(true);
Ignite ignite = Ignition.ignite("ignite");
loadCaches(ignite);
}
public static boolean loadCaches(Ignite ignite) {
mappingsCache = ignite.getOrCreateCache("MappingsCache");
mappingsCache.loadCache(null);
System.out.println("Data Loaded");
DecouplingController dc=new DecouplingController();
dc.start();
return true;
}
}
Below is the exception
Data Loaded
Exception in thread "main" java.lang.NullPointerException
at com.ignite.spring.decoupling.controller.DecouplingController.start(DecouplingController.java:126)
at com.ignite.spring.decoupling.DecouplingApplication.loadCaches(DecouplingApplication.java:64)
at com.ignite.spring.decoupling.DecouplingApplication.main(DecouplingApplication.java:37)
Instead of manually creating an object of DecouplingController , autowire the dependency in DecouplingApplication.
#Autowired
DecouplingController deDecouplingController;
The ApplicationContext which deals with autowired dependencies is not aware about the object you manually created using "new". The autowired kafkaListenerEndpointRegistry is unknown to the new DecouplingController object you created.
Seems like the gliKafkaListener1 was not registred an some part of ConsumerConfig/ListenerConfig

SpringMVC, MessageListener and Injection

Is there anyway to inject services in MessageListener?
#Autowired
#Qualifier("myServices")
MyServices myServices;
#Override
public void onMessage(final Message message, final byte[] pattern) {
myServices.call();
}
This way not inject service. That's because in #Configuration we do
#Bean
MessageListenerAdapter messageListener() {
return new MessageListenerAdapter( new RedisMessageListener() );
}
Any idea?
Thanks
This should work: (no need for a MessageListenerAdapter if the delegate is already a MessageListener):
#Component
public class MyListener implements MessageListener {
#Autowired
#Qualifier("myServices")
MyServices myServices;
#Override
public void onMessage(final Message message, final byte[] pattern) {
myServices.call();
}
}
If you don't want to create standalone class, you can also do this:
#Bean
MessageListenerAdapter messageListener(#Qualifier("myServices") MyServices myServices) {
//use myServices instance somehow
return new MessageListenerAdapter( new RedisMessageListener() );
}

Resources