I have a spring boot application(spring version 2.2.2.RELEASE) where I have configured Kafka consumer which processes the data from Kafka and serves to multiple web sockets. The subscription is successful to kafka, but Not all messages from selected Kafka topic is processed by the consumer. Few messages are delayed and few are missed out. But the producer is sending out data which is perfectly ensured. Below I have shared the configuration properties that I have used.
#Bean
public ConsumerFactory<String, String> consumerFactory() {
final String BOOTSTRAP_SERVERS = kafkaBootstrapServer;
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.GROUP_ID_CONFIG, consumerGroupId);
return new DefaultKafkaConsumerFactory<>(props);
}
Is there any configuration I am missing?
For a new Consumer (never committed offsets for the group.id) you must set AUTO_OFFSET_RESET to earliest to avoid missing any existing records in the topic (default is latest).
Related
I have an application using spring boot & spring kafka that gets the delivery attempt header in a record interceptor so that I can include it in log messages. It has been working well until I upgraded to spring boot 2.6.3 and spring kafka 2.8.2 (from 2.5.5/2.7.7)
Now when I try to read the delivery attempt header it is not available. If I try and do the exact same thing within a message listener then it works fine so the header is clearly there.
This is what a simplified record interceptor and the listener container factory look like:
#Bean
public RecordInterceptor<Object, Object> recordInterceptor() {
return record -> {
int delivery = ByteBuffer.wrap(record.headers().lastHeader(KafkaHeaders.DELIVERY_ATTEMPT).value()).getInt();
log.info("delivery " + delivery);
return record;
};
}
#Bean
public ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ConsumerFactory<Object, Object> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory);
factory.getContainerProperties().setDeliveryAttemptHeader(true);
factory.setRecordInterceptor(recordInterceptor());
return factory;
}
I can't see anything in the spring docs suggesting the behaviour should have changed. Any ideas?
This is a bug; I opened an issue. https://github.com/spring-projects/spring-kafka/issues/2082
I'm using Kafka with Spring Boot. I use Rest Controllers to call Producer/Consumer API's. Producer Class is able to add messages to the topic. I verified using command line utility (Console-consumer.sh). However my Consumer class is not able to receive them in Java for further processing.
#KafkaListener used in Consumer class listener method should be able to receive messages when my Producer class posts messages to the topic which is not happening. Any help appreciated.
Is it still necessary for consumer to subscribe and poll for records when I have already created KafkaListenerContainerFactory that is responsible for invoking Consumer Listener method when a message is posted to the topic?
Consumer Class
#Component
public class KafkaListenersExample {
private final List<KafkaPayload> messages = new ArrayList<>();
#KafkaListener(topics = "test_topic", containerFactory = "kafkaListenerContainerFactory")
public void listener(KafkaPayload data) {
synchronized (messages){
messages.add(data);
}
//System.out.println("message from kafka :"+data);
}
public List<KafkaPayload> getMessages(){
return messages;
}
}
Consumer Config
#Configuration
class KafkaConsumerConfig {
private String bootstrapServers = "localhost:9092";
#Bean
public ConsumerFactory<String, KafkaPayload> consumerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
bootstrapServers);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
return new DefaultKafkaConsumerFactory<>(props) ;
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, KafkaPayload> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, KafkaPayload> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerConfigs());
return factory;
}
}
The listener container creates the consumer, subscribes, and takes care of the polling.
Turning on DEBUG logging should help determine what's wrong.
If the records are already in the topic, you need to set ConsumerConfig.AUTO_OFFSET_RESET_CONFIG to earliest. Otherwise, the consumer starts consuming from the end of the topic (latest).
I've configured a kafka consumer with SeekToCurrentErrorHandler in Spring boot application using spring-kafka. My consumer configuration is :
#Bean
public ConsumerFactory<String, String> consumerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafkaserver");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "group-id");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer2.class);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer2.class);
props.put(ErrorHandlingDeserializer2.KEY_DESERIALIZER_CLASS, StringDeserializer.class);
props.put(ErrorHandlingDeserializer2.VALUE_DESERIALIZER_CLASS, StringDeserializer.class.getName());
props.put(JsonDeserializer.KEY_DEFAULT_TYPE, "java.lang.String");
props.put(JsonDeserializer.VALUE_DEFAULT_TYPE, "java.lang.String");
return new DefaultKafkaConsumerFactory<>(props);
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
SeekToCurrentErrorHandler seekToCurrentErrorHandler = new SeekToCurrentErrorHandler(5);
seekToCurrentErrorHandler.setCommitRecovered(true);
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.getContainerProperties().setAckOnError(false);
factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
factory.setErrorHandler(seekToCurrentErrorHandler);
return factory;
}
To test SeekToCurrentErrorHandler config, I pushed a record in kafka with incorrect format so that it fails with deserialization exception. As per my understanding the error handler should try to handle the failed record 5 times and after that it should log and move on to the next record.
But it keeps on reading the failed record infinite number of times.
Please tell me where am I going wrong.
I have exactly the same problem and the only fix I do is make sure the concurrency level is same as the number of partition for the topic. Otherwise it will keeps retrying infinitely.
Sounds like a bug of spring kafka.
I have a activemq:queue inQueue in my JbossFuse. How do I consume those JMS messages which are enqueued so that my process instance is triggered in the Spring boot application integrated with Camunda ? Any link to references or samples would be helpful ?
Currently I am able to consume messages from activemq but I am not sure how to consume the messages from Jboss Fuse ActiveMQ ?
#Component
public class ActiveMQConsumer {
#Autowired
CamelContext camelContext;
#Autowired
ProducerTemplate producerTemplate;
#SuppressWarnings("unchecked")
#JmsListener(destination = "inQueue")
public void consumeMessage(JSONObject employeeRecord) throws Exception {
if (employeeRecord instanceof JSONObject) {
HashMap<String, Object> employeeRecordMap = (HashMap<String, Object>) employeeRecord.toMap();
Exchange exchange = ExchangeBuilder.anExchange(camelContext).withBody(employeeRecordMap).build();
HashMap<String, Object> employeeDetails = (HashMap<String, Object>) employeeRecordMap.get("employeeDetails");
exchange.setProperty("CamundaBpmBusinessKey", employeeDetails.get("employeeADId"));
producerTemplate.send("camunda-bpm:start?processDefinitionKey=camunda-camel-activeMQ", exchange);
}
}
}
application.properties
# activeMQ config
spring.activemq.broker-url=tcp://localhost:61616
spring.activemq.user=admin
spring.activemq.password=admin
Expected to consume messages from JbossFuse.
I would recommend using the maven archetype io.fabric8.archetypes spring-boot-camel-amq-archetype version 2.2.197. This can be found:
Spring Boot example running a Camel route connecting to ActiveMQ
http://repo1.maven.org/maven2/
This will get you a nice sample project that has all of the Camel and Spring dependencies and some nice samples.
By following this tutorial, I was able to create a simple producer-consumer example. In my example there was only 1 topic and i was listening to that topic. For that reason, the code in ReceiverConfig makes sense. Specially the point around GROUP_ID_CONFIG i.e., i create topic topic_name and then it was configured in this configuration. Now my question is that, what if I have more than 1 topic. Let's say i have topic_1, topic_2 and so on? Shall i create ReceiverConfig for each individual topic?
#EnableKafka
#Configuration
public class ReceiverConfig {
#Value("${spring.kafka.bootstrap-servers}")
private String bootstrapServers;
#Bean
public Map<String, Object> consumerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(GROUP_ID_CONFIG, "topic_name");
props.put(AUTO_OFFSET_RESET_CONFIG, "earliest");
return props;
}
#Bean
public ConsumerFactory<String, String> consumerFactory() {
return new DefaultKafkaConsumerFactory<>(consumerConfigs());
}
#Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
return factory;
}
}
The short answer is no, you do not need to create multiple configuration for each topic.
Before going any further, I think there is good to specify that the groupId is the group which the Consumer process belongs to and topic to be consumed by the the Consumer process are two different things.
With the sentence below you will tell to the Consumer that its belong to the topic_name group, nothing more.
props.put(GROUP_ID_CONFIG, "topic_name");
If you want a Consumer to read data from multiple topics, there is a subscribe method which receives a Collections as a parameter, in that way you specify all the topics to read data without having to create a new configuration for each topic.
Please check this example out, you will see the method I mentioned
// Subscribe to the topic.
consumer.subscribe(Collections.singletonList(TOPIC));