How can we change the consumption mode to earliest for only one consumer. I tried to set the auto-offset-reset in the properties but it does not work
#KafkaListener(topics = "#{'${app.kafka.mytopic}'}",
groupId = "#{'${app.kafka.group-id}'}",
properties = {"auto-offset-reset:earliest"})
public void consumeAllEvents(Event event) {
}
What's the right to override the kafka config ?
#OneCricketeer is it possible to seek the consumer using # KafkaListener ?
Have your listener extend AbstractConsumerSeekAware and call seekToBeginning() from onPartitionsAssigned().
https://docs.spring.io/spring-kafka/docs/current/reference/html/#seek
(Scroll down to near the end of that section).
Related
I'm using SpringBoot along with #JmsListener to retrieve IBM MQ messages from multiple queues within the same QManager. So far I can get messages without any issues. But there could be scenarios, where I had to stop consuming msgs from one of these queues temporarily. It doesn't have to be dynamic.
I'm not using any custom ConnectionFactory methods. When needed, I would like to make config changes in application.properties to disable that particular Queue consumption and restart the process. Is this possible? Can't find any specific info for this scenario. Would appreciate any suggestions. TIA.
#Component
public class MyJmsListener {
#JmsListener(destination = "{ibm.mq.queue.queue01}")
public void handleQueue01(String message) {
System.out.println("received: "+message);
}
#JmsListener(destination = "{ibm.mq.queue.queue02}")
public void handleQueue02(String message) {
System.out.println("received: "+message);
}
}
application.properties
ibm.mq.queue.queue01=IBM.QUEUE01
ibm.mq.queue.queue02=IBM.QUEUE02
If you give each #JmsListener an id property, you can start and stop them individually using the JmsListenerEndpointRegistry bean.
registry.getListenerContainer(id).stop();
I want to send something to Kafka topic in producer-only (not in read-write process) transaction using output-channel.
I read documentation and another topic on StackOverflow (Spring cloud stream kafka transactions in producer side).
Problem is that i need to set unique transactionIdPrefix per node.
Any suggestion how to do it?
Here is one way...
#Component
class TxIdCustomizer implements EnvironmentAware {
#Override
public void setEnvironment(Environment environment) {
Properties properties = new Properties();
properties.setProperty("spring.cloud.stream.kafka.binder.transaction.transactionIdPrefix",
UUID.randomUUID().toString());
((StandardEnvironment) environment).getPropertySources()
.addLast(new PropertiesPropertySource("txId", properties));
}
}
i have a problem i made an apache kafka consumer in spring boot to consume 3 different topics. but I need to consume all the data from the first topic first and then consume the data from the following topics, is there any way to do that? or will you always read them the same way?
#Component
public class KafkaTestListener {
#KafkaListener(topics = "${message.topic.name}", groupId = "${message.group.name}")
public void listenTopic1(String message) {....}
#KafkaListener(topics = "${message.topic.name2}", groupId = "${message.group.name}")
public void listenTopic3(String message) {....}
#KafkaListener(topics = "${message.topic.name3}", groupId = "${message.group.name}")
public void listenTopic3(String message) {.....}
}
Give each listener an id; set autoStartup to false.
Set the container property idleEventInterval to some value.
Add an #EventListener method to receive ListenerContainerIdleEvents - see https://docs.spring.io/spring-kafka/docs/2.5.3.RELEASE/reference/html/#events and https://docs.spring.io/spring-kafka/docs/2.5.3.RELEASE/reference/html/#event-consumption
Use the KafkaListenerEndpointRegistry to start and stop the containers (via id) as needed.
I have a Kafka topic with data, called "topic01"
I want to create a consumer that every time I start my Spring Boot 2 application, start reading that topic from the beginning.
I have the following code, that if I add something new to the topic if it reaches me, but when starting the first time, it won't read me from the beginning of the topic.
#KafkaListener(topics = "topic01")
public void listenTopic01(ConsumerRecord<String, MiDTO> consumerRecord) throws Exception {
logger.info("KafkaHandler");
logger.info(consumerRecord.value().toString());
logger.info(consumerRecord.key().toString());
latch.countDown();
}
application.properties:
spring.kafka.consumer.group-id=XXXXX
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer
What configuration should I add, so that this #KafkaListener reads the topic from the beginning, every time I restart my application.
Either use a unique (random) group-id each time, or have your listener class implement ConsumerSeekAware and add
#Override
public void onPartitionsAssigned(Consumer<?, ?> consumer, Collection<TopicPartition> partitions) {
consumer.seekToBeginning(partitions);
}
or
#KafkaListener(topics = "topic01",
groupId = "#{T(java.util.UUID).randomUUID().toString()}")
I'm trying to configure a simple Spring Cloud Stream application with RabbitMQ. The code I use is mostly taken from spring-cloud-stream-samples.
I have an entry point:
#SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
and a simple messages producer from the example:
#EnableBinding(Source.class)
public class SourceModuleDefinition {
private String format = "yyyy-MM-dd HH:mm:ss";
#Bean
#InboundChannelAdapter(value = Source.OUTPUT, poller = #Poller(fixedDelay = "${fixedDelay}", maxMessagesPerPoll = "1"))
public MessageSource<String> timerMessageSource() {
return () -> new GenericMessage<>(new SimpleDateFormat(this.format).format(new Date()));
}
}
Additionally, here is application.yml configuration:
fixedDelay: 5000
spring:
cloud:
stream:
bindings:
output:
destination: test
When I run the example, it connects to Rabbit and creates an exchange called test. But my problem is, it doesn't create a queue and binding automatically. I can see traffic going in Rabbit, but all my messages are then gone. While I need them to stay in some queue unless they are read by consumer.
Maybe I misunderstand something, but from all the topics I read, it seems like Spring Cloud Stream should create a queue and a binding automatically. If not, how do I configure it so my messages are persisted?
I'm using Spring Cloud Brixton.SR5 and Spring Boot 1.4.0.RELEASE.
A queue would be created as soon as you start a consumer application.
In the case of Rabbit MQ, where we have separate queues for each consumer group and we cannot know all groups beforehand, if you want to have the queues created automatically for consumer groups that are known in advance, you can use the requiredGroups property of the producers. This will ensure that messages are persisted until a consumer from that group is started.
See details here: http://docs.spring.io/spring-cloud-stream/docs/Brooklyn.BUILD-SNAPSHOT/reference/htmlsingle/#_producer_properties
You will need a consumer in order to have a queue created.
Here you can find an example of a producer and a consumer using rabbitMq:
http://ignaciosuay.com/how-to-implement-asyncronous-communication-between-microservices-using-spring-cloud-stream/