How can a producer set up from a Kafka factory be disabled?
I started from this example from https://docs.spring.io/spring-kafka/reference/html/
I expected that adding a property like
props.put("autoStartup", "false");
would disable the message sending, but it does not seem to work.
Running the application still sends messages.
There's no producer property like autoStartup, so it doesn't mean disable the message sending. For the producer configuration, you could see here.
In Spring Kafka, autoStartup is used for Listener Container.
You could set this property by:
In #KafkaListener
#KafkaListener(id = "myContainer", topics = "myTopic", autoStartup = "false")
Or
**ListenerContainerFactory.setAutoStartup(Boolean autoStartup)
Related
I'm just starting with ActiveMQ Artemis and have Artemis 2.17.0 installed on my machine. Created SpringBoot test app where both JMS and MQTT publishers and receivers exist. Created also small RestController so I can send messages using both JMS and MQTT producers. Receivers are quite simple and just create a log message to console. Now when I create a message using MQTT producer, both JMS and MQTT receivers get and log message to console. But when I send a message using JMS producer, the message is being received only in JMS receiver, no MQTT message in console. Tried several times. Implementation is ok I think as MQTT producer example is working fine. Is there any limitation for routing messages among protocols in Artemis in this way? Or what kind of problem it can be?
Code info about JMS implementation: https://dmarko.tcl-digitrade.com/post/2021/activemq-artemis-spring-boot/
Code info about MQTT implementation: https://dmarko.tcl-digitrade.com/post/2021/activemq-artemis-mqtt/
Apache ActiveMQ Artemis has a flexible addressing model that supports both Point-to-Point and Publish-Subscribe patterns.
By default, Spring Boot creates a JmsTemplate configured to transmit Point-to-Point while MQTT uses a Publish-Subscribe pattern, so the JMS and MQTT receivers are using different messaging patterns and this is causing your issue.
To configure a JmsTemplate for the Publish-Subscribe pattern set spring.jms.pub-sub-domain=true through Boot’s application.properties or set the JmsTemplate pubSubDomain to true, ie:
jmsTemplate.setPubSubDomain(true);
To configure a JmsListener for the Publish-Subscribe pattern set spring.jms.pub-sub-domain=true through Boot’s application.properties or set the JmsListenerContainerFactory pubSubDomain to true, ie:
#Bean
public JmsListenerContainerFactory<?> topicConnectionFactory(ConnectionFactory connectionFactory, DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setPubSubDomain(true);
return factory;
}
#JmsListener(destination = "${prices.mqtt.east}", containerFactory = "topicConnectionFactory")
public void receiveFromTopic(String message) {
...
}
I have got an application that uses SpringBoot 2.10.0.Release and kafka in the version 2.10.0. The application has got a simple producer and consumer: The sender works with KafkaTemplate and the consumer with KafkaListener.
What I try to achieve is to be able to start the SpringBoot application even if the KafkaServer is not running.
Currently without a running KafkaBroker the application cannot be started with this error message:
org.springframework.context.ApplicationContextException:
Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry';
nested exception is org.apache.kafka.common.errors.TimeoutException
Is there a way to achieve this and if yes could anybody give me hint or a keyword how to manage this?
When running the Spring-Boot application with a KafkaListener, the listener will per default try to listen to Kafka. If the KafkaBroker is invalid or missing, then you will get a org.apache.kafka.common.KafkaException.
You can change the default behaviour of the container factory by setting the autoStartup property to false. One way to do this is by adding autoStartup = "false" element to your KafkaListener annotation:
#KafkaListener(topics = "some_topic", autoStartup = "false")
public void fooEventListener(){
Now your spring boot application will start. You will still get an error when trying to use the KafkaListener if the broker is still down or invalid, but you will now be able to handle the error within your Java code instead of a Spring Boot server crash.
Documentation about KafkaListner autoStartup element.
It have to be mentioned that the error you are receiving (TimeoutException) is not because the broker is down, it is what Kafka will throw if the buffer is full.
The batch records will then be removed from the queue and will not be delivered to the broker. This error will not be the reason for you application using Kafka not to start.
I use Kafka and Spring Boot with Spring Kafka. After abnormal application termination and then restart, my application started receiving the old, already processed messages from Kafka queue.
What may be the reason for that and how to find and resolve the issue?
my Kafka properties:
spring.kafka.bootstrap-servers=${kafka.host}:${kafka.port}
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.group-id=postfenix
spring.kafka.consumer.enable-auto-commit=false
My Spring Kafka factory and listener:
#Bean
public ConcurrentKafkaListenerContainerFactory<String, Post> postKafkaListenerContainerFactory(KafkaProperties kafkaProperties) {
ConcurrentKafkaListenerContainerFactory<String, Post> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.getContainerProperties().setAckMode(AckMode.MANUAL);
factory.setConsumerFactory(postConsumerFactory(kafkaProperties));
return factory;
}
#KafkaListener(topics = "${kafka.topic.post.send}", containerFactory = "postKafkaListenerContainerFactory")
public void sendPost(ConsumerRecord<String, Post> consumerRecord, Acknowledgment ack) {
Post post = consumerRecord.value();
// do some logic here
ack.acknowledge();
}
When using Kafka, the clients need to commit offsets themselves. This is in contrast to other message brokers, such as AMQP brokers, where the broker keeps track of messages a client did already receive.
In your case, you do not commit offsets automatically and therefore Kafka expects you to commit them manually (because of this setting: spring.kafka.consumer.enable-auto-commit=false). If you do not commit offsets manually in your program, the behaviour you describe is pretty much the expected one. Kafka simply does not know what messages your program did process successfully. Each time you restart your program, Kafka will see that your program did not commit any offsets yet and will apply the strategy you provide in spring.kafka.consumer.auto-offset-reset=earliest, which means the first message in the queue.
If this is all new to you, I suggest reading up this documentation on Kafka and this Spring documentation, because Kafka is quite different than other message brokers.
I want some messageId kind of thing when I am sending to Rabbit Mq Queue as I will get when sending to IBM MQ using jms. I am using spring MQ amqp starter dependency with Spring Boot. Configuration is done only in application.yml (property file). I am using Rabbit template for sending.
rabbitMqTemplate.convertAndSend(EMPTY_STRING,queueName, message, messagePostProcessor);
I have tried messagePostProcessor. Any help is appreciated. I had a look into below content. But didnt understand how to implement. Does it require special configuration (connectionfactory/ container)?
https://www.rabbitmq.com/confirms.html
Unlike JMS, the rabbit client doesn't assign message ids.
However, you can configure the RabbitTemplate's MessageConverter to create an id, which you can then retrieve with a post processor.
See AbstractMessageConverter...
/**
* Flag to indicate that new messages should have unique identifiers added to their properties before sending.
* Default false.
* #param createMessageIds the flag value to set
*/
public void setCreateMessageIds(boolean createMessageIds) {
this.createMessageIds = createMessageIds;
}
For message confirmations, see the reference manual. But that is unrelated to the message id property.
My environment: spring 4.1, JBoss EAP 6.4, IBM MQ 8.0:
Messages are not redelivered in the case where Listener throws RuntimeException.
I have the following in JmsConfig:
#Bean
DefaultMessageListenerContainer defaultMessageListenerContainer(QueueConnectionFactory connectionFactory, JndiDestinationResolver dr, MessageListener ml) {
DefaultMessageListenerContainer mlc = new DefaultMessageListenerContainer();
mlc.setConnectionFactory(connectionFactory);
mlc.setMessageListener(ml);
mlc.setDestinationName(jndiInQueue);
mlc.setDestinationResolver(dr);
mlc.setSessionTransacted(true);
mlc.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
return mlc;
}
If I use a JmsTransactionManager and pass it to the above method and use like so:
mlc.setTransactionManager(tm)
Following warnings are written to the log:
It is not valid to commit a non-transacted session, and the behavior is the same, no redelivery.
ConnectionFactory is obtained via JNDI, I wonder if sourcing the ConnectionFactory through jndi has something to do with this?
From the AbstractMessageListenerContainer Javadocs:
In order to consistently arrange for redelivery with any container variant, consider "CLIENT_ACKNOWLEDGE" mode or - preferably - setting "sessionTransacted" to "true" instead
There is a similar question on SO.
Flip your ack mode to Session.SESSION_TRANSACTED instead of CLIENT_ACKNOWLEDGE.
Client Ack mode doesn't work as most folks want it.. and is a common "gotcha" in JMS. It acknowledges current message AND all previous messages in the session. It is not per-message acknowledgement.
Edit:
Also check related post-- IBM MQ may require you to use the "XA" versions of the connection factory class.
ref: Websphere Liberty profile - transacted Websphere MQ connection factory