I'm working on a project that makes use of Spring's JmsTemplate, ActiveMQ and Hibernate. I have a method wrapped in a transaction which sends a message through the JmsTemplate, does a bit more work and then returns so that the transaction can commit. I want the message to only be sent once the transaction commits, i.e. the JmsListener should only trigger once the aforementioned method returns.
Take the following example sender and receiver:
#Service
#Transactional
public class TestService{
#Autowired
private JmsTemplate jmsTemplate;
public void test() throws InterruptedException {
jmsTemplate.convertAndSend("test_queue", "Test");
Thread.sleep(1000L);
System.out.println("This should run first");
}
}
#Service
#Transactional
public class Listener {
#JmsListener(destination = "test_queue", containerFactory = "jmsListenerContainerFactory")
public void onMessage() {
System.out.println("This should run last.");
}
}
I want the text "This should run first" to print before "This should run last", but because of the Thread.sleep it never does! I tried a number of changes to the configuration on my jmsListenerContainerFactory, but none make any difference.
Not sure if XA is involved in this case. Is the actual send of the message part of a separate transaction? If so the issue is probably that the two transactions aren't synchronizing, but I don't know how to solve that.
I had to set Session Transacted on JmsTemplate instead of JmsListenerContainerFactory:
#Bean
public JmsTemplate jmsTemplate(ConnectionFactory connectionFactory) {
JmsTemplate jmsTemplate = new JmsTemplate(connectionFactory);
jmsTemplate.setSessionTransacted(true);
return jmsTemplate;
}
Related
I'm very new with Kafka.
Using spring-boot kafka, I developed a publisher and a consumer using one Message object and manual ack. My code uses spring annotation. That's works perfectly.
Now, when I connect to production brokers, this one not send one Message but a list of message.
My listener method has the following signature:
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload Message message, Acknowledgment ack)
so I can acknowledge each Message. Good.
But now it's seems I must replace it with
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload List<Message> messages, Acknowledgment ack)
Even following the documentation it seems that I should use
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload List<Message> messages, Acknowledgment ack, Consumer<?,?> consumer)
Should I set batchmode to true ?
Now the question is : how can I acknowledge each message when this one has been completely handled?
Many many thanks for your help
Something like this one can help you either if you do want to manually commit offset.
If you do not want it then switch setAckMode to other value.
Here's this thing done the spring-way.
CoreAutoConfiguration class:
#Configuration
#Import({KafkaAutoConfiguration.class})
public class CoreAutoConfiguration {
#Bean("batchKafkaListenerContainerFactory")
public ConcurrentKafkaListenerContainerFactory<?, ?> batchKafkaListenerContainerFactory(ConcurrentKafkaListenerContainerFactoryConfigurer configurer, ConsumerFactory<Object, Object> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory);
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
factory.setBatchListener(true);
return factory;
}
}
Then there goes your Config class:
#Configuration
#Import({
CoreAutoConfiguration.class,
KafkaAutoConfiguration.class,
})
#EnableKafka
#EnableRetry
public class Config {
}
Finally the consumer:
#KafkaListener(
topics = "MessagesTopic",
containerFactory = "batchKafkaListenerContainerFactory"
)
public void dataReceived(#Payload List<String> payload) throws RuntimeException {
yourService.processIncomingData(payload);
}
And lastly, the properties:
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=helloworld
spring.kafka.listener.type=batch
spring.kafka.consumer.enable-auto-commit=false
# this is size of incoming list if broker has this many entries, can be lower eventually
spring.kafka.consumer.max-poll-records=100
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
My app successfully sends Kafka messages, but only after Kafka is initialized. Before that i get the error "Dispatcher has no subscribers". How do i wait for subscribers to finish being registered for channels?
Here's a trace of the order of events (timing in second.ms):
17.165 SenderClass created
17.816 initialization class, #PostConstruct starts PollingTask
24.781 PollingTask sends first Kafka message
24.816 First error: "Dispatcher has no subscribers"
25.778 Registering MessageChannel my-channel
still seeing Dispatcher errors
27.067 Channel my-channel' has 1 subscriber
No more errors after this, messages send fine
i'm not sure how to approach this. Wild guesses have included:
Place sending code in #PostConstruct
Add #AutoConfigureBefore(BindingServiceConfiguration.class) to Sender
Add #AutoConfigureAfter(BindingServiceConfiguration.class) to SenderClass
Add #AutoConfigureBefore(BindingServiceConfiguration.class) to Main
Place #DependsOn({"EnableBindingClass"}) on Task
Place #DependsOn({"ApplicationLifeCycle"}) on SchedulerClass, where ApplicationLifeCycle is a class that does nothing but
implements SmartLifecycle with getPhase returning MAX_INT
Making sure ComponentScan is on for whole package (a suggestion from other SO threads)
Various combinations of the above
Created a new app, made it as simple as i could:
public interface Source {
#Output(channelName)
MessageChannel outboundChannel();
}
#EnableBinding(Source.class)
#Component
public class Sender {
#Autowired
private Source source;
public boolean send(SomeObject object) {
return source.outboundChannel().send(MessageBuilder.withPayload(object).build());
}
#Service
public class Scheduler {
#Autowired
Sender sender;
#Autowired
ThreadPoolTaskScheduler taskScheduler;
#PostConstruct
public void initialize() {
taskScheduler.schedule(new PollingTask(), nextTime);
}
private class PollingTask implements Runnable {
#Override
public void run() {
List<SomeObject> objects = getDummyData();
for(SomeObject object : objects)
{
sender.send(interval);
}
Instant nextTime = Instant.now().plusMillis(1_000L);
try {
taskScheduler.schedule(new PollingTask(), nextTime);
} catch (Exception e) {
logger.error(e);
}
}
}
Edit to add Solution
It works now! In my scheduler that starts the things that send the messages i switched from starting things in #PostConstruct to SmartLifecycle::start().
#Service
public class Scheduler implements SmartLifecycle {
#Autowired
Sender sender;
#Autowired
ThreadPoolTaskScheduler taskScheduler;
#Override
public void start() {
taskScheduler.schedule(new PollingTask(), nextTime);
}
private class PollingTask implements Runnable {
#Override
public void run() {
List<SomeObject> objects = getDummyData();
for(SomeObject object : objects)
{
sender.send(interval);
}
Instant nextTime = Instant.now().plusMillis(1_000L);
try {
taskScheduler.schedule(new PollingTask(), nextTime);
} catch (Exception e) {
logger.error(e);
}
}
}
#PostConstruct is too early to send messages; the context is still being built.. Implememt SmartLifecycle, put the bean in a high phase (Integer.MAX_VALUE) and do the sends in start().
Or do the sends in an ApplicationRunner.
I faced a similar problem in Webflux + Spring Cloud Stream functional style. Spring Cloud Function in 2022 is the preferred way.
My hypothesis after a lot of debugging was that beans were not created in right order. The bean was probably not registered in spring-cloud-stream's dispatchers before kafka message processing started. similar to what #gary mentioned.
So I added #Order(1) before my consumer beans. Hoping that this bean would be created before it is dispatcher-registrations starts.
#Bean
#Order(1)
public Function<Flux<Message<Pojo>>, Mono<Void>> pojoConsumer() {
This seems to fix my issue for now.
So I got a advanced queue working with a Connectionfactory:
ConnectionFactory jmsQueueConnectionFactory() throws JMSException, SQLException {
final OracleDataSource dataSource = new OracleDataSource();
dataSource.setUser(username);
dataSource.setPassword(password);
dataSource.setURL(url);
dataSource.setImplicitCachingEnabled(true);
dataSource.setFastConnectionFailoverEnabled(true);
return AQjmsFactory.getConnectionFactory(dataSource);
}
This is running on a shared Database which might be restarted or sometimes the network just has a short hickup. Which results in no more messages from the Queue.
I use a spring MessageListener to retrieve messages and there is actually no indicator or what so ever that the queue is not running anymore. After restarting the application I then get a load of older messages that should have been processed already.
Is there a way or specific data source implementation which reconnects or something?
Update: Listener Impl
#Bean
OracleAqQueueFactoryBean etlQueueFactory() throws JMSException, SQLException {
final OracleAqQueueFactoryBean bean = new OracleAqQueueFactoryBean();
bean.setConnectionFactory(jmsQueueConnectionFactory());
bean.setOracleQueueUser("USER");
bean.setOracleQueueName("QUEUE");
return bean;
}
#Bean
DefaultMessageListenerContainer jmsContainer() throws JMSException, SQLException {
final DefaultMessageListenerContainer bean = new DefaultMessageListenerContainer();
bean.setConnectionFactory(jmsQueueConnectionFactory());
bean.setDestination(etlQueueFactory().getObject());
bean.setMessageListener(new MyListener());
bean.setSessionTransacted(false);
return bean;
}
public class MyListener implements MessageListener {
#Override
public void onMessage(Message message) {
...
}
}
I guess you have to do it on the JMS level, not DB level.
Not sure what type of listener you are using, but a DefaultMessageListenerContainer in Spring is implemented with a consumer.receive(timeout) loop. It's more robust than using a plain listener as it will attempt to reconnect on each poll cycle (if needed).
Here are my Hornetq configuration in spring boot.
spring.hornetq.mode=embedded
spring.hornetq.embedded.enabled=true
spring.hornetq.embedded.persistent=true
spring.hornetq.port=5445
spring.hornetq.embedded.queues=jms.testqueue
Here is my Producer
public class Producer {#Inject
private JmsTemplate jmsTemplate;
public void resolveError( String message) {
try{
jmsTemplate.convertAndSend(DATA_QUEUE, message);
}catch(Exception e){
//log error
}
}}
Here is my Consumer
#JmsListener(destination = DATA_QUEUE)
public void consume(String message) throws InterruptedException {
log.info("Receiving event: {}", message);
try {
//do stuff with message
}catch (Exception e){
log.error(e.toString());
}
}
Here is my config file
#Configuration#EnableJms public class JmsConfig {
public static final String LOGGING_SCRAPPER_KEY ="DATA_SYNC_ERROR";
public static final String DATA_QUEUE = "jms.testqueue"; }
I want to slow down the consuming process of #JMSlistener, I don't want to the JMS listener hit the queue all the time any help is appreciated, thanks!!
The listeners that are created under the covers for each #JmsListener annotated method are held in a registry as explained in the documentation
If you want to pause your listener, it is very easy to look it up and stop it. Let's assume you have a way to invoke the following bean (JMX endpoint, secure rest mapping, whatever):
static class YourService {
private final JmsListenerEndpointRegistry registry;
#Autowired
public YourService(JmsListenerEndpointRegistry registry) {
this.registry = registry;
}
public void stopListener() {
this.registry.getListenerContainer("myListener").stop();
}
public void startListener() {
this.registry.getListenerContainer("myListener").start();
}
}
Then you need to associate the proper id to your listener (myListener) in the example above.
#JmsListener(id = "myListener", destination = DATA_QUEUE)
public void consume(String message) throws InterruptedException { ... }
I'm not able to set the consuming time of JmsListener but I found an alternative where I'm able to set delivery delay time limit on jmsTemplate instead, use jmsTemplate setDeliveryDelay which will delay sending it to the queue. Either way, it is delayed only if you go with delaying the consuming process of JMS listener you will have the message in the queue in my approach it won't be in the queue until the delay delivery time.
I am using Spring AMQP's MessageListenerContainer for recieving messages from RabbitMq Broker . Though I am able to receive message inside the listener , autowiring is not working inside listener .
Here is how I have configured my Listener
#Bean
public SimpleMessageListenerContainer listenerContainer() {
SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer();
listenerContainer.setConnectionFactory(this.inputQueueMgr
.getRabbitConnectionFactory());
JsonMessageConverter converter = new JsonMessageConverter();
listenerContainer.setMessageConverter(converter);
listenerContainer.setMessageListener(new InputQueueEventDispatcher());
listenerContainer.setQueueNames("queue1");
listenerContainer.setAcknowledgeMode(AcknowledgeMode.AUTO);
listenerContainer.setPrefetchCount(1);
return listenerContainer;
}
Here is the class where I am listening to the messages from rabbitMq
#Component(value = "inputMessageListner")
public class InputQueueEventDispatcher implements Serializable, MessageListener {
private static final long serialVersionUID = -5391659256992655430L;
#Autowired
private volatile InputQueueManagerImpl inputQueueMgr;
#Autowired
private volatile NotificationQueueManagerImpl notificationManager;
#Value("${input.rabbitmq.events.queue}")
private String queueName;
#Autowired
private SubscriptionRepository subRepository;
#SuppressWarnings({ "unchecked" })
#Override
public void onMessage(Message message) {
try {
String messageContent = new String(message.getBody());
.....
}
The problem is inside onMessage(Message message) all the autowire components are coming as null .
PS-> Please note that I have declared all the autowire instances as #Component and doing appropriate ComponentScan to scan their packages appropriately . Infact these components do get autowired in normal flow but since onMessage(Message message) gets executed on a seperate thread , these values are showing null . Is there any way to enable autowiring here inside listener .
Thanks
You've set a #Component annotation on your listener, but you don't get this bean from the context. Instead, you're creating the instance yourself, using new. So Spring has no way to know that this instance has been created and must be autowired.
Remove that #Component annotation, and change your code to
#Bean
public SimpleMessageListenerContainer listenerContainer() {
SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer();
listenerContainer.setConnectionFactory(this.inputQueueMgr
.getRabbitConnectionFactory());
JsonMessageConverter converter = new JsonMessageConverter();
listenerContainer.setMessageConverter(converter);
listenerContainer.setMessageListener(inputMessageListener());
listenerContainer.setQueueNames("queue1");
listenerContainer.setAcknowledgeMode(AcknowledgeMode.AUTO);
listenerContainer.setPrefetchCount(1);
return listenerContainer;
}
#Bean
public InputQueueEventDispatcher inputMessageListener() {
return new InputQueueEventDispatcher();
}
Now, since the bean is returned from a #Bean-annotated method, Spring will make is a Spring bean and autowire it.
Most probably the other thread you mentioned on your question is not getting that instance from spring's application context. So no injection happens for it.
You should use
#RabbitListener(queues = "queue_name")
on your onMessage method.
But you should also change onMessage method syntax and and
onMessage(Message<String> originalMessage)
that way spring will automatically call that method with message