I've been trying to configure spring integration dsl to read from a Tibco EMS topic , do some processing on the received message and then push it to an ActiveMQ queue. I was able to set this up successfully using XML configuration, but wanted to use spring integration dsl instead. I couldn't figure out, neither could find any help online about it.
My configuration for pushing message to ActiveMQ is something like this -
#Bean
public IntegrationFlow toActiveMQFlow(
MessageChannel channel,
ActiveMQQueue queue,
CachingConnectionFactory cachingConnectionFactory) {
return IntegrationFlows.from(channel)
.transform(Object::toString)
.handle(Jms.outboundAdapter(cachingConnectionFactory).destination(queue))
.get();
}
And I'm thinking that the configuration for reading from Tibco EMS topics should be something like this -
#Bean
public IntegrationFlow fromTibcoTopicFlow(
MessageChannel channel,
ConnectionFactory tibcoEmsConnectionFactory,
Topic tibcoTopic
) {
return IntegrationFlows
.from(SomeInboundAdapter(tibcoEmsConnectionFactory).destination(tibcoTopic))
.transform(Object::toString)
.channel(channel)
.get();
}
Since I did not find much help on the latter configuration, is resorting to the XML configuration my only option here?
Kindly correct/edit/point out any mistakes I've made, still learning Spring Integration DSL.
Appreciate your help!
You need to use a Jms.messageDrivenChannelAdapter(ConnectionFactory connectionFactory).
And souldn't use a spring-integration-java-dsl. It was merged to the core project since version 5.0: https://docs.spring.io/spring-integration/docs/5.0.9.RELEASE/reference/html/whats-new.html#_java_dsl
We have fixed the issue with an old Java DSL jar on classpath: https://jira.spring.io/browse/INT-4551
Related
#Component
#RequiredArgsConstructor
public class EventListener {
private final EventProcessingService eventProcessingService;
#JmsListener(destination = "inputQueue", constainerFactory = "myContainerFactory)
public void receiveMessage(Message message) {
eventProcessingService.doSome(message).subscribe(); // return Mono<Void>
}
}
#Service
public class EventProcessingService {
public Mono<Void> doSome(Message message) {
//...
}
}
#Configuration
#RequiredArgsConstructor
public class MqIntegration {
private final ConnectionFactory connectionFactory;
#Bean
public Publisher<Message<String>> mqReactiveFlow() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(this.connectionFactory)
.destination("testQueue"))
.channel(MessageChannels.queue())
.toReactivePublisher();
}
}
I have some webflux application which interacts with ibm mq and a JmsListener which listens for messages from the queue when a message is received EventProcessingService makes requests to other services depending on the messages.
I would like to know how I can create a JmsListener that works with reactive threads using Spring Integration. In other words I want to know if it is possible to create an Integration flow which will receive messages from the queue and call the EvenProcessingService when the messages are received so that it does not have a negative effect on the threads inside webflux application
I think we need to clean up some points in your question.
WebFlux is not a project by itself. It is Spring Framework module about Web on top of reactive server: https://docs.spring.io/spring-framework/docs/current/reference/html/web-reactive.html#spring-webflux
The #JmsListener is a part of another Spring Framework module - spring-jms. And there is nothing relevant to threads used by reactive server for WebFlux layer. https://docs.spring.io/spring-framework/docs/current/reference/html/integration.html#jms
Spring Integration is a separate project which implement EIP on top of Spring Framework dependency injection container. It indeed has its own WebFlux module for channel adapters on top of WebFlux API in Spring Framework: https://docs.spring.io/spring-integration/docs/current/reference/html/webflux.html#webflux. And it also has a JMS module on top of JMS module from Spring Framework: https://docs.spring.io/spring-integration/docs/current/reference/html/jms.html#jms. However there is nothing related to #JmsLisntener since its Jms.messageDrivenChannelAdapter() fully covers that functionality and from a big height it does it the same way - via MessageListenerContainer.
All of this is might not be relevant to the question, but it is better to have a clear context of what you are asking so we will feel that we are on the same page with you.
Now trying to answer to your concern.
As long as you don't deal with JMS from WebFlux layer (#RequestMapping or WebFlux.inboundGateway()), you don't effect those non-blocking thread. The JMS MessageListenerContainer spawns its own threads and perform pulling from the queue and message processing.
What you are explaining with your JMS configuration and service looks more like this:
#Bean
public IntegrationFlow mqReactiveFlow() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(this.connectionFactory)
.destination("testQueue"))
.handle(this.eventProcessingService)
.nullChannel();
}
There is really no reason to shift messages just after JMS into a QueueChannel since JMS listening is already an async operation.
We need that nullChannel in the end of your flow just because your service method returns Mono and framework knows nothing what to do with that. Starting with version 5.4.3 the NullChannel is able to subscribe to the Publisher payload of the message produced to it.
You could have though a FluxMessageChannel in between to really simulate a back-pressure for JMS listener, but that won't make to much different for your next service.
I think you are going to have to bypass #JmsListener as that is registering an on message, which although asynchronous isn't going to be reactive. JMS is essentially blocking, so patching a reactive layer on top, is going to be just a patch.
You will need to use the Publisher that you have created to generate the back pressure. I think you are going to have to define and instantiate your own listener bean which does something on the lines of :
public Flux<String> mqReactiveListener() {
return Flux.from(mqReactiveFlow())
.map(Message::getPayload);
}
I have a local ActiveMQ server and i want to poll messages from a queue named "test" using Spring Integration.
After i have polled the message i want to send it to another channel which would write it on a text file in the file system.
I have seen some examples using
<int-jms:message-driven-channel-adapter id="jmsIn" destination="inQueue" channel="exampleChannel"/>
I want to create this JMS "poller" using Java Annotations. I could not find any reference on how to replace the above XML stuff to annotations.
Could anyone provide a working snippet that would have connection factory configuration and jms:message-driven-channel-adapter done with annotations?
P.S. Here is a reference that has XML configuration
https://examples.javacodegeeks.com/enterprise-java/spring/integration/spring-boot-integration-activemq-example/
Thanks a lot in advance !
Well, for proper Java & Annotations configuration you need to consider to use Spring Integration Java DSL.
Here is some example for the <int-jms:message-driven-channel-adapter> equivalent:
#Bean
public IntegrationFlow jmsMessageDrivenRedeliveryFlow() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(jmsConnectionFactory())
.errorChannel(IntegrationContextUtils.ERROR_CHANNEL_BEAN_NAME)
.destination("jmsMessageDrivenRedelivery")
.configureListenerContainer(c -> c
.transactionManager(mock(PlatformTransactionManager.class))
.id("jmsMessageDrivenRedeliveryFlowContainer")))
.<String, String>transform(p -> {
throw new RuntimeException("intentional");
})
.get();
}
To write to file you need to use a Files.outboundAdapter(): https://docs.spring.io/spring-integration/docs/5.0.6.RELEASE/reference/html/files.html#_configuring_with_the_java_dsl_9
I agree that we are missing similar Docs for JMS part, so feel free to raise a JIRA on the matter.
I use Kafka and Spring Boot with Spring Kafka. After abnormal application termination and then restart, my application started receiving the old, already processed messages from Kafka queue.
What may be the reason for that and how to find and resolve the issue?
my Kafka properties:
spring.kafka.bootstrap-servers=${kafka.host}:${kafka.port}
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.group-id=postfenix
spring.kafka.consumer.enable-auto-commit=false
My Spring Kafka factory and listener:
#Bean
public ConcurrentKafkaListenerContainerFactory<String, Post> postKafkaListenerContainerFactory(KafkaProperties kafkaProperties) {
ConcurrentKafkaListenerContainerFactory<String, Post> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.getContainerProperties().setAckMode(AckMode.MANUAL);
factory.setConsumerFactory(postConsumerFactory(kafkaProperties));
return factory;
}
#KafkaListener(topics = "${kafka.topic.post.send}", containerFactory = "postKafkaListenerContainerFactory")
public void sendPost(ConsumerRecord<String, Post> consumerRecord, Acknowledgment ack) {
Post post = consumerRecord.value();
// do some logic here
ack.acknowledge();
}
When using Kafka, the clients need to commit offsets themselves. This is in contrast to other message brokers, such as AMQP brokers, where the broker keeps track of messages a client did already receive.
In your case, you do not commit offsets automatically and therefore Kafka expects you to commit them manually (because of this setting: spring.kafka.consumer.enable-auto-commit=false). If you do not commit offsets manually in your program, the behaviour you describe is pretty much the expected one. Kafka simply does not know what messages your program did process successfully. Each time you restart your program, Kafka will see that your program did not commit any offsets yet and will apply the strategy you provide in spring.kafka.consumer.auto-offset-reset=earliest, which means the first message in the queue.
If this is all new to you, I suggest reading up this documentation on Kafka and this Spring documentation, because Kafka is quite different than other message brokers.
Is it in a way possible to, say in memory, start a broker that can be used to execute automated test cases using Spring Integration MQTT?
I've tried achieving this with ActiveMQ (following https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-messaging.html) but somehow didn't succeed, maybe anyone has a short working example?
It's not Spring Integration (Spring Boot) responsibility to provide some embedded broker for such a protocol. If there is one, we could consider to implement an auto-configuration on the matter , similar to what we do for embedded RDBMS, JMS and MongoDB. You really need to consult ActiveMQ documentation.
Looks like we can do it like this in the test class:
private static BrokerService activeMQBroker;
...
#BeforeClass
public static void setup() throws Exception {
activeMQBroker = new BrokerService();
activeMQBroker.addConnector("mqtt://localhost:1883");
activeMQBroker.setPersistent(false);
activeMQBroker.setUseJmx(false);
activeMQBroker.start();
}
I didn't try it, but this is exactly what I do to test against STOMP.
I am trying to configure a Spring Batch listener to send a message to a Spring Integration Gateway for StepExecution events.
The following link explains how to configure this with XML
http://docs.spring.io/spring-batch/trunk/reference/html/springBatchIntegration.html#providing-feedback-with-informational-messages
How can this be setup using Spring Integration DSL? I've found no way to configure a gateway with a service interface using DSL.
At the moment I worked around this by implementing an actual StepExecutionListener, and have this then calling an interface which is annotated with #MessagingGateway (calling the corresponding #Gateway method) in order to get a message to a channel. And I then setup an Integration DSL flow for this channel.
Is there a simpler way using DSL, avoiding that workaround? Is there some way to connect a Batch listener direct to a gateway, like one can using XML config?
Cheers,
Menno
First of all SI DSL is just an extension of existing SI Java and Annotation configuration, so it can be used together with any other Java config. Of course an XML #Import is also posible.
There is no gateway configuration in the DSL, because its methods can't be wired with linear IntegrationFlow. There is need to provide downstream flows for each method.
So, #MessagingGateway is a right way to go ahead:
#MessagingGateway(name = "notificationExecutionsListener", defaultRequestChannel = "stepExecutionsChannel")
public interface MyStepExecutionListener extends StepExecutionListener {}
From other side #MessagingGateway parsing as well as <gateway> tag parsing ends up with GatewayProxyFactoryBean definition. So, you just can declare that bean, if you don't want to introduce a new class:
#Bean
public GatewayProxyFactoryBean notificationExecutionsListener(MessageChannel stepExecutionsChannel) {
GatewayProxyFactoryBean gateway = new GatewayProxyFactoryBean(StepExecutionListener.class);
gateway.setDefaultRequestChannel(stepExecutionsChannel);
return gateway;
}
After the latest Milestone 3 I have an idea to introduce nested flows, when we may be able to introduce Gateway support for flows. Something like this:
#Bean
public IntegrationFlow gatewayFlow() {
return IntegrationFlows
.from(MyGateway.class, g ->
g.method("save", f -> f.transform(...)
.filter(...))
.method("delete", f -> f.handle(...)))
.handle(...)
.get();
}
However I'm not sure that it will simplify the life, as far as any nested Lambda just adds more noise and might break loosely coupling principle.