I am trying to set up an integration flow where an input channel is a queue that would act as a non-blocking queue. What I see now is that if I trigger a message processing from a Spring MVC Controller, if a message processing takes time, controller won't return and will wait for message handler (Service Activator) to complete.
Here's an integration configuration
#Bean(name = "createIssue.input")
MessageChannel queueInput() {
return MessageChannels.queue(10)
.get();
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
PollerMetadata poller() {
return Pollers.fixedRate(100)
.maxMessagesPerPoll(1)
.get();
}
#Bean
IntegrationFlow createIssue() {
return IntegrationFlows.from(queueInput())
.split()
.transform(mytransformer, "convert")
.handle(myservice, "createIssue")
.get();
}
myservice and mytransformer are just regular Spring beans.
I have a Spring MVC REST Controller that writes to the createIssue.input queue using a gateway in one of its GET handlers.
If I set a breapoint in myservice.createIssue() method, I can see that controller does not return from it's method, so an external service that triggered controller has to wait for my service to complete.
What I am trying to achieve is to have an async processing queue where a gateway would just write a message into a queue and return immediately. How can I achieve that?
You should use void Gateway there, which really acts as a "just send" component:
public interface Cafe {
#Gateway(requestChannel="orders")
void placeOrder(Order order);
}
And your .handle() in the end of IntegrationFlow should replies to the nullChannel or just doesn't return anything from the createIssue() method.
Related
I was reading about Spring Integration's FluxMessageChannel here and here, but I still don't understand exactly what are the differences between using a DirectChannel and FluxMessageChannel when using Project Reactor. Since the DirectChannel is stateless and controlled by its pollers, I'd expect the FluxMessageChannel to not be needed. I'm trying to understand when exactly should I use each and why, when speaking on Reactive Streams applications that are implemented with Spring Integration.
I currently have a reactive project that uses DirectChannel, and it seems to work fine, even the documentation says:
the flow behavior is changed from an imperative push model to a reactive pull model
I'd like to understand when to use each of the channels and what is the exact difference when working with Reactive Streams.
The DirectChannel does not have any poller and its implementation is very simple: as long as a message is sent to it, the handler is called. In the same caller's thread:
public class DirectChannel extends AbstractSubscribableChannel {
private final UnicastingDispatcher dispatcher = new UnicastingDispatcher();
private volatile Integer maxSubscribers;
/**
* Create a channel with default {#link RoundRobinLoadBalancingStrategy}.
*/
public DirectChannel() {
this(new RoundRobinLoadBalancingStrategy());
}
Where that UnicastingDispatcher is:
public final boolean dispatch(final Message<?> message) {
if (this.executor != null) {
Runnable task = createMessageHandlingTask(message);
this.executor.execute(task);
return true;
}
return this.doDispatch(message);
}
(There is no executor option for the DirectChannel)
private boolean doDispatch(Message<?> message) {
if (tryOptimizedDispatch(message)) {
return true;
}
...
protected boolean tryOptimizedDispatch(Message<?> message) {
MessageHandler handler = this.theOneHandler;
if (handler != null) {
try {
handler.handleMessage(message);
return true;
}
catch (Exception e) {
throw IntegrationUtils.wrapInDeliveryExceptionIfNecessary(message,
() -> "Dispatcher failed to deliver Message", e);
}
}
return false;
}
That's why I call it " imperative push model". The caller is this case is going to wait until the handler finishes its job. And if you have a big flow, everything is going to be stopped in the sender thread until a sent message has reached the end of the flow of direct channels. In two simple words: the publisher is in charge for the whole execution and it is blocked in this case. You haven't faced any problems with your solution based on the DirectChannel just because you didn't use reactive non-blocking threads yet like Netty in WebFlux or MongoDB reactive driver.
The FluxMessageChannel was really designed for Reactive Streams purposes where the subscriber is in charge for handling a message which it pulls from the Flux on demand. This way just after sending the publisher is free to do anything else. Just because it is already a subscriber responsibility to handle the message.
I would say it is definitely OK to use DirectChannel as long as your handlers are not blocking. As long as they are blocking you should go with FluxMessageChannel. Although don't forget that there are other channel types for different tasks: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-implementations
I'm trying to build integration flow, which will prevent the loss of messages during delivery to AMQP broker (rabbitMQ).
In the case of broker stopping, I see some unexpected for me behavior:
Failed messages are saving to the message store, but not for long. This flow isn't waiting for broker availability, it extracts messages from messages store even the broker still be stopped
In case of successful restarting of rabbitmq, records from the message-store(if they still are presented) are not be delivered to the queue.
Please help me in investigations. Code Example:
#Bean
public MessageChannel messageStoreBackedChannel() {
return new QueueChannel(
new MessageGroupQueue(jdbcChannelMessageStore(), "Group_ID")
);
}
#Bean
public IntegrationFlow someFlow() {
return IntegrationFlows
.from("messageStoreBackedChannel")
.channel("amqpMessageChannel")
.get();
}
#Bean
public IntegrationFlow jmsExtractFlow(EntityManagerFactory entityManagerFactory) {
return IntegrationFlows
.from("amqpMessageChannel")
.handle(message -> System.out.println(message.getPayload()))
.get();
}
#Bean
public MessageChannel amqpMessageChannel() {
return new PollableAmqpChannel("amqpMessageChannel", amqpTemplate);
}
#Bean
public JdbcChannelMessageStore jdbcChannelMessageStore() {
var jdbcChannelMessageStore = new JdbcChannelMessageStore(dataSource);
jdbcChannelMessageStore.setChannelMessageStoreQueryProvider(new PostgresChannelMessageStoreQueryProvider());
return jdbcChannelMessageStore;
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setTrigger(new PeriodicTrigger(10));
return pollerMetadata;
}
Consider to configure an endpoint in between your .from("messageStoreBackedChannel").channel("amqpMessageChannel") as transactional().
Something like this:
.from("messageStoreBackedChannel")
.bridge(e -> e.poller(p -> p.fixedDelay(10).transactional()))
.channel("amqpMessageChannel")
So, whenever delivery to the amqpMessageChannel fails, a transaction is going to roll back and the failed message will come back to the store until the next poll.
Of course you can stop that bridge endpoint when you get an error connecting to RabbitMQ. But how can you determine then that connection comes back?..
I am using SpringBoot with Spring AMQP and I want to use RPC pattern using synchronous sendAndReceive method in producer. My configuration assumes 1 exchange with 2 distinct bindings (1 for each operation on the same resource). I want to send 2 messages with 2 different routingKeys and receive response on distinct reply-to queues
Problem is, as far as I know, sendAndReceive will wait for reply on a queue with name ".replies" so both replies will be sent to products.replies queue (at least that is my understanding).
My publisher config:
#Bean
public DirectExchange productsExchange() {
return new DirectExchange("products");
}
#Bean
public OrderService orderService() {
return new MqOrderService();
}
#Bean
public RabbitTemplate rabbitTemplate(final ConnectionFactory connectionFactory) {
final RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(producerJackson2MessageConverter());
return rabbitTemplate;
}
#Bean
public Jackson2JsonMessageConverter producerJackson2MessageConverter() {
return new Jackson2JsonMessageConverter();
}
and the 2 senders:
...
final Message response = template.sendAndReceive(productsExchange.getName(), "products.get", message);
...
final Message response = template.sendAndReceive(productsExchange.getName(), "products.stock.update", message);
...
consumer config:
#Bean
public Queue getProductQueue() {
return new Queue("getProductBySku");
}
#Bean
public Queue updateStockQueue() {
return new Queue("updateProductStock");
}
#Bean
public DirectExchange exchange() {
return new DirectExchange("products");
}
#Bean
public Binding getProductBinding(DirectExchange exchange) {
return BindingBuilder.bind(getProductQueue())
.to(exchange)
.with("products.get");
}
#Bean
public Binding modifyStockBinding(DirectExchange exchange) {
return BindingBuilder.bind(updateStockQueue())
.to(exchange)
.with("products.stock.update");
}
and #RabbitListeners with following sigratures:
#RabbitListener(queues = "getProductBySku")
public Message getProduct(GetProductResource getProductResource) {...}
#RabbitListener(queues = "updateProductStock")
public Message updateStock(UpdateStockResource updateStockResource) {...}
I noticed that the second sender receives 2 responses, one of which is of invalid type (from first receiver). Is there any way in which I can make these connections distinct? Or is using separate exchange for each operation the only reasonable solution?
as far as I know, sendAndReceive will wait for reply on a queue with name ".replies"
Where did you get that idea?
Depending on which version you are using, either a temporary reply queue will be created for each request or RabbitMQ's "direct reply-to" mechanism is used, which again means each request is replied to on a dedicated pseudo queue called amq.rabbitmq.reply-to.
I don't see any way for one producer to get another's reply; even if you use an explicit reply container (which is generally not necessary any more), the template will correlate the replies to the requests.
Try enabling DEBUG logging to see if provides any hints.
Please refer to system diagram attached.
system diagram here
ISSUE: When I try to post message to input channel, the code tries to connect to the DB and throws an exception that it is unable to connect.
Code inside 5 -> Read from a channel, apply Business Logic (empty for now) and send the response to another channel.
#Bean
public IntegrationFlow sendToBusinessLogictoNotifyExternalSystem() {
return IntegrationFlows
.from("CommonChannelName")
.handle("Business Logic Class name") // Business Logic empty for now
.channel("QueuetoAnotherSystem")
.get();
}
I have written the JUnit for 5 as given below,
#Autowired
PublishSubscribeChannel CommonChannelName;
#Autowired
MessageChannel QueuetoAnotherSystem;
#Test
public void sendToBusinessLogictoNotifyExternalSystem() {
Message<?> message = (Message<?>) MessageBuilder.withPayload("World")
.setHeader(MessageHeaders.REPLY_CHANNEL, QueuetoAnotherSystem).build();
this.CommonChannelName.send((org.springframework.messaging.Message<?>) message);
Message<?> receive = QueuetoAnotherSystem.receive(5000);
assertNotNull(receive);
assertEquals("World", receive.getPayload());
}
ISSUE: As you can see from the system diagram, my code also has a DB connection on a different flow.
When I try to post message to producer channel, the code tries to connect to the DB and throws an exception that it is unable to connect.
I do not want this to happen, because the JUnit should never be related to the DB, and should run anywhere, anytime.
How do I fix this exception?
NOTE: Not sure if it matters, the application is a Spring Boot application. I have used Spring Integration inside the code to read and write from/to queues.
Since the common channel is a publish/subscribe channel, the message goes to both flows.
If this is a follow-up to this question/answer, you can prevent the DB flow from being invoked by calling stop() on the sendToDb flow (as long as you set ignoreFailures to true on the pub/sub channel, like I suggested there.
((Lifecycle) sendToDb).stop();
JUNIT TEST CASE - UPDATED:
#Autowired
PublishSubscribeChannel CommonChannelName;
#Autowired
MessageChannel QueuetoAnotherSystem;
#Autowired
SendResponsetoDBConfig sendResponsetoDBConfig;
#Test
public void sendToBusinessLogictoNotifyExternalSystem() {
Lifecycle flowToDB = ((Lifecycle) sendResponsetoDBConfig.sendToDb());
flowToDB.stop();
Message<?> message = (Message<?>) MessageBuilder.withPayload("World")
.setHeader(MessageHeaders.REPLY_CHANNEL, QueuetoAnotherSystem).build();
this.CommonChannelName.send((org.springframework.messaging.Message<?>) message);
Message<?> receive = QueuetoAnotherSystem.receive(5000);
assertNotNull(receive);
assertEquals("World", receive.getPayload());
}
CODE FOR 4: The flow that handles message to DB
public class SendResponsetoDBConfig {
#Bean
public IntegrationFlow sendToDb() {
System.out.println("******************* Inside SendResponsetoDBConfig.sendToDb ***********");
return IntegrationFlows
.from("Common Channel Name")
.handle("DAO Impl to store into DB")
.get();
}
}
NOTE: ******************* Inside SendResponsetoDBConfig.sendToDb *********** never gets printed.
I am currently new to Spring Integration.
Basically trying to poll onto multiple file locations asynchronously with Java Spring integration DSL. I am required to get the file name and perform some operations with filename and push the file to S3 finally, my question is can these tasks of performing operations with file be performed in the task executor or the service activator handler . I am not sure which is the right place.
#Autowired
private AWSFileManager awsFileManager;
#Bean
public IntegrationFlow inboundChannelFlow(#Value("${file.poller.delay}") long delay,
#Value("${file.poller.messages}") int maxMsgsPerPoll,
TaskExecutor taskExecutor, MessageSource<File> fileSource)
{
return IntegrationFlows.from(fileSource,
c -> c.poller(Pollers.fixedDelay(delay)
.taskExecutor(taskExecutor)
.maxMessagesPerPoll(maxMsgsPerPoll)))
.handle("AWSFileManager", "fileUpload")
.channel(ApplicationConfiguration.inboundChannel)
.get();
}
#Bean
TaskExecutor taskExecutor(#Value("${file.poller.thread.pool.size}") int poolSize) {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
//Runnable task1 = () -> {this.methodsamp();};
taskExecutor.setCorePoolSize(poolSize);
//taskExecutor.execute(task1);
return taskExecutor;
}
#Async
public void methodsamp()
{
try
{
awsFileManager.fileUpload();
System.out.println("test");
}
catch(Exception ex)
{
}
I have attached the sample code here.
Also is there a way I could retrieve the filename of the files in the channel as I need to pass this as parameter to the fileUpload method.
Please advise.
Your question isn't clear. The TaskExecutor is for the thread context in the flow. The Service Activator (.handle()) is exactly for your business logic method. This one can be performed on a thread from the executor. And you really use them in your IntegrationFlow correctly.
The FileReadingMessageSource produces message with the java.io.File as a payload. So, that is the way to get a file name - just from File.getName()!