Spring Integration: Manual channel handling - spring

What I want: Build a configurable library that
uses another library that has an internal routing and a subscribe method like: clientInstance.subscribe(endpoint, (endpoint, message) -> <handler>) , e.g. Paho MQTT library
later in my code I want to access the messages in a Flux.
My idea:
create MessageChannels like so:
integrationFlowContext
.registration(IntegrationFlows.from("message-channel:" + endpoint)).bridge().get())
.register()
forward to reactive publishers:
applicationContext.registerBean(
"publisher:" + endpoint,
Publisher.class,
() -> IntegrationFlows.from("message-channel:" + endpoint)).toReactivePublisher()
);
keep the message channels in a set or similar and implement the above handler: (endpoint, message) -> messageChannels.get(endpoint).send( <converter>(message))
later use (in a #PostConstruct method):
Flux
.from((Publihser<Message<?>>)applicationContext.getBean("publisher:" + enpoint))
.map(...)
.subscribe()
I doubt this to be the best way to do what I want. Feels like abusing spring integration. Any suggestions are welcome at this point.
In general however (at least in my tests) this seemed to be working. But when I run my application, I get errors like: "Caused by: org.springframework.messaging.core.DestinationResolutionException: no output-channel or replyChannel header available".
This is especially bad, since after this exception the publishers claim to not have a subscriber anymore. Thus, in a real application no messages are proceeded anymore.
I am not sure what this message means, but I can kind of reproduce it (but don't understand why):
#Test
public void channelTest() {
integrationFlowContext
.registration(
IntegrationFlows.from("any-channel").bridge().get()
)
.register();
registryUtil.registerBean(
"any-publisher",
Publisher.class,
() -> IntegrationFlows.from("any-channel").toReactivePublisher()
);
Flux
.from((Publisher<Message<?>>) applicationContext.getBean("any-publisher"))
.subscribe(System.out::println);
MessageChannel messageChannel = applicationContext.getBean("any-channel", MessageChannel.class);
try {
messageChannel.send(MessageBuilder.withPayload("test").build());
} catch (Throwable t) {
log.error("Error: ", t);
}
}
I of course read parts of the spring integration documentation, but don't quite get what happens behind the scenes. Thus, I feel like guessing possible error causes.
EDIT:
This, however works:
#TestConfiguration
static class Config {
GenericApplicationContext applicationContext;
Config(
GenericApplicationContext applicationContext,
IntegrationFlowContext integrationFlowContext
) {
this.applicationContext = applicationContext;
// optional here, but needed for some reason in my library,
// since I can't find the channel beans like I will do here,
// if I didn't register them like so:
//integrationFlowContext
// .registration(
// IntegrationFlows.from("any-channel").bridge().get())
// .register();
applicationContext.registerBean(
"any-publisher",
Publisher.class,
() -> IntegrationFlows.from("any-channel").toReactivePublisher()
);
}
#PostConstruct
void connect(){
Flux
.from((Publisher<Message<?>>) applicationContext.getBean("any-publisher"))
.subscribe(System.out::println);
}
}
#Autowired
ApplicationContext applicationContext;
#Autowired
IntegrationFlowContext integrationFlowContext;
#Test
#SneakyThrows
public void channel2Test() {
MessageChannel messageChannel = applicationContext.getBean("any-channel", MessageChannel.class);
try {
messageChannel.send(MessageBuilder.withPayload("test").build());
} catch (Throwable t) {
log.error("Error: ", t);
}
}
Thus apparently my issue above is realted to messages arriving "too early" .. I guess?!

No, your issue is related to round-robin dispatched on the DirectChannel for the any-channel bean name.
You define two IntegrationFlow instances starting with that channel and then you declare their own subscribers, but at runtime both of them are subscribed to the same any-channel instance. And that one comes with the round-robin balancer by default. So, one message goes to your Flux.from() subscriber, but another to that bridge() which doesn't know what to do with your message, so it tries to resolve a replyChannel header.
Therefore your solution just only with one IntegrationFlows.from("any-channel").toReactivePublisher() is correct. Although you could just do a FluxMessageChannel registration and use it from one side for regular messages sending and from other side as a reactive source for Flux.from().

Related

Difference between DirectChannel and FluxMessageChannel

I was reading about Spring Integration's FluxMessageChannel here and here, but I still don't understand exactly what are the differences between using a DirectChannel and FluxMessageChannel when using Project Reactor. Since the DirectChannel is stateless and controlled by its pollers, I'd expect the FluxMessageChannel to not be needed. I'm trying to understand when exactly should I use each and why, when speaking on Reactive Streams applications that are implemented with Spring Integration.
I currently have a reactive project that uses DirectChannel, and it seems to work fine, even the documentation says:
the flow behavior is changed from an imperative push model to a reactive pull model
I'd like to understand when to use each of the channels and what is the exact difference when working with Reactive Streams.
The DirectChannel does not have any poller and its implementation is very simple: as long as a message is sent to it, the handler is called. In the same caller's thread:
public class DirectChannel extends AbstractSubscribableChannel {
private final UnicastingDispatcher dispatcher = new UnicastingDispatcher();
private volatile Integer maxSubscribers;
/**
* Create a channel with default {#link RoundRobinLoadBalancingStrategy}.
*/
public DirectChannel() {
this(new RoundRobinLoadBalancingStrategy());
}
Where that UnicastingDispatcher is:
public final boolean dispatch(final Message<?> message) {
if (this.executor != null) {
Runnable task = createMessageHandlingTask(message);
this.executor.execute(task);
return true;
}
return this.doDispatch(message);
}
(There is no executor option for the DirectChannel)
private boolean doDispatch(Message<?> message) {
if (tryOptimizedDispatch(message)) {
return true;
}
...
protected boolean tryOptimizedDispatch(Message<?> message) {
MessageHandler handler = this.theOneHandler;
if (handler != null) {
try {
handler.handleMessage(message);
return true;
}
catch (Exception e) {
throw IntegrationUtils.wrapInDeliveryExceptionIfNecessary(message,
() -> "Dispatcher failed to deliver Message", e);
}
}
return false;
}
That's why I call it " imperative push model". The caller is this case is going to wait until the handler finishes its job. And if you have a big flow, everything is going to be stopped in the sender thread until a sent message has reached the end of the flow of direct channels. In two simple words: the publisher is in charge for the whole execution and it is blocked in this case. You haven't faced any problems with your solution based on the DirectChannel just because you didn't use reactive non-blocking threads yet like Netty in WebFlux or MongoDB reactive driver.
The FluxMessageChannel was really designed for Reactive Streams purposes where the subscriber is in charge for handling a message which it pulls from the Flux on demand. This way just after sending the publisher is free to do anything else. Just because it is already a subscriber responsibility to handle the message.
I would say it is definitely OK to use DirectChannel as long as your handlers are not blocking. As long as they are blocking you should go with FluxMessageChannel. Although don't forget that there are other channel types for different tasks: https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-implementations

Exponential backoff for business exceptions when using reactive spring-amqp?

I'm using Spring AMQP 2.1.6 and Spring Boot 2.1.5 and I'm looking for the recommended way to configure spring-amqp to retry business exceptions for reactive components (Mono) with exponential backoff. For example:
#RabbitListener
public Mono<Void> myListener(MyMessage myMessage) {
Mono<Void> mono = myService.doSomething(myMessage);
return mono;
}
I'd like spring-amqp to retry automatically if doSomething returns an error. Usually one can configure this for blocking RabbitListener's when setting up the container factory:
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
...
factory.setAdviceChain(retryInterceptor(..));
Where retryInterceptor might be defined like this:
private static RetryOperationsInterceptor retryInterceptor(long backoffInitialInterval, double backoffMultiplier, long backoffMaxInterval, int maxAttempts) {
ExponentialBackOffPolicy backOffPolicy = new ExponentialBackOffPolicy();
backOffPolicy.setInitialInterval(backoffInitialInterval);
backOffPolicy.setMultiplier(backoffMultiplier);
backOffPolicy.setMaxInterval(backoffMaxInterval);
RetryTemplate retryTemplate = new RetryTemplate();
retryTemplate.setRetryPolicy((new SimpleRetryPolicy(maxAttempts)));
retryTemplate.setBackOffPolicy(backOffPolicy);
StatelessRetryOperationsInterceptorFactoryBean bean = new StatelessRetryOperationsInterceptorFactoryBean();
bean.setRetryOperations(retryTemplate);
return bean.getObject();
}
But the advice chain doesn't seem to be used for reactive RabbitListener's. This is probably because, if I understand it correctly, the RetryTemplate/ExponentialBackOffPolicy actually blocks the thread.
As a workaround I could of course do something like (switching to Kotlin because it's a bit easier):
#RabbitListener
fun myListener(MyMessage myMessage) : Mono<Void> {
return myService.doSomething(myMessage)
.retryExponentialBackoff(10, Duration.ofMillis(100), Duration.ofSeconds(5)) { ctx ->
log.info("Caught exception ${ctx.exception()}")
}
}
But I'd like this retry logic to be applied to for all instances of Mono returned from RabbitListener's. Is something like this possible or should you configure this another way when using reactive sequences from project reactor with spring-amqp?
It is really better to apply retry logic into your reactive sequence, similar way you do with the retryExponentialBackoff(). Just because the Reactive Streams execution doesn't happen on the same thread we can apply that Retrytemplate for the myListener().
Right now the logic internally is like this:
private static class MonoHandler {
static boolean isMono(Object result) {
return result instanceof Mono;
}
#SuppressWarnings("unchecked")
static void subscribe(Object returnValue, Consumer<? super Object> success,
Consumer<? super Throwable> failure) {
((Mono<? super Object>) returnValue).subscribe(success, failure);
}
}
That Consumer<? super Throwable> failure does this:
private void asyncFailure(Message request, Channel channel, Throwable t) {
this.logger.error("Future or Mono was completed with an exception for " + request, t);
try {
channel.basicNack(request.getMessageProperties().getDeliveryTag(), false, true);
}
catch (IOException e) {
this.logger.error("Failed to nack message", e);
}
}
So, we don't have any way to to initiate that RetryTemplate at all, but at the same time with an explicit basicNack() we have a natural retry with the re-fetching the same message from the RabbitMQ back.
We could probably apply a Reactor retry for that Mono internally, but it doesn't look like RetryOperationsInterceptor can simply be converted to the Mono.retry().
So, in other words, the RetryOperationsInterceptor is a wrong way for reactive processing. Use Mono.retry() explicitly in your own code.
You may expose some common utility method and apply it as a Mono.transform(Function<? super Mono<T>, ? extends Publisher<V>> transformer) whenever you have a reactive return for the #RabbitListener method.

Use JSON transformer in Spring Integration

I have a problem that seems unaddressed in any of the examples I can find.
My application reads an ActiveMQ topic of JSON messages. It will build a completely new outbound REST call based on this data. Note that this is not a "transformation". It is given "X" produce "Y" i.e. ServiceActivator.
My flows thus far are
public IntegrationFlow splitInputFlow() {
return IntegrationFlows.from("inboundJmsChannel")
.split()
.log(LoggingHandler.Level.DEBUG)
.route(Message.class, m -> m.getHeaders().get("x-bn-class").equals("Healthcheck.class") ? "healthcheckChannel" : "metricChannel")
.get();
}
public IntegrationFlow healthcheckFlow() {
return IntegrationFlows.from("healthcheckChannel")
.log(LoggingHandler.Level.DEBUG)
.transform(Transformers.fromJson(Healthcheck.class))
.handle("healthcheckActivator", "process")
.get();
}
There are dozens of examples on how to use spring transformers. I have even considered trying a MessageConverter. But I don't see why it would help and it doesn't seem to be the normal approach.
The main problem here is that Integration calls healthcheckActivator.process(String payload). The payload itself is the expected valid JSON string.
I am a little surprised it does not call healtcheckActivator.process(Message payload) but But that wouldn't help so it doesn't much matter.
The real question is why does it not call healtcheckActivator.process(Healthcheck healthcheck)?
Well actually I understand "why". It is because DSL generates an internal channel to tie the steps together and as far as I understand anything on a channel is a spring.messaging.Message.
I can easily instantiate my Healthcheck object once I get inside the SA. But that leaves the nagging question: What possible good is the entire transform step? If it always "serializes" the object back into a Message -- what's the point.
Like I said I think I'm missing something fundamental here.
EDIT
My new (and probably last) idea is maybe I'm publishing it wrong.
To publish it I am using
jmsTemplate.convertAndSend(topicName, healthcheck, messagePostProcessor -> {
messagePostProcessor.setJMSType("TextMessage");
messagePostProcessor.setStringProperty("x-bn-class", "Healthcheck.class");
messagePostProcessor.setStringProperty("x-bn-service-name", restEndpoint.getServiceName());
messagePostProcessor.setStringProperty("x-bn-service-endpoint-name", restEndpoint.getEndpointName());
messagePostProcessor.setLongProperty("x-bn-heathcheck-timestamp", queryDate);
messagePostProcessor.setStringProperty("x-bn-healthcheck-status", subsystemStatus.getStatus(subsystemStatus));
messagePostProcessor.setIntProperty("httpStatus", httpStatus.value());
return messagePostProcessor;
});
What arrives in the SI process(String payload) method is:
LoggingHandler - GenericMessage [payload={"healthcheckType":"LOCAL","outcome":"PASS","dependencyType":"DB","endpoint":"NODE TABLE","description":"Read from DB","durationSecs":0.025}, headers={x-bn-service-name=TG10-CS2, x-bn-service-endpoint-name=TG Q10-CS2 Ready Check, jms_destination=topic://HEALTH_MONITOR, _type=com.healthcheck.response.Healthcheck, x-bn-heathcheck-timestamp=1558356538000, priority=4, jms_timestamp=1558356544244, x-bn-healthcheck-status=SEV0, jms_redelivered=false, x-bn-class=Healthcheck.class, httpStatus=200, jms_type=TextMessage, id=b29ffea7-7128-c543-9a14-8bab450f0ac6, jms_messageId=ID:39479-1558356520091-1:2:1:1:1, timestamp=1558356544409}]
I hadn't noticed the _type parameter in the jms_destination header before. But before I started screwing around with this (because it didn't work) that is the correct class name for what the other team provided.
I have not implemented a JMS message converter. But the supplied SimpleMessageConverter seems that it should do exactly what I want.
Your understanding is correct; works fine for me, so something else is going on...
#SpringBootApplication
public class So56169938Application {
public static void main(String[] args) {
SpringApplication.run(So56169938Application.class, args);
}
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from(() -> "{\"foo\":\"bar\"}", e -> e.poller(Pollers.fixedDelay(5000)))
.transform(Transformers.fromJson(Foo.class))
.handle("myBean", "method")
.get();
}
#Bean
public MyBean myBean() {
return new MyBean();
}
public static class MyBean {
public void method(Foo foo) {
System.out.println(foo);
}
}
public static class Foo {
private String foo;
String getFoo() {
return this.foo;
}
void setFoo(String foo) {
this.foo = foo;
}
#Override
public String toString() {
return "Foo [foo=" + this.foo + "]";
}
}
}
and
Foo [foo=bar]
Foo [foo=bar]
Foo [foo=bar]
Foo [foo=bar]
Foo [foo=bar]
Foo [foo=bar]
Well, Spring Integration is a Messaging framework. It transfers messages from endpoint to endpoint via channels in between. That's already the target endpoint responsibility to deal with consumed message the proper way. The framework doesn't care about the payload. It is really a business part of the target application. That's how we can make framework components as generic as possible leaving the room for target business types for end-users.
Anyway the Framework provides some mechanisms to interact with payloads. We call it POJO method invocation. So, you provide some business with arbitrary contract, however following some Spring Integration rules: https://docs.spring.io/spring-integration/docs/current/reference/html/#service-activator.
So, according your description it is really a surprise that it doesn't work for healtcheckActivator.process(Healthcheck healthcheck). Your transform(Transformers.fromJson(Healthcheck.class)) should really produce a Message with Healthcheck object as a payload. The framework consults a method signature and tries to map a payload and/or headers to the method invocation arguments, having the whole message as a container for data to delegate to the method call.
From here it would be great to see your healtcheckActivator.process() method to determine why the transform(Transformers.fromJson(Healthcheck.class)) result cannon be mapped to that method arguments.

Spring Integration + SpringBoot JUnit tries to connect to DB unexpectedly

Please refer to system diagram attached.
system diagram here
ISSUE: When I try to post message to input channel, the code tries to connect to the DB and throws an exception that it is unable to connect.
Code inside 5 -> Read from a channel, apply Business Logic (empty for now) and send the response to another channel.
#Bean
public IntegrationFlow sendToBusinessLogictoNotifyExternalSystem() {
return IntegrationFlows
.from("CommonChannelName")
.handle("Business Logic Class name") // Business Logic empty for now
.channel("QueuetoAnotherSystem")
.get();
}
I have written the JUnit for 5 as given below,
#Autowired
PublishSubscribeChannel CommonChannelName;
#Autowired
MessageChannel QueuetoAnotherSystem;
#Test
public void sendToBusinessLogictoNotifyExternalSystem() {
Message<?> message = (Message<?>) MessageBuilder.withPayload("World")
.setHeader(MessageHeaders.REPLY_CHANNEL, QueuetoAnotherSystem).build();
this.CommonChannelName.send((org.springframework.messaging.Message<?>) message);
Message<?> receive = QueuetoAnotherSystem.receive(5000);
assertNotNull(receive);
assertEquals("World", receive.getPayload());
}
ISSUE: As you can see from the system diagram, my code also has a DB connection on a different flow.
When I try to post message to producer channel, the code tries to connect to the DB and throws an exception that it is unable to connect.
I do not want this to happen, because the JUnit should never be related to the DB, and should run anywhere, anytime.
How do I fix this exception?
NOTE: Not sure if it matters, the application is a Spring Boot application. I have used Spring Integration inside the code to read and write from/to queues.
Since the common channel is a publish/subscribe channel, the message goes to both flows.
If this is a follow-up to this question/answer, you can prevent the DB flow from being invoked by calling stop() on the sendToDb flow (as long as you set ignoreFailures to true on the pub/sub channel, like I suggested there.
((Lifecycle) sendToDb).stop();
JUNIT TEST CASE - UPDATED:
#Autowired
PublishSubscribeChannel CommonChannelName;
#Autowired
MessageChannel QueuetoAnotherSystem;
#Autowired
SendResponsetoDBConfig sendResponsetoDBConfig;
#Test
public void sendToBusinessLogictoNotifyExternalSystem() {
Lifecycle flowToDB = ((Lifecycle) sendResponsetoDBConfig.sendToDb());
flowToDB.stop();
Message<?> message = (Message<?>) MessageBuilder.withPayload("World")
.setHeader(MessageHeaders.REPLY_CHANNEL, QueuetoAnotherSystem).build();
this.CommonChannelName.send((org.springframework.messaging.Message<?>) message);
Message<?> receive = QueuetoAnotherSystem.receive(5000);
assertNotNull(receive);
assertEquals("World", receive.getPayload());
}
CODE FOR 4: The flow that handles message to DB
public class SendResponsetoDBConfig {
#Bean
public IntegrationFlow sendToDb() {
System.out.println("******************* Inside SendResponsetoDBConfig.sendToDb ***********");
return IntegrationFlows
.from("Common Channel Name")
.handle("DAO Impl to store into DB")
.get();
}
}
NOTE: ******************* Inside SendResponsetoDBConfig.sendToDb *********** never gets printed.

How to set a Message Handler programmatically in Spring Cloud AWS SQS?

maybe someone has an idea to my following problem:
I am currently on a project, where i want to use the AWS SQS with Spring Cloud integration. For the receiver part i want to provide a API, where a user can register a "message handler" on a queue, which is an interface and will contain the user's business logic, e.g.
MyAwsSqsReceiver receiver = new MyAwsSqsReceiver();
receiver.register("a-queue-name", new MessageHandler(){
#Override
public void handle(String message){
//... business logic for the received message
}
});
I found examples, e.g.
https://codemason.me/2016/03/12/amazon-aws-sqs-with-spring-cloud/
and read the docu
http://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html#_sqs_support
But the only thing i found there to "connect" a functionality for processing a incoming message is a annotation on a method, e.g. #SqsListener or #MessageMapping.
These annotations are fixed to a certain queue-name, though. So now i am at a loss, how to dynamically "connect" my provided "MessageHandler" (from my API) to the incoming message for the specified queuename.
In the Config the example there is a SimpleMessageListenerContainer, which gets a QueueMessageHandler set, but this QueueMessageHandler does not seem
to be the right place to set my handler or to override its methods and provide my own subclass of QueueMessageHandler.
I already did something like this with the Spring Amqp integration and RabbitMq and thought, that it would be also similar here with AWS SQS.
Does anyone have an idea, how to accomplish this?
thx + bye,
Ximon
EDIT:
I found, that Spring JMS could actually do that, e.g. www.javacodegeeks.com/2016/02/aws-sqs-spring-jms-integration.html. Does anybody know, what consequences using JMS protocol has here, good or bad?
I am facing the same issue.
I am trying to go in an unusual way where I set up an Aws client bean at build time and then instead of using sqslistener annotation to consume from the specific queue I use the scheduled annotation which I can programmatically pool (each 10 secs in my case) from which queue I want to consume.
I did the example that iterates over queues defined in properties and then consumes from each one.
Client Bean:
#Bean
#Primary
public AmazonSQSAsync awsSqsClient() {
return AmazonSQSAsyncClientBuilder
.standard()
.withRegion(Regions.EU_WEST_1.getName())
.build();
}
Consumer:
// injected in the constructor
private final AmazonSQSAsync awsSqsClient;
#Scheduled(fixedDelay = 10000)
public void pool() {
properties.getSqsQueues()
.forEach(queue -> {
val receiveMessageRequest = new ReceiveMessageRequest(queue)
.withWaitTimeSeconds(10)
.withMaxNumberOfMessages(10);
// reading the messages
val result = awsSqsClient.receiveMessage(receiveMessageRequest);
val sqsMessages = result.getMessages();
log.info("Received Message on queue {}: message = {}", queue, sqsMessages.toString());
// deleting the messages
sqsMessages.forEach(message -> {
val deleteMessageRequest = new DeleteMessageRequest(queue, message.getReceiptHandle());
awsSqsClient.deleteMessage(deleteMessageRequest);
});
});
}
Just to clarify, in my case, I need multiple queues, one for each tenant, with the queue URL for each one passed in a property file. Of course, in your case, you could get the queue names from another source, maybe a ThreadLocal which has the queues you have created in runtime.
If you wish, you can also try the JMS approach where you create message consumers and add a listener to each one you wish (See the doc Aws Jms documentation).
When we do Spring and SQS we use the spring-cloud-starter-aws-messaging.
Then just create a Listener class
#Component
public class MyListener {
#SQSListener(value="myqueue")
public void listen(MyMessageType message) {
//process the message
}
}

Resources