Intro:
We're currently using the spring mail integration to receive and send emails which works without flaws if there's no exception such as a connection error to the exchange server or the database.
These mails come in as Messages and are passed to a handler method which will parse the MimeMessage to a custom mail data object. JPA saves those entities as the last step to our database.
Question/Problem:
There's a problem if the database is down or the mail can't be processed for any other reason, as the IntegrationFlow will still mark it as /SEEN once the message gets passed to the handler.
Setting this flag to false won't fix our problem, because we want Spring to set the /SEEN flag if the mail is processed and saved correctly
shouldMarkMessagesAsRead(false)
Searching for:
Would there be a possibility to set flags AFTER successfully saving the mail to the database?
We'd like to process the failed email again after the cause for the responsible error is fixed, which won't work as long Spring marks them as /SEEN no matter the result.
Reference:
The messages comes in and gets passed to the handler which will parse the mail and execute the CRUD-Repository save(mailDAO) method. The handleMimeMessage() is more or less just a mapper.
#Bean
fun imapIdleFlow(imapProperties: ImapProperties): IntegrationFlow {
imapProperties.username.let { URLEncoder.encode(it, charset) }
return IntegrationFlows
.from(
Mail.imapIdleAdapter(
ImapMailReceiver("imap://${imapProperties.username}:${imapProperties.password}#${imapProperties.host}/Inbox")
.apply {
setSimpleContent(true)
setJavaMailProperties(imapProperties.properties.toProperties())
})
.autoStartup(true)
.shouldReconnectAutomatically(true)
)
.handle(this::handleMimeMessage)
.get()
}
Is it even possible to mark the messages in the same flow afterward as you need to access the exchange a second time or would I need a second flow to get and flag the same mail?
I think it is possible with something like transaction synchronization: https://docs.spring.io/spring-integration/reference/html/mail.html#mail-tx-sync
So, you set transactional(TransactionManager transactionManager) on that Mail.imapIdleAdapter to the JpaTransactionManager to start transaction from this IMAP Idle channel adapter and propagate it to your handleMimeMessage() where you do those JPA saves.
Plus you add:
/**
* Configure a {#link TransactionSynchronizationFactory}. Usually used to synchronize
* message deletion with some external transaction manager.
* #param transactionSynchronizationFactory the transactionSynchronizationFactory.
* #return the spec.
*/
public ImapIdleChannelAdapterSpec transactionSynchronizationFactory(
TransactionSynchronizationFactory transactionSynchronizationFactory) {
To react for commit and rollback of the mentioned transaction.
The DefaultTransactionSynchronizationFactory with some TransactionSynchronizationProcessor impl can give you a desired behavior, where you take a Message and its payload from the provided IntegrationResourceHolder and perform something like message.setFlag(Flag.SEEN, true); on the MimeMessage.
You may consider to use the mentioned in docs an ExpressionEvaluatingTransactionSynchronizationProcessor.
To avoid folder reopening, you may consider to use a public ImapIdleChannelAdapterSpec autoCloseFolder(boolean autoCloseFolder) { with a false option. You need to consider to close it in that TX sync impl or some other way.
Related
In a spring boot application, I have a class with jms listener.
public class PaymentNotification{
#JmsListener(destination="payment")
public void receive(String payload) throws Exception{
//mapstring conversion
....
paymentEvent = billingService.insert(paymentEvent); //transactional method
//call rest...
billingService.save(paymentEvent);
//send info to jms
}
}
I saw then when a error happen, data is inserted in the database, that ok, but it's like receive method is called again and again... but queue is empty when I check on the server.
If there is an error, I don't want method is called again, Is there something for that.
The JMS Message Headers might contain additional information to help with your processing. In particular JMSRedelivered could be of some value. The Oracle doc states that "If a client receives a message with the JMSRedelivered field set, it is likely, but not guaranteed, that this message was delivered earlier but that its receipt was not acknowledged at that time."
I ran the following code to explore what was available in my configuration (Spring Boot with IBM MQ).
#JmsListener(destination="DEV.QUEUE.1")
public void receive(Message message) throws Exception{
for (Enumeration<String> e = message.getPropertyNames(); e.hasMoreElements();)
System.out.println(e.nextElement());
}
From here I could find JMSXDeliveryCount is available in JMS 2.0. If that property is not available, then you may well find something similar for your own configuration.
One strategy would be to use JMSXDeliveryCount, a vendor specific property or maybe JMSRedelivered (if suitable for your needs) as a way to check before you process the message. Typically, the message would be sent to a specific blackout queue where the redelivery count exceeds a set threshold.
Depending on the messaging provider you are using it might also be possible to configure back out queue processing as properties of the queue.
I currently have a Spring Integration application which is utilizing a number of TCP inbound and outbound adapter combinations for message handling. All of these adapter combinations utilize the same single MessageEndpoint for request processing and the same single MessagingGateway for response sending.
The MessageEndpoint’s final output channel is a DirectChannel that is also the DefaultRequestChannel of the MessageGateway. This DirectChannel utilizes the default RoundRobinLoadBalancingStrategy which is doing a Round Robin search for the correct Outbound Adapter to send the given response through. Of course, this round robin search does not always find the appropriate Outbound Adapter on first search and when it doesn’t it logs accordingly. Not only is this producing a large amount of unwanted logging but it also raises some performance concerns as I anticipate several hundred inbound/outbound adapter combinations existing at any given time.
I am wondering if there is a way in which I can more closely correlate the inbound and outbound adapters in a way that there is no need for the round robin processing and each response can be sent directly to the corresponding outbound adapter? Ideally, I would like this to be implemented in a way that the use of a single MessageEndpoint and single MessageGateway can be maintained.
Note: Please limit solutions to those which use the Inbound/Outbound Adapter combinations. The use of TcpInbound/TcpOutboundGateways is not possible for my implementation as I need to send multiple responses to a single request and, to my knowledge, this can only be done with the use of inbound/outbound adapters.
To add some clarity, below is a condensed version of the current implementation described. I have tried to clear out any unrelated code just to make things easier to read...
// Inbound/Outbound Adapter creation (part of a service that is used to dynamically create varying number of inbound/outbound adapter combinations)
public void configureAdapterCombination(int port) {
TcpNioServerConnectionFactory connectionFactory = new TcpNioServerConnectionFactory(port);
// Connection Factory registered with Application Context bean factory (removed for readability)...
TcpReceivingChannelAdapter inboundAdapter = new TcpReceivingChannelAdapter();
inboundAdapter.setConnectionFactory(connectionFactory);
inboundAdapter.setOutputChannel(context.getBean("sendFirstResponse", DirectChannel.class));
// Inbound Adapter registered with Application Context bean factory (removed for readability)...
TcpSendingMessageHandler outboundAdapter = new TcpSendingMessageHandler();
outboundAdapter.setConnectionFactory(connectionFactory);
// Outbound Adapter registered with Application Context bean factory (removed for readability)...
context.getBean("outboundResponse", DirectChannel.class).subscribe(outboundAdapter);
}
// Message Endpoint for processing requests
#MessageEndpoint
public class RequestProcessor {
#Autowired
private OutboundResponseGateway outboundResponseGateway;
// Direct Channel which is using Round Robin lookup
#Bean
public DirectChannel outboundResponse() {
return new DirectChannel();
}
// Removed additional, unrelated, endpoints for readability...
#ServiceActivator(inputChannel="sendFirstResponse", outputChannel="sendSecondResponse")
public Message<String> sendFirstResponse(Message<String> message) {
// Unrelated message processing/response generation excluded...
outboundResponseGateway.sendOutboundResponse("First Response", message.getHeaders().get(IpHeaders.CONNECTION_ID, String.class));
return message;
}
// Service Activator that puts second response on the request channel of the Message Gateway
#ServiceActivator(inputChannel = "sendSecondResponse", outputChannel="outboundResponse")
public Message<String> processQuery(Message<String> message) {
// Unrelated message processing/response generation excluded...
return MessageBuilder.withPayload("Second Response").copyHeaders(message.getHeaders()).build();
}
}
// Messaging Gateway for sending responses
#MessagingGateway(defaultRequestChannel="outboundResponse")
public interface OutboundResponseGateway {
public void sendOutboundResponse(#Payload String payload, #Header(IpHeaders.CONNECTION_ID) String connectionId);
}
SOLUTION:
#Artem's suggestions in the comments/answers below seem to do the trick. Just wanted to make a quick note about how I was able to add a replyChannel to each Outbound Adapter on creation.
What I did was create two maps that are being maintained by the application. The first map is populated whenever a new Inbound/Outbound adapter combination is created and it is a mapping of ConnectionFactory name to replyChannel name. The second map is a map of ConnectionId to replyChannel name and this is populated on any new TcpConnectionOpenEvent via an EventListener.
Note that every TcpConnectionOpenEvent will have a ConnectionFactoryName and ConnectionId property defined based on where/how the connection is established.
From there, whenever a new request is received I use theses maps and the 'ip_connectionId' header on the Message to add a replyChannel header to the Message. The first response is sent by manually grabbing the corresponding replyChannel (based on the value of the replyChannel header) from the application's context and sending the response on that channel. The second response is sent via Spring Integration using the replyChannel header on the message as Artem describes in his responses.
This solution was implemented as a quick proof of concept and is just something that worked for my current implementation. Including this to hopefully jumpstart other viewer's own implementations/solutions.
Well, I see now your point about round-robin. You create many similar TCP channel adapters against the same channels. In this case it is indeed hard to distinguish one flow from another because you have a little control over those channels and their subscribers.
On of the solution would be grate with Spring Integration Java DSL and its dynamic flows: https://docs.spring.io/spring-integration/reference/html/dsl.html#java-dsl-runtime-flows
So, you would concentrate only on the flows and won't worry about runtime registration. But since you are not there and you deal just with plain Java & Annotations configuration, it is much harder for you to achieve a goal. But still...
You may be know that there is something like replyChannel header. It is taken into an account when we don't have a outputChannel configured. This way you would be able to have an isolated channel for each flow and the configuration would be really the same for all the flows.
So,
I would create a new channel for each configureAdapterCombination() call.
Propagate this one into that method for replyChannel.subscribe(outboundAdapter);
Use this channel in the beginning of your particular flow to populate it into a replyChannel header.
This way your processQuery() service-activator should go without an outputChannel. It is going to be selected from the replyChannel header for a proper outbound channel adapter correlation.
You don't need a #MessagingGateway for such a scenario since we don't have a fixed defaultRequestChannel any more. In the sendFirstResponse() service method you just take a replyChannel header and send a newly created message manually. Technically it is exactly the same what you try to do with a mentioned #MessagingGateway.
For Java DSL variant I would go with a filter on the PublishSubscribeChannel to discard those messages which don't belong to the current flow. Anyway it is a different story.
Try to figure out how you can have a reply channel per flow when you configure particular configureAdapterCombination().
I've been working with spring-boot 2.0.0.RC1 using the webflux starter (spring-boot-starter-webflux). I created a simple controller that returns a infinite flux. I would like that the Publisher only does its work if there is a client (Subscriber). Let's say I have a controller like this one:
#RestController
public class Demo {
#GetMapping(value = "/")
public Flux<String> getEvents(){
return Flux.create((FluxSink<String> sink) -> {
while(!sink.isCancelled()){
// TODO e.g. fetch data from somewhere
sink.next("DATA");
}
sink.complete();
}).doFinally(signal -> System.out.println("END"));
}
}
Now, when I try to run that code and access the endpoint http://localhost:8080/ with Chrome, then I can see the data. However, once I close the browser the while-loop continues since no cancel event has been fired. How can I terminate/cancel the streaming as soon as I close the browser?
From this answer I quote that:
Currently with HTTP, the exact backpressure information is not
transmitted over the network, since the HTTP protocol doesn't support
this. This can change if we use a different wire protocol.
I assume that, since backpressure is not supported by the HTTP protocol, it means that no cancel request will be made either.
Investigating a little bit further, by analyzing the network traffic, showed that the browser sends a TCP FIN as soon as I close the browser. Is there a way to configure Netty (or something else) so that a half-closed connection will trigger a cancel event on the publisher, making the while-loop stop?
Or do I have to write my own adapter similar to org.springframework.http.server.reactive.ServletHttpHandlerAdapter where I implement my own Subscriber?
Thanks for any help.
EDIT:
An IOException will be raised on the attempt to write data to the socket if there is no client. As you can see in the stack trace.
But that's not good enough, since it might take a while before the next chunk of data will be ready to send and therefore it takes the same amount of time to detect the gone client. As pointed out in Brian Clozel's answer it is a known issue in Reactor Netty. I tried to use Tomcat instead by adding the dependency to the POM.xml. Like this:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</dependency>
Although it replaces Netty and uses Tomcat instead, it does not seem reactive due to the fact that the browser does not show any data. However, there is no warning/info/exception in the console. Is spring-boot-starter-webflux as of this version (2.0.0.RC1) supposed to work together with Tomcat?
Since this is a known issue (see Brian Clozel's answer), I ended up using one Flux to fetch my real data and having another one in order to implement some sort of ping/heartbeat mechanism. As a result, I merge both together with Flux.merge().
Here you can see a simplified version of my solution:
#RestController
public class Demo {
public interface Notification{}
public static class MyData implements Notification{
…
public boolean isEmpty(){…}
}
#GetMapping(value = "/", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent<? extends Notification>> getNotificationStream() {
return Flux.merge(getEventMessageStream(), getHeartbeatStream());
}
private Flux<ServerSentEvent<Notification>> getHeartbeatStream() {
return Flux.interval(Duration.ofSeconds(2))
.map(i -> ServerSentEvent.<Notification>builder().event("ping").build())
.doFinally(signalType ->System.out.println("END"));
}
private Flux<ServerSentEvent<MyData>> getEventMessageStream() {
return Flux.interval(Duration.ofSeconds(30))
.map(i -> {
// TODO e.g. fetch data from somewhere,
// if there is no data return an empty object
return data;
})
.filter(data -> !data.isEmpty())
.map(data -> ServerSentEvent
.builder(data)
.event("message").build());
}
}
I wrap everything up as ServerSentEvent<? extends Notification>. Notification is just a marker interface. I use the event field from the ServerSentEvent class in order to separate between data and ping events. Since the heartbeat Flux sends events constantly and in short intervals, the time it takes to detect that the client is gone is at most the length of that interval. Remember, I need that because it might take a while before I get some real data that can be sent and, as a result, it might also take a while before it detects that the client is gone. Like this, it will detect that the client is gone as soon as it can’t sent the ping (or possibly the message event).
One last note on the marker interface, which I called Notification. This is not really necessary, but it gives some type safety. Without that, we could write Flux<ServerSentEvent<?>> instead of Flux<ServerSentEvent<? extends Notification>> as return type for the getNotificationStream() method. Or also possible, make getHeartbeatStream() return Flux<ServerSentEvent<MyData>>. However, like this it would allow that any object could be sent, which I don’t want. As a consequence, I added the interface.
I'm not sure why this behaves like this, but I suspect it is because of the choice of generation operator. I think using the following would work:
return Flux.interval(Duration.ofMillis(500))
.map(input -> {
return "DATA";
});
According to Reactor's reference documentation, you're probably hitting the key difference between generate and push (I believe a quite similar approach using generate would probably work as well).
My comment was referring to the backpressure information (how many elements a Subscriber is willing to accept), but the success/error information is communicated over the network.
Depending on your choice of web server (Reactor Netty, Tomcat, Jetty, etc), closing the client connection might result in:
a cancel signal being received on the server side (I think this is supported by Netty)
an error signal being received by the server when it's trying to write on a connection that's been closed (I believe the Servlet spec does not provide that that callback and we're missing the cancel information).
In short: you don't need to do anything special, it should be supported already, but your Flux implementation might be the actual problem here.
Update: this is a known issue in Reactor Netty
I am trying implement sample retry mechanism for RabbitMQ using Spring's StatefulRetryOperationsInterceptor.
As stated in documentation, I need to setup message key generator as the message id is absent. What I don't understand is the real usage of unique id generated per message. i.e. when I used below implementation I did not have any issue with retry:
StatefulRetryOperationsInterceptor interceptor =
RetryInterceptorBuilder.stateful()
.maxAttempts(3)
.backOffOptions(2000, 1, 2000)
.messageKeyGenerator(
new MessageKeyGenerator() {
#Override
public Object getKey(Message message) {
return 1;
}
);
container.setAdviceChain(new Advice[] {interceptor});
Stateful retry needs the originating message to be somehow unique - so the retry "state" for the message can be determined - the simplest way is to have the message publisher add a unique message id header.
However, it's possible something in your message body or some other header might be used to uniquely identify the message. Enter the MessageKeyGenerator which is used to determine the unique id.
Using a constant (1 in your case) won't work because every message has the same message key and will all be considered to be deliveries of the same message (from a retry perspective).
The framework does provide a MissingMessageIdAdvice which can provide limited support for stateful retry (if added to the advice chain before the retry advice). It adds a messageId to the incoming message.
"Limited" means full stateful retry support is not available - only one redelivery attempt is allowed.
If the redelivery fails, the message is rejected which causes it to be discarded or routed to the DLX/DLQ if so configured. In all cases the "temporary" state is removed from the cache.
Generally, if full retry support is needed and there is no messageId property and there is no way to generate a unique key with a MessageKeyGenerator, I would recommend using stateless retry.
I'm experimenting with Spring 4 WebSocket STOMP application. Is there a way to reply to a single unauthenticated user on condition that each user has unique session ID? Right now I can only either broadcast a message or send it directly to an authenticated user.
#Controller
public class ProductController {
#MessageMapping("/products/{id}")
#SendTo("/topic") // This line makes return value to be broadcasted to every connected user.
public String getProduct(#DestinationVariable int id) {
return "Product " + id;
}
}
You can assign an anonymous identity to incoming users. There are two ways to do it.
One, you can configure a sub-class of DefaultHandshakeHandler that overrides determineUser and assigns some kind of identity to every WebSocketSession. This requires 4.0.1 by the way (currently build snapshots are available) that will be released on Monday Jan 23, 2014.
Two, the WebSocket session will fall back on the value returned from HttpServletRequest.getUserPrincipal on the handshake HTTP request. You could have a servlet Filter wrap the HttpServletRequest and decide what to return from that method. Or if you're using Spring Security which has the AnonymousAuthenticationFilter, override its createAuthentication method.
#SendToUser("/products") should result in a message to destination "/user/{username}/products". That message will be handled by the UserDestinationMessageHandler, which transforms the destination to "/products-user{sessionId}" and re-sends the message.
So I'm not quite sure what "/user/products-user0" is. It surprises me in two ways. First if it starts with "/user" then that's the destination before the transformation and should be followed by the user name (i.e. "/user/{username}/products").
The fact that it ends with "-user0" makes it look like the destination after the transformation but then it shouldn't start with "/user". In any case the 0, 1 in that case would be the WebSocket session id. What server is this?