I came across call back queue feature in RMQ. And its pretty fancy too. The whole idea is I have created One Message queue (queue1), its callback queue(queue1_cb) and its dlq(queue1_dlq). I am implementing HA feature with 2 nodes.
The problem comes when I am deploying 2 instances of my application(I have one sender and one receiver app in Spring boot). Both are listening to same HA cluster. The scenario is as below.
Sender publishes a message to RMQ.
Receiver app consumes message. Receiver app has to call third party API which is socket based API and its asynchronous so i do not get response in same connection. SO i store object of Channel & Message which i need to ack the message. (Please note i am delaying the ack till i receive response from third party API.
When i deploy 2 instances of receiver app, any instance will get response from third party API. And both will not have object of Channel and Message to ack message and send message to callback queue.
Can any one suggest me a solution on proiority?
Below is my code.
At Receiver side :
#Override
public void onMessage(Message arg0, Channel arg1) throws Exception {
String msg = new String (arg0.getBody());
AppObject obj = mapper.readValue(msg, AppObject.class);
Packet packet = new Packet();
packet.setChannel(arg1);
packet.setMessage(arg0);
packet.setAppObject(obj);
AppParam.objects.put(
String.valueOf(key , packet);
//Call third party API
}
At the time of acking and sending callback message:
public boolean pushMessageToCallBack(String key , AppObject packet, Channel channel, Message message){
RabbitTemplate replyRabbitTemplate = //Get the RabbitTemplate object. It is handled properly.
replyRabbitTemplate.convertAndSend(packet);
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
}
You need a different callback queue for each instance or, more simply, just use Direct Reply-to where you don't need a queue at all.
Related
How can I view/browse the message on the queue programmatically? I don't want to consume the message.
Below is the code I am using to send the message:
#Autowired
private JmsTemplate producerJmsTemplate;
#Value("${.jms.host}")
private String jmsHost;
public void sendMessage(String message, String destQueue) {
this.producerJmsTemplate.convertAndSend(destQueue, message);
}
If you're using JMS then you can use a QueueBrowser. The Solace documentation has additional details:
Create a QueueBrowser object.
Call Session.createBrowser(...) and pass in the queue that you want to browse.
Optionally, you can also pass in a selector string for the Selector property. Using a selector enables the client to only browse messages that match a selector. Note that it could take longer for the message bus to evaluate spooled messages against a selector, especially if the selector used is complex. For more information on selectors, refer to Selectors.
Use the QueueBrowser to get an Enumeration object that can be used to browse the current queue messages in the order they were received (from oldest to newest) by the queue.
Call QueueBrowser.getEnumeration().
Iterate over the messages on the queue.
Call Enumeration.hasMoreElements(). This method returns true if there is at least one message available in the browser’s local message buffer; otherwise, it returns false.
If it returns false, it does not necessarily mean that the queue is empty, rather the local buffer does not currently contain any messages. Subsequent calls to Enumeration.hasMoreElements() or Enumeration.nextElement() could return true and return a message, respectively.
I have jms message endpoint like:
#Bean
public JmsMessageDrivenEndpoint fsJmsMessageDrivenEndpoint(ConnectionFactory fsConnectionFactory,
Destination fsInboundDestination,
MessageConverter fsMessageConverter) {
return Jms.messageDrivenChannelAdapter(fsConnectionFactory)
.destination(fsInboundDestination)
.jmsMessageConverter(fsMessageConverter)
.outputChannel("fsChannelRouter.input")
.errorChannel("fsErrorChannel.input")
.get();
}
So, my questions is did I get next message before current message will be processed? If it will...Did it will get all messages in mq queue until it fills up all the memory? How to avoid it?
The JmsMessageDrivenEndpoint is based on the JmsMessageListenerContainer, its threading model and MessageListener callback for pulled messages. As long as your MessageListener blocks, it doesn't go to the next message in the queue to pull. When we build an integration flow starting with JmsMessageDrivenEndpoint, it becomes as a MessageListener callback. As long as we process the message downstream in the same thread (DirectChannel by default in between endpoints), we don't pull the next message from JMS queue. If you place a QueueChannel or an ExecutorChannel in between, you shift a processing to a different thread. The current one (JMS listener) gets a control back and it is ready to pull the next message. And in this case your concern about the memory is correct. You can still use QueueChannel with limited size or your ExecutorChannel can be configured with limited thread pool.
In any way my recommendation do not do any thread shifting in the flow when you start from JMS listener container. It is better to block for the next message and let the current transaction to finish its job. So you won't lose a message when something crashes.
I am following the quick-start guide on Spring websocket with sockJs and Stomp here:
https://spring.io/guides/gs/messaging-stomp-websocket/
At this point, my code looks like to one from guide and works as intended. I have a controller class with a method accepting incoming messages and sending them back to all who subscribed on the topic.
What I want to do, is to change the code, so my #MessageMapping annotated method sends response to all subscribers excluding the one who send the message to the controller in the first place (because the sender is also subscribed to the same topic, but i dont want the sender to keep receiving messages it send itself, it is kind of a loop I guess).
I have seen many docs describing how to send a message to a single subscriber, but have not yet seen on describing how to send to all but one - the initial message sender.
Is there any built-in way to do this easily in Spring websocket?
Ok so i've managed to find some solution which works for me at this point of time:
i was able to filter subscribers by principal user name.
I got all simp users form org.springframework.messaging.simp.user.SimpUserRegistry,
and a current sender from org.springframework.messaging.simp.stomp.StompHeaderAccessor.
My code looks something like this:
#MessageMapping("/game/doStuff")
public void gameGrid(DoStuffMessage doStuffMessage,
StompHeaderAccessor headers) {
sendTo("/game/doStuff", doStuffMessage, headers);
}
private void sendTo(String destination, Object payload, StompHeaderAccessor headers) {
Optional<String> user = Optional.ofNullable(headers.getUser())
.map(Principal::getName);
if (user.isPresent()) {
List<String> subscribers = simpUserRegistry.getUsers().stream()
.map(SimpUser::getName)
.filter(name -> !user.get().equals(name))
.collect(Collectors.toList());
subscribers
.forEach(sub -> simpMessagingTemplate.convertAndSendToUser(sub, destination, payload));
}
}
Client is subscribing to /user/game/doStuff
It works for now. What I am worried about is if this code can scale horizontally - if someone has any insight on this I'd greatly appreciate that.
I want to implement an asynchronous mechanism using websockets.
Here's the idea:
The client performs a REST call
The server returns a "subscribingID" and starts a background process
The client registers as subscriber on this topic (suppose 12232442 is the id):
this.stompClient.subscribe('/callback/12232442', (messageOutput) => {
let mess = JSON.parse(messageOutput.body);
console.log(mess);
});
Once done the server simply sends the message and closes the connection:
stompSession.send("callback/12232442", new MessageOutput());
It should work but here's the catch: how can I be sure that another client can't simply subscribe to an ID that exists but does not belong to them?
Also, is there any built-in mechanism to achieve this?
When the server receives a REST request for a subscription ID, you can store the newly generated ID in a Subscription HashMap.
In order to do processing when a new subscription request comes you can implement a custom StompEventHandler, like so
#Controller
public class StompEventHandler{
#EventListener
public void handleSubscription(SessionSubscribeEvent event) {
//Get incoming sessionDetails from event.
//get the destination.
// Validate that the destination is present in Subscription HashMap
// and also that no client maps to the topic id.
// Based on the result either send the message or send Unauth message to
client.
}
}
Documentation
Note that you have to store details about session ID of the client as well for this. Instead of broadcasting the message to /topic/callback/<your_id>, you would need to send the message to destination like so: /user/queue/callback/<your_id>. For sending to a destination as such you would need to use simpMessagingTemplate.convertAndSendToUser(username, destination, payload, Headers)
Good Read for this
So since you are sending messages to only a particular session of a particular user, your messages are confidential.
If you want to ensure that you do not even have the subscription from the client you can send an UNSUBSCRIBE message to the client in the StompEventHandler class. This would force unsubscribe the client.
Good Read for this
I currently have a Spring Integration application which is utilizing a number of TCP inbound and outbound adapter combinations for message handling. All of these adapter combinations utilize the same single MessageEndpoint for request processing and the same single MessagingGateway for response sending.
The MessageEndpoint’s final output channel is a DirectChannel that is also the DefaultRequestChannel of the MessageGateway. This DirectChannel utilizes the default RoundRobinLoadBalancingStrategy which is doing a Round Robin search for the correct Outbound Adapter to send the given response through. Of course, this round robin search does not always find the appropriate Outbound Adapter on first search and when it doesn’t it logs accordingly. Not only is this producing a large amount of unwanted logging but it also raises some performance concerns as I anticipate several hundred inbound/outbound adapter combinations existing at any given time.
I am wondering if there is a way in which I can more closely correlate the inbound and outbound adapters in a way that there is no need for the round robin processing and each response can be sent directly to the corresponding outbound adapter? Ideally, I would like this to be implemented in a way that the use of a single MessageEndpoint and single MessageGateway can be maintained.
Note: Please limit solutions to those which use the Inbound/Outbound Adapter combinations. The use of TcpInbound/TcpOutboundGateways is not possible for my implementation as I need to send multiple responses to a single request and, to my knowledge, this can only be done with the use of inbound/outbound adapters.
To add some clarity, below is a condensed version of the current implementation described. I have tried to clear out any unrelated code just to make things easier to read...
// Inbound/Outbound Adapter creation (part of a service that is used to dynamically create varying number of inbound/outbound adapter combinations)
public void configureAdapterCombination(int port) {
TcpNioServerConnectionFactory connectionFactory = new TcpNioServerConnectionFactory(port);
// Connection Factory registered with Application Context bean factory (removed for readability)...
TcpReceivingChannelAdapter inboundAdapter = new TcpReceivingChannelAdapter();
inboundAdapter.setConnectionFactory(connectionFactory);
inboundAdapter.setOutputChannel(context.getBean("sendFirstResponse", DirectChannel.class));
// Inbound Adapter registered with Application Context bean factory (removed for readability)...
TcpSendingMessageHandler outboundAdapter = new TcpSendingMessageHandler();
outboundAdapter.setConnectionFactory(connectionFactory);
// Outbound Adapter registered with Application Context bean factory (removed for readability)...
context.getBean("outboundResponse", DirectChannel.class).subscribe(outboundAdapter);
}
// Message Endpoint for processing requests
#MessageEndpoint
public class RequestProcessor {
#Autowired
private OutboundResponseGateway outboundResponseGateway;
// Direct Channel which is using Round Robin lookup
#Bean
public DirectChannel outboundResponse() {
return new DirectChannel();
}
// Removed additional, unrelated, endpoints for readability...
#ServiceActivator(inputChannel="sendFirstResponse", outputChannel="sendSecondResponse")
public Message<String> sendFirstResponse(Message<String> message) {
// Unrelated message processing/response generation excluded...
outboundResponseGateway.sendOutboundResponse("First Response", message.getHeaders().get(IpHeaders.CONNECTION_ID, String.class));
return message;
}
// Service Activator that puts second response on the request channel of the Message Gateway
#ServiceActivator(inputChannel = "sendSecondResponse", outputChannel="outboundResponse")
public Message<String> processQuery(Message<String> message) {
// Unrelated message processing/response generation excluded...
return MessageBuilder.withPayload("Second Response").copyHeaders(message.getHeaders()).build();
}
}
// Messaging Gateway for sending responses
#MessagingGateway(defaultRequestChannel="outboundResponse")
public interface OutboundResponseGateway {
public void sendOutboundResponse(#Payload String payload, #Header(IpHeaders.CONNECTION_ID) String connectionId);
}
SOLUTION:
#Artem's suggestions in the comments/answers below seem to do the trick. Just wanted to make a quick note about how I was able to add a replyChannel to each Outbound Adapter on creation.
What I did was create two maps that are being maintained by the application. The first map is populated whenever a new Inbound/Outbound adapter combination is created and it is a mapping of ConnectionFactory name to replyChannel name. The second map is a map of ConnectionId to replyChannel name and this is populated on any new TcpConnectionOpenEvent via an EventListener.
Note that every TcpConnectionOpenEvent will have a ConnectionFactoryName and ConnectionId property defined based on where/how the connection is established.
From there, whenever a new request is received I use theses maps and the 'ip_connectionId' header on the Message to add a replyChannel header to the Message. The first response is sent by manually grabbing the corresponding replyChannel (based on the value of the replyChannel header) from the application's context and sending the response on that channel. The second response is sent via Spring Integration using the replyChannel header on the message as Artem describes in his responses.
This solution was implemented as a quick proof of concept and is just something that worked for my current implementation. Including this to hopefully jumpstart other viewer's own implementations/solutions.
Well, I see now your point about round-robin. You create many similar TCP channel adapters against the same channels. In this case it is indeed hard to distinguish one flow from another because you have a little control over those channels and their subscribers.
On of the solution would be grate with Spring Integration Java DSL and its dynamic flows: https://docs.spring.io/spring-integration/reference/html/dsl.html#java-dsl-runtime-flows
So, you would concentrate only on the flows and won't worry about runtime registration. But since you are not there and you deal just with plain Java & Annotations configuration, it is much harder for you to achieve a goal. But still...
You may be know that there is something like replyChannel header. It is taken into an account when we don't have a outputChannel configured. This way you would be able to have an isolated channel for each flow and the configuration would be really the same for all the flows.
So,
I would create a new channel for each configureAdapterCombination() call.
Propagate this one into that method for replyChannel.subscribe(outboundAdapter);
Use this channel in the beginning of your particular flow to populate it into a replyChannel header.
This way your processQuery() service-activator should go without an outputChannel. It is going to be selected from the replyChannel header for a proper outbound channel adapter correlation.
You don't need a #MessagingGateway for such a scenario since we don't have a fixed defaultRequestChannel any more. In the sendFirstResponse() service method you just take a replyChannel header and send a newly created message manually. Technically it is exactly the same what you try to do with a mentioned #MessagingGateway.
For Java DSL variant I would go with a filter on the PublishSubscribeChannel to discard those messages which don't belong to the current flow. Anyway it is a different story.
Try to figure out how you can have a reply channel per flow when you configure particular configureAdapterCombination().