"Java Message Service" book by O'Reilly Media says:
use request/reply model in point-to-point messaging.
We can use message selectors in pub/sub messaging, so writing a request/reply model is as simple as writing a simple selector on reply topic:
publisher publishes a message with some unique property(such as UUID as correlationID)
subscriber responds for the message with the same UUID as correlationID
publisher(also subscriber of the reply topic) selects messages with the UUID sent.
Is this a wrong pattern?
Request/Reply messaging pattern is typically used for invoking a service hosted by service provider. Based on service request, a provider will reply with an appropriate response. So it's one-to-one. Here requestor and responder know each other.
In case of pub/sub, publisher and subscriber do not know each other. There could be a number of publisher publishing on a topic and there could be thousands of subscribers listening for that topic. So after receiving publication, if a subscriber replies to request using a topic, then that publication could go to a number of subscribers. Such a thing might flood the network.
In my opinion Request/Reply model must be used in P2P messaging and not Pub/Sub.
Related
Context and Problem
We are building a notification system.
Publisher users can send messages to other online subscriber users
online subscriber users will receive the sent messages.
Publisher users and Subscriber users are on different instances and have no direct way to reach each other.
It's okay for subscribers to miss some notifications in rare scenarios (other methods of retrieving all notifications are provided)
Solution
- Publishing
Publisher user publishes a message into RabbitMQ.
Business logic is applied to the message in RabbitMQ consumer.
RabbitMQ consumer publishes the message to Redis event notification_[subscriber_id].
- Subscribing
Subscriber user connects to a WebSocket server.
WebSocket server has a connection to Redis and subscribes to Redis event notification_[subscriber_id] on user connection.
Upon receiving messages on notification_[subscriber_id] a message is sent to the user over WebSocket.
Question
Both publisher users and subscriber users can be any amount (infinite), from my research it seems Redis has no limit on the number of subscriptions (around 4billion if there's any), so
Is this "dynamic" way of creating subscriptions in Redis, scalable?
Yes,
you can scale horizontally in Redis Cluster mode, which will allow you to continue serving requests during the scaling process.
Also,
It will be smart to also design your application to cleanup subscriptions, as you seem to be planning to deal with millions/billions of subscriptions, so a good planning before implementation is important.
Let me start by describing the system. There are 2 applications, let's call them Client and Server. There are also 2 queues, request queue and reply queue. The Client publishes to the request queue, and the server listens for that request to process it. After the Server processes the message, it publishes it to the reply queue, which the Client is subscribed to. The Server application always publishes the reply to the predefined reply queue, not a queue that the Client application determines.
I cannot make updates to the Server application. I can only update the Client application. The queues are created and managed by the Server application.
I am trying to implement request/reply pattern from Client, such that the reply from the Server is synchronously returned. I am aware of the "sendAndReceive" approach with spring, and how it works with a temporary queue for reply purposes, and also with a fixed reply queue.
Spring AMQP - 3.1.9 Request/Reply Messaging
Here are the questions I have:
Can I utilize this approach with existing queues, which are managed and created by the Server application? If yes, please elaborate.
If my Client application is a scaled app (multiple instances of it are running at the same time), then how do I also implement it in such a way, that the wrong instance (one in which the request did not originate) does not read the reply from the queue?
Am I able to use the "Default" exchange to my advantage here, in addition to a routing key?
Thanks for your time and your responses.
Yes; simply use a Reply Listener Container wired into the RabbitTemplate.
IMPORTANT: the server must echo the correlationId message property set by the client, so that the reply can be correlated to the request in the client.
You can't. Unlike JMS, RabbitMQ has no notion of message selection; each consumer (in this case, reply container) needs its own queue. Otherwise, the instances will get random replies and it is possible (highly likely) that the reply will go to the wrong instance.
...it publishes it to the reply queue...
With RabbitMQ, publishers don't publish to queues, they publish to exchanges with a routing key. It is bad practice to tightly couple publishers to queues. If you can't change the server to publish the reply to an exchange, with a routing key that contains something from the request message (or use the replyTo property), you are out of luck.
Using the default exchange encourages the bad practice I mentioned in 2 (tightly coupling producers to queues). So, no, it doesn't help.
EDIT
If there's something in the reply that allows you to correlate it to a request; one possibility would be to add a delegating consumer on the server's reply queue. Receive the reply, perform the correlation, route the reply to the proper replyTo.
I'am trying to understand the topology of queues and exchanges MT creates in RabbitMQ.
I cannot get these two statements:
we generate an exchange for each queue so that we can do direct sends
to the queue. it is bound as a fanout exchange (is it about sending vs publishing?)
control queues are exclusive and auto-delete - they go away when you
go away and are not shared.
What for does MT need to send direct messages? Does this relate to control queues used by MT internally?
There is also no mentioning about dead letter queue, does that imply MT does not support one out of the box?
Oops, looked in a wrong place. It's here.
Michael Aldworth has described this in details in his excellent MassTransit Send vs. Publish blog post.
Essentially, for each MT endpoint, you get a exchange-queue pair, which is named after the queueName parameter value in the endpoint configuration. Topics, at the other hand, are RabbitMQ exchanges named after the message contract full type name.
Publish
You send a message to the topic, meaning to the message type exchange. MassTransit creates binding between the message type exchange and queue-name exchange on start up. In such way subscriptions work on the RabbitMQ levels. Publisher never knows who will be receiving the published message, if anyone at all.
Send
When sending, however, you need to specify the receiver address. By doing this you instruct MassTransit to deliver the message directly to the queue-name exchange. There is no binding between message-type exchange and queue-name exchange involved here. Therefore, the message will be delivered even if there is no consumer for this type of message at the target service. In such case the message will be moved to the deadletter queue (queue-name_Skipped).
One of our customers has a JMS based implementation in which there are queues for reading/writing messages. The JMS client needs to write to an outbound queue and the it will read the response from an inbound queue. The JMS client will be deployed across multiple sites and will talk to a single outbound queue for writing messages and will read from a inbound queue (one only) for the responses. Consider the scenario in which there are 100 unique outbound requests and then the consumer gets 100 different responses for the sent requests (assume the messages got delivered correctly). How do I ensure that the messages that the consumer is reading from the inbound queue is for the designated recipient? Do we have to write our own logic to map the request/response? or does JMS have any delivery mechanism based on connection id … etc so that message get delivered to correct requester. Thank you very much in advance, need your expert inputs to design the application correctly. The JMS provider I am using is Apache ActiveMQ.
Regards,
Sumeet C
It sounds like you need REQUEST/REPLY...
Request/Reply - Synchronous
A queue sender sends a REQUEST message, then in the same thread, receives a REPLY. The sending thread blocks until the receiver sends back a reply message, ensuring the reply is for the original request. It's a basic set up that uses temporary queues, REPLY_TO addressing, and JMSCorrelationID...
Apache ActiveMQ Request/Reply
EAI Patterns for JMS Request/Reply
Point-to-Point - Async
If the customer's JMS implementation provides distinct queues for sending requests and receiving replies, you can send messages asynchronously with a unique JMSCorrelationID, and provided the customer sends back a response with that same id, you can receive the response message in a different thread and correlate them with the original request based on the JMSCorrelationID. Technically-speaking, REQUEST/REPLY does the same thing except it blocks and uses temporary queues for sending response messages back to the requestor instead of explicitly-named queues.
I am trying to find a message bus provider that supports Durable Subscribers and allows me to replay, in order, based on the message timestamp, all messages for a given topic. Futhermore, I would like the message bus to reset each durable consumer's checkpoint when a message arrives late. E.g.
Client subscribes to topic 1 at 2009-12-22 12:00:00
Message 1 arrives, Timestamped 2009-12-22
Client receives Message 1
Client disconnects
Message 2 arrives, Timestamped 2009-12-21 18:00:00
Client connects
Client receives Message 2, then Message 1
I would strongly prefer an open source provider. Does anyone know of a message bus provider that supports this? I've tried to read the intro documentation for ActiveMQ, Mass Transit, etc but I have to admit that I am behind the curve on message bus terminology, so a lot of it went over my head.
AMQP (implemented by RabbitMQ, et al) lets you define durable queues and attach them to the same exchange. Each client that wants to receive messages first sets up its own durable queue, which will hold messages received from the exchange even while the client is disconnected.
The only limitation of this is that clients that have never connected, and which arrive on the scene unexpectedly, cannot belatedly setup a queue and request a dump of all previous messages. AMQP 1.0 might allow such universal persistence, but I don't know the new model that well, so I can't say for sure.
you may want to look at the spring integration project.
http://www.springsource.org/spring-integration