Using NATS.IO message queue; Is it OK to send messages in a request/reply manner (using nc.Request(...)) but subscribe in a queue manner (using nc.QueueSubscribe(...) and reply from there?
Yes, this is by design and allows you to easily scale the group of receivers.
Related
I'm new to ZeroMQ.
Today I am trying the pub/sub pattern with NetMQ (the ZMQ library for .NET). I noticed some strange behavior (at least, to me!).
I create a subscriber socket and subscribes to topic "12345".
The publisher then publishes messages with topic "1234567890".
The subscriber can receive the messages!
That means, the filter does not compare the whole topic string, but only checks if the published topic "starts with" the subscribed topic.
To confirm that, I changed the subscribed topic to "2345". And the subscriber did not receive the messages.
If I change the publishing topic to "23456890" (while the subscribed topic is "2345"), then the messages come!
I'd like to know, is that the normal behavior of the topic filter in ZeroMQ (and NetMQ)?
Thank you very much!
" ...is that the normal behavior of the topic filter in ZeroMQ (and NetMQ)? "
Yes, this is documented property of ZeroMQ implementation of ultra-fast TOPIC-filtering. It works this way since ever ( v2.1.1+, v3.+, v4.+ and most probably it will remain so due to its ultimate performance and scaling envelopes ).
Also be informed, that a similar approach was taken by Martin SUSTRIK, the ZeroMQ co-father, in foundations of nanomsg and its ports and more recent breeds ( pynano, pynng et al ), so it could be called a de-facto industry best-practice, could it not?
Establish a new message filter. Newly created Subsriber sockets will
filtered out all incoming messages. Call this method to subscribe to
messages beginning with the given prefix. Multiple filters may be
attached to a single socket, in which case a message shall be accepted
if it matches at least one filter. Subscribing without any filters
shall subscribe to all incoming messages. const sub = new Subscriber()
// Listen to all messages beginning with 'foo'. sub.subscribe("foo")
// Listen to all incoming messages. sub.subscribe() Params: prefixes –
The prefixes of messages to subscribe to.
subscribe(...prefixes: Array<Buffer | string>): void;
I have implemented the example from the RabbitMQ website:
RabbitMQ Example
I have expanded it to have an application with a button to send a message.
Now I started two consumer on two different computers.
When I send the message the first message is sent to computer1, then the second message is sent to computer2, the thrid to computer1 and so on.
Why is this, and how can I change the behavior to send each message to each consumer?
Why is this
As noted by Yazan, messages are consumed from a single queue in a round-robin manner. The behavior your are seeing is by design, making it easy to scale up the number of consumers for a given queue.
how can I change the behavior to send each message to each consumer?
To have each consumer receive the same message, you need to create a queue for each consumer and deliver the same message to each queue.
The easiest way to do this is to use a fanout exchange. This will send every message to every queue that is bound to the exchange, completely ignoring the routing key.
If you need more control over the routing, you can use a topic or direct exchange and manage the routing keys.
Whatever type of exchange you choose, though, you will need to have a queue per consumer and have each message routed to each queue.
you can't it's controlled by the server check Round-robin dispatching section
It decides which consumer turn is. i'm not sure if there is a set of algorithms you can pick from, but at the end server will control this (i think round robin algorithm is default)
unless you want to use routing keys and exchanges
I would see this more as a design question. Ideally, producers should create the exchanges and the consumers create the queues and each consumer can create its own queue and hook it up to an exchange. This makes sure every consumer gets its message with its private queue.
What youre doing is essentially 'worker queues' model which is used to distribute tasks among worker nodes. Since each task needs to be performed only once, the message is sent to only one node. If you want to send a message to all the nodes, you need a different model called 'pub-sub' where each message is broadcasted to all the subscribers. The following link shows a simple pub-sub tutorial
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
I am trying to implement an application(Java) which will subscribe to different message types (XMLs) from other different applications via TIBCO EMS. Each of these message types will have a specific purpose. I am of the opinion that I should have multiple queues with multiple subscribers in my application, however, the TIBCO guy is adamant that there should be only one queue where all of these messages will be published and I will have one subscriber and the subscriber then should have logic to different tasks based on the XML received.
Which approach is better? One with multiple queues and subscribers OR the one queue and one subscriber? Please let me know reasons for the choice.
Thanks!
-Naveen
In general, if the same application is reading all the messages, it is much cleaner for that application to have a single input queue instead of multiple input queues. With multiple then the application will need to have logic to know which order to process the queues and so on. With one input queue, the messaging system can deal with the order of the messages - whether FIFO or by priority etc, and the application can just read the next message and process it.
Use unique message header for each type of xml while sending the message. And use message selectors / filters while receiving the same, so that it can be routed / delegated to the respective handler based on the header value. This way, you will be able to handle different type of xml messages by single queue as well.
if I am sending a message onto a multicast topic using:
TibrvMsg replyMessage = TibRvdTransport.sendRequest(message,timeout)
and there are two subscribers, which one actually sends the replyMessage, and what happens to the other replyMessage ?
I can only guess the fastest one that that answers. But I cannot see this documented anywhere.
Since your components are decoupled, they are unaware of each other. Rendezvous is pub-sub, which means that all subscribers receive all messages published to subjects that they have subscribed to. Furthermore, Rendezvous uses a peer-to-peer messaging approach vis-a-vis a centralized message forwarding approach. Therefore both components will receive the message and both components will reply.
If this is not the desired behavior, with Rendezvous you can use a distributed queue (RVDQ). With that approach a "scheduler" assigns work to workers, ensuring that messages get processed only once.
What can be the best way to aggregate messages from many different sources (actually queues/topics) into a single queue/topic and then consume it. I am trying to design an application to receive messages from different topics in JMS using weblogic.
You could write your own "aggregator" as a stand-alone Java application:
For each queue/topic have a reader in its own thread.
Each reader sends its received message again on a "aggregate queue".
Have another thread to listen on the "aggregate queue".
As a variation, you could use a JVM Queue (like java.util.concurrent.ArrayBlockingQueue) as the "aggregate queue". This is faster, does not require another MQ queue, does not need network bandwidth, but it's not persistent.
Another idea is to use a "Message driven bean (MDB)" for each incoming queue/topic:
Again, each of these MDBs just reads the message and resends it to the "aggregate queue".
Have another MDB listening on the "aggregate queue".
A few suggestions on quality requirements. I belive you have to consider them.
They will be highly relate with your technical solution.
is that message loss acceptable?
client ack could be considered.
e.g. A memory queue sit in middle, e.g. incoming queue1...n -> ArrayBlockingQueue in memory -> outgoing queue. The data in the ArrayBlockingQueue , will lost when app crash.
is that message duplicate acceptable for the single outgoing queue?
I would suggest yes.
Set applicable level PossibleDuplicateFlag to make the client aware of that.
how fast the incoming messages per second on the diff incoming queue?
one queue session has only a uniqe thread. Performance has to be considered in advance.