I am creating a hosted system where multiple customers can send messages. I am receiving thoses messages on a JMS queue.
Now, all processing is done in a similar way and I want my process to poll all incoming queues for messages and handle them. Is there a way in WSO2 ESB to subscribe to multiple queues?
If not possible, the workaround would be to create a seperate listener process for each queue and have this post the message to a central processing queue. But that seems to be a less clean solution (and I think it will scale worse than listening to multiple queues).
Any ideas on this?
If changes to activeMQ server is possible ie. if OP is able to influence the configuration to the server, something like ActiveMQ diverts could do the trick.
<divert name="prices-divert">
<address>jms.queue.ABC</address>
<forwarding-address>jms.queue.theone</forwarding-address>
<exclusive>true</exclusive>
</divert>
<divert name="prices-divert">
<address>jms.queue.xyz</address>
<forwarding-address>jms.queue.theone</forwarding-address>
<exclusive>true</exclusive>
</divert>
Basically, multiple diverts that converge the messages from multiple queues to the single queue. This method has advantage over the reading and writing to single queue-as mentioned by the OP and would in my view scale well as it is inbuilt feature.
You can define a sequence with all the required logic in it and then call it from multiple proxy services (each listening to a specific queue). Otherwise you can try something similar to this sample.
Related
We have a requirement to copy messages from one ActiveMQ broker to another. Here the message has to just copy and the message should exist in both broker.
I can think of a custom application that subscribes to a certain destination and read that message and re-post the messages to the destination in multiple brokers.
I do not have access to make changes in the Broker so I couldn't think of Network of Brokers option.
Is there any best practice or tools available to copy A-MQ messages from one broker to another?
Without having access to the target broker, as far as I know and I have read, I believe there is not shortcut to avoid the custom application that re-post those messages.
However, depending on the messages you want to re-post, there might be some functionalities offered by ActiveMQ that could facilitate your implementation (but they would not be for free, regarding the computational costs).
For example, in the case you want to copy ALL the messages sent through that broker to the other, then you might consider using Mirrored Queues, with a specific prefix (e.g. "copy"), that would allow you to just have a single consumer using a wildcard after that prefix (e.g. "copy.>"). That consumer would get ALL the messages sent to the broker, and it would simplify your implementation since you would just have to care about that single consumer and re-post from it. However this has costs, since as it is described in the documentation, enabling the mirrored queues will make a duplicate of each queue/topic in the system, and will post each message twice. You need to consider if this is an important inconvenient in your case, depending on the amount of messages and the available memory that your broker disposes.
In case you just wanted to copy SOME of the messages and not all, then I believe the most elegant way to handle it is by creating an abstraction of your Consumer class (or specific implementation), and use that special implementation for those queues you want to re-post. That class would be responsible of re-posting the messages to the other broker, in a way that would be transparent from the other Consumer class when using it.
I have talked above about consumers, but the same concept could apply to topics and subscribers. Hope these ideas help :)
I have an environment where I have only one app server. I have some messages that take awhile to service (like 10 seconds or so) and I'd like to increase throughput by configuring multiple instances of my consumer application running code to process these messages. I've read about the "competing consumer" pattern and gather that this should be avoided when using MassTransit. According to the MassTransit docs here, each receive endpoint should have a unique queue name. I'm struggling to understand how to map this recommendation to my environment. Is it possible to have N instances of consumers running that each receive the same message, but only one of the instances will actually act on it? In other words, can we implement the "competing consumer" pattern but across multiple queues instead of one?
Or am I looking at this wrong? Do I really need to look into the "Send" method as opposed to "Publish"? The downside with "Send" is that it requires the sender to have direct knowledge of the existence of an endpoint, and I want to be dynamic with the number of consumers/endpoints I have. Is there anything built in to MassTransit that could help with the keeping track of how many consumer instances/queues/endpoints there are that can service a particular message type?
Thanks,
Andy
so the "avoid competing consumers" guidance was from when MSMQ was the primary transport. MSMQ would fall over if multiple threads where reading from the queue.
If you are using RabbitMQ, then competing consumers work brilliantly. Competing consumers is the right answer. Each competing consume will use the same receive from endpoint.
In JMS there are Queues and Topics. As I understand it so far queues are best used for producer/consumer scenarios, where as topics can be used for publish/subscribe. However in my scenario I need a way to combine both approaches and create a producer-consumer-observer architecture.
Particularly I have producers which write to some queues and workers, which read from these queues and process the messages in those queues, then write it to a different queue (or topic). Whenever a worker has done a job my GUI should be notified and update its representation of the current system state. Since workers and GUI are different processes I cannot apply a simple observer pattern or notify the GUI directly.
What is the best way to realize this using a combination of queues and/or topics? The GUI should always be notified, but it should never consume anything from a queue?
I would like to solve this with JMS directly and not use any additional technology such as RMI to implement the observer part.
To give a more concrete example:
I have a queue with packages (PACKAGEQUEUE), produced by machine (PackageProducer)
I have a worker which takes a package from the PACKAGEQUEUE adds an address and then writes it to a MAILQUEUE (AddressWorker)
Another worker processes the MAILQUEUE and sends the packages out by mail (MailWorker).
After step 2. when a message is written to the MAILQUEUE, I want to notify the GUI and update the status of the package. Of course the GUI should not consume the messages in the MAILQUEUE, only the MailWorker must consume them.
You can use a combination of queue and topic for your solution.
Your GUI application can subscribe to a topic, say MAILQUEUE_NOTIFICATION. Every time (i.e at step 2) PackageProducer writes message to MAILQUEUE, a copy of that message should be published to MAILQUEUE_NOTIFICATION topic. Since the GUI application has subscribed to the topic, it will get that publication containing information on status of the package. GUI can be updated with the contents of that publication.
HTH
I’m writing a server/client game, a typical scenario looks like this: one client (clientA) send a message to the server, there is a MessageDrivenBean in server to handle such messages. After the MDB finished its job, it sends the result message back to another client (clientB).
In my opinion I only need two queues for such communication, one for input the other for output. Creating new queue for each connection is not a good idea, right?
The Input queue is relative clear, if more clients are sending message at the same time, the messages are just waiting in the queue, while there are more MDB instances in server, that should not a big performance issue.
But on the other side I am not quite clear about the output queue, should I use a topic instead of a queue? Every client is listening the output queue, one of them gets the new message and checks the property to determine if the message is to it, if not, it rollback the transaction, the message goes back to queue and be ready for other client … It should work but must be very slow. If I use topic instead, every client gets a copy of the message, if it’s not to it, just ignores the message. It should be better, right?
I’m new about message system. Is there any suggestion about my implementation? Thanks!
To begin with, choosing JMS as a gaming platform is, well, unusual — businesses use JMS brokers for delivery reliability and transaction support. Do you really need this heavy lifiting in a game? Shouldn't you resort to your own HTTP-based protocol, for example?
That said, two queues are a standard pattern for point-to-point communication. Creating a queue for a new connection is definitely not OK — message-driven beans are attached to queues at deployment time, so you won't be able to respond to queue creation events. Besides, queues are not meant to be created and destroyed in short cycles, they're rather designed to be long-living entities. If you need to deliver a message to one precise client, have the client listen on the server response queue with a message selector set to filter only the messages intended for this client (see javax.jms.Message API).
With topics it's exactly as you noted — each connected client will get a copy of the message — so again, it's not a good pattern to send to n clients a message that has to be discarded by n-1 clients.
MaDa;
You could stick one output queue (or topic) and simply tag the message with a header that identifies the intended client. Then, clients can listen on the queue/topic using a selector. Hopefully your JMS implementation has efficient server-side listener evaluation.
I need some help with topics and selectors.
I have a scenario with a topic having multiple durable subscribers (each with a selector)
not all messages going into the topic aren't read by the consumers - because of unmatching selectors.
This is correct behavior.
However the problem occurs when the unmatched messages reach a certain quantity threshold, because at that point no other messages are being delivered to the consumers
activemq tries to dispatch those old unmatchable messages, but since there is no consumer for them everything is stuck
can anybody help with this?
My setup is ActiveMq 5.5
is there some configuration option, or is it just a flawed design?
I'd say this is a flawed design given there are better alternatives and perhaps a bug in ActiveMQ.
First question: is your producer publishing to this topic setting the JMSExpiration header on those messages?
If yes, the first thing I'd do is create a Jira issue detailing the scenario you described above, because it does seem incorrect that ActiveMQ will continue hold on to and continue send messages for which no selectors apply.
As for flawed design, the minute you hear yourself saying "I need durable subscribers" and you are using ActiveMQ, you should immediately turn to using Virtual Destinations instead. Virtual Destinations have the advantages of topics in that a producer can send a message to a destination and have that message propagated to N number of other destinations for consumption, but doesn't have the disadvantages that come with having durable subscribers on a topic. Read more about Virtual Destinations here.
This is related to the way that ActiveMQ handles sparse selectors. The current implementation doesn't page into the store to look for message matching sparse selectors so you need to make some configuration changes to try and work around this. You can set the maxBrowsePageSize in the configured destination policy, the default is 400. See this page.
Virtual destinations in ActiveMQ is probably the better choice, it almost always is when thinking about using durable subscribers. You could however add some message expiration to you messages and configure the policy to expire messages on inactive durable subscribers if you use a SNAPSHOT version of ActiveMQ 5.6.
It does seem like a bug, (or dashedly inconvenient at the least) but there is a work around using Virtual Destinations.
Quote:
Note that making a topic virtual does add a small CPU overhead when
sending messages to the topic but it is fairly small. From version
5.4, dispatch from virtual topics to subscription queues can be
selectorAware such that only messages that match one of the existing
subscribers are actually dispatched. Using this option prevents the
build up of unmatched messages when selectors are used by exclusive
consumers.