We have a requirement to copy messages from one ActiveMQ broker to another. Here the message has to just copy and the message should exist in both broker.
I can think of a custom application that subscribes to a certain destination and read that message and re-post the messages to the destination in multiple brokers.
I do not have access to make changes in the Broker so I couldn't think of Network of Brokers option.
Is there any best practice or tools available to copy A-MQ messages from one broker to another?
Without having access to the target broker, as far as I know and I have read, I believe there is not shortcut to avoid the custom application that re-post those messages.
However, depending on the messages you want to re-post, there might be some functionalities offered by ActiveMQ that could facilitate your implementation (but they would not be for free, regarding the computational costs).
For example, in the case you want to copy ALL the messages sent through that broker to the other, then you might consider using Mirrored Queues, with a specific prefix (e.g. "copy"), that would allow you to just have a single consumer using a wildcard after that prefix (e.g. "copy.>"). That consumer would get ALL the messages sent to the broker, and it would simplify your implementation since you would just have to care about that single consumer and re-post from it. However this has costs, since as it is described in the documentation, enabling the mirrored queues will make a duplicate of each queue/topic in the system, and will post each message twice. You need to consider if this is an important inconvenient in your case, depending on the amount of messages and the available memory that your broker disposes.
In case you just wanted to copy SOME of the messages and not all, then I believe the most elegant way to handle it is by creating an abstraction of your Consumer class (or specific implementation), and use that special implementation for those queues you want to re-post. That class would be responsible of re-posting the messages to the other broker, in a way that would be transparent from the other Consumer class when using it.
I have talked above about consumers, but the same concept could apply to topics and subscribers. Hope these ideas help :)
Related
We are migrating from Tibco to start using ActiveMQ Artemis. There are several ack settings that are available on Tibco, but we haven't found anything that's simply similar to this in Artemis. We are using the amqpnetlite .NET library to interface with Artemis, and as part of our code using DistributionMode to either move or copy based on the boolean value we are assigning to a configuration flag that we are calling as UseAutoAcknowledge. I haven't found much documentation about DistributionMode but for one that isn't very clear here - http://docs.oasis-open.org/amqp/core/v1.0/amqp-core-messaging-v1.0.html.
My question is if DistributionMode is set to move - does Artemis send an acknowledgement to the client and doesn't when it is set to copy?
I can't talk to Tibco but I can try to explain AMQP DistributionMode. Essentially the DistributionMode is a setting as to the behaviour of the receiver - a receiver with a move mode is expecting the messages to be sent only to it, not to other receivers - this is the normal behaviour of a consumer on a queue. A receiver with a copy mode is expecting other receivers to also receive the message (like a queue browser, or - sort of - like a subscriber to a topic). In a traditional Client-Broker topology, the DistributionMode is only really interesting when receiving messages from the Broker, and is unlikely to have effect when sending messages to the Broker.
Acknowledgement is separate from the DistributionMode. AMQP has the concept of Disposition which is similar to but not the same as Acknowledgement. Disposition is ultimately the action that the sender will apply at the completion of the message transfer (and so interacts with DistributionMode for messages sent by the Broker). Conceptually for each message transfer a Broker might decide that the transfer has completed successfully; that it has failed - but in a way that retrying might succeed; that it has failed in a way that will not succeed on retry; or some other more subtle outcome. Here the behaviour at the Broker is probably different depending upon whether the DistributionMode was move or copy (the specification left this vague to allow flexibility in implementations). If the receiver is asking for messages to be moved, and it declares that the transfer was unsuccessful, a broker is likely to make that message available for all competing consumers. If the receiver was asking for copy, then it never held an exclusive lock on the message, and so the choice is only whether to retry sending the copy to that same consumer.
Perhaps the simplest thing here is if you can describe the behaviour that you desire, and experts on Apache Artemis can weigh in on if/how that can be achieved.
I'm confused when it comes to JMS Queue/Topic. What I want is messages should go to every subscriber and I want subscribers to receive messages from inactive time when they become active. However, I don't have control over whether or not subscribers have durable subscription. Is there a way to set up a persistent Queue, and set it up so that every subscriber will receive same message? And how to set this up using spring config
Thanks much.
This is mostly a question where the design of your system affects the outcome.
You could use UI tooling to create durable subscriptions for the clients that need to but that is cumbersome and error prone. You could use something like camel or other configuration on the target broker to fanout messages from an incoming Queue to outgoing Queues that map to the consumer subscriptions.
It all depends mostly on the requirements and your overall design so a real answer is beyond the scope of a SO answer without you doing some more legwork to narrow the scope a bit. JMS itself does not define any answer for this so it will come down a bit to the broker you've chosen and possibly other third party tooling that you might pick to do what you need.
I'm looking at swapping out ActiveMQ with RabbitMQ for a few reasons. I currently have multiple services which are each capable of publishing events (and they publish those events to a specific VirtualTopic in AMQ). Each of the services is also capable of consuming messages from the other services. Consumers are set up such that they subscribe as a consumer to a queue on the VirtualTopic.
This buys me the ability to fan messages out to multiple queues (topic-like functionality) while keeping the benefits of queues (load balancing and persistence).
It seems like this is roughly equivalent to RabbitMQ's fanout exchange. However, the part that I found very useful in ActiveMQ is that the producer doesn't need to have any knowledge of the consumers. It simply publishes to the virtual topic. It seems that in RabbitMQ, when the exchange is created, I need a definitive of queues to publish that message to.
tl;dr
Is there any routing scheme in RabbitMQ that is equivalent to ActiveMQ's Virtual Topic, such that I can produce messages to a topic that are distributed to any queue that has been created off of that Virtual Topic, without requiring a hard-coded routing scheme somewhere in RMQ?
I realized after posting this question that it is pretty trivial to do this (not sure why I never thought of it before).
I was looking at it from the wrong direction, wondering how I could automatically have the publisher configure queues for the recipients - which isn't the right way to approach this question.
Instead, I have the subscribers, when they start up, bind themselves to the exchange that the publisher users, which provides in the inversion of control I'm looking for (publishers need not know anything about their consumers).
I’m writing a server/client game, a typical scenario looks like this: one client (clientA) send a message to the server, there is a MessageDrivenBean in server to handle such messages. After the MDB finished its job, it sends the result message back to another client (clientB).
In my opinion I only need two queues for such communication, one for input the other for output. Creating new queue for each connection is not a good idea, right?
The Input queue is relative clear, if more clients are sending message at the same time, the messages are just waiting in the queue, while there are more MDB instances in server, that should not a big performance issue.
But on the other side I am not quite clear about the output queue, should I use a topic instead of a queue? Every client is listening the output queue, one of them gets the new message and checks the property to determine if the message is to it, if not, it rollback the transaction, the message goes back to queue and be ready for other client … It should work but must be very slow. If I use topic instead, every client gets a copy of the message, if it’s not to it, just ignores the message. It should be better, right?
I’m new about message system. Is there any suggestion about my implementation? Thanks!
To begin with, choosing JMS as a gaming platform is, well, unusual — businesses use JMS brokers for delivery reliability and transaction support. Do you really need this heavy lifiting in a game? Shouldn't you resort to your own HTTP-based protocol, for example?
That said, two queues are a standard pattern for point-to-point communication. Creating a queue for a new connection is definitely not OK — message-driven beans are attached to queues at deployment time, so you won't be able to respond to queue creation events. Besides, queues are not meant to be created and destroyed in short cycles, they're rather designed to be long-living entities. If you need to deliver a message to one precise client, have the client listen on the server response queue with a message selector set to filter only the messages intended for this client (see javax.jms.Message API).
With topics it's exactly as you noted — each connected client will get a copy of the message — so again, it's not a good pattern to send to n clients a message that has to be discarded by n-1 clients.
MaDa;
You could stick one output queue (or topic) and simply tag the message with a header that identifies the intended client. Then, clients can listen on the queue/topic using a selector. Hopefully your JMS implementation has efficient server-side listener evaluation.
I need some help with topics and selectors.
I have a scenario with a topic having multiple durable subscribers (each with a selector)
not all messages going into the topic aren't read by the consumers - because of unmatching selectors.
This is correct behavior.
However the problem occurs when the unmatched messages reach a certain quantity threshold, because at that point no other messages are being delivered to the consumers
activemq tries to dispatch those old unmatchable messages, but since there is no consumer for them everything is stuck
can anybody help with this?
My setup is ActiveMq 5.5
is there some configuration option, or is it just a flawed design?
I'd say this is a flawed design given there are better alternatives and perhaps a bug in ActiveMQ.
First question: is your producer publishing to this topic setting the JMSExpiration header on those messages?
If yes, the first thing I'd do is create a Jira issue detailing the scenario you described above, because it does seem incorrect that ActiveMQ will continue hold on to and continue send messages for which no selectors apply.
As for flawed design, the minute you hear yourself saying "I need durable subscribers" and you are using ActiveMQ, you should immediately turn to using Virtual Destinations instead. Virtual Destinations have the advantages of topics in that a producer can send a message to a destination and have that message propagated to N number of other destinations for consumption, but doesn't have the disadvantages that come with having durable subscribers on a topic. Read more about Virtual Destinations here.
This is related to the way that ActiveMQ handles sparse selectors. The current implementation doesn't page into the store to look for message matching sparse selectors so you need to make some configuration changes to try and work around this. You can set the maxBrowsePageSize in the configured destination policy, the default is 400. See this page.
Virtual destinations in ActiveMQ is probably the better choice, it almost always is when thinking about using durable subscribers. You could however add some message expiration to you messages and configure the policy to expire messages on inactive durable subscribers if you use a SNAPSHOT version of ActiveMQ 5.6.
It does seem like a bug, (or dashedly inconvenient at the least) but there is a work around using Virtual Destinations.
Quote:
Note that making a topic virtual does add a small CPU overhead when
sending messages to the topic but it is fairly small. From version
5.4, dispatch from virtual topics to subscription queues can be
selectorAware such that only messages that match one of the existing
subscribers are actually dispatched. Using this option prevents the
build up of unmatched messages when selectors are used by exclusive
consumers.