I'm working on updating an existing Mule configuration and the task is to enhance it to route messages to different endpoints depending on some properties of the messages, therefore it would be nice to have some pros and cons on the two options I have at hand:
Add properties on the message, using the "message-properties-transformer" transformer which is later used by a "filtering-router" to single out the message and put it on the correct endpoint. This option allows me to use a single queue for all destinations.
Create one queue for each destination and thus instead of adding some property for later routing, I just put on on the right queue at once. I.e. this option would mean one queue per destination.
Any feedback would be welcome. Is there any "best practices" with regards to this?
I've had a great deal of success with using your first approach with a filtering-router. It reduces cohesion between your message producers and consumers. It forms a valuable abstraction, so any service can blindly drop messages within the generic "outbox".
We've come to depend on mule for filtering and routing messages so much so that we have a dedicated cluster of hardware to do only this. Using mule I was able to get far greater performance and not have to maintain connections to all queues.
The down side will be having to very carefully maintain your messaging object version globally, and having to keep a set of transformers on hands to accept and convert from different versions if you plan to upgrade only a portion of your infrastructure.
thanks, matt
Related
I'm using RabbitMQ in my pet project (Spring Boot based). In #Configuration I declare beans like Queue,Binding,DirectExchange. So, when I run the application all these exchanges and bindings with queues are created automatically. I'm concerned about whether this is the correct way to configure these RabbitMQ-related "entities". Should I separate this into separate steps before application startup? For example, calling series of curl to the management HTTP API to create all needed queues (with exchanges and bindings) before application startup. What are the best practices for creating/configuring routing-related stuff?
The thing is there is no one way of using RabbitMQ. However there are a few questions I always ask myself before working with any broker. I'll apply them to your question here.
Let's question the approach:
In regards to creating the exchanges, bindings, queues etc:
Try to understand if the elements you are using are durable. If so, then you could create this within your code on startup & apply a simple health check. You also want to check if your RabbitMQ server persists data. If not, then you'll HAVE to create your queues, exchanges & bindings every time.
In regards to routing & binding queues with exchanges:
There are two major questions you need to consider
How much does latency matter?
If latency does matter, try to use direct exchanges as much as you can. The reason for this is simple. You're simply going from exchange to queue instead of having to route your message. Routing adds latency, never forget! If those few extra ms won't make the difference for you, then the following question needs to be kept in mind to understand how to define your exchanges, bindings & queues.
How will I use my broker?
Some people use their broker to simply pub/sub messages. This is a perfectly reasonable use case. In this case a fanout exchange would be the most viable option. If you're trying to minimize the amount of queues you're creating a topic exchange may be interesting as well.
However more important, are you using your broker exclusively for between-service communication or is your service going to be both the producer and consumer? Hell, is it a mix between the previous two cases? This boundary needs to be defined clearly. Else you'll run into a mess where you suddenly notice you're consuming messages that can actually be handled internally by another library or simply passing arguments to functions.
Example of how to apply it:
Case: we have a logging service & user service:
1. How will I use my broker:
between service communication.
2. Which messages do I need to get across?
CRUD operations
login/logout
3. How would my routing table look like (draft)?
Exchange Name
Exchange Type
Binding
Queue
user
topic
user.cmd.*
user:crud
user
topic
user.event.login
user:login
Above you can clearly see we can handle all CRUD operations using one simple queue. Is this the most efficient approach? It depends on your service. It may be better that user.cmd.create should go to user:created. This is another boundary that you'll need to define.
Something that also needs to be mentioned is that you should use your Queues & routing keys as pieces of information. Debugging a micro service can be hellish. So applying a general naming convention would be most appropriate. There is no one naming convention, so this depends on your use case once again.
Conclusion:
In general the practice that is best in regards to any broker is clearly defining the scope of your broker and its underlying elements. If not done properly, it does not matter if you do a health check or not on startup. It's easy to get lost in the complexity and interesting features rabbitMQ offers. Try to keep it as simple as possible at first and ask yourself: "is it going to cost me a lot of time to refactor/debug/fix later?" If the answer is yes go back to the first sentence in this paragraph.
Documentation:
AMQP concepts: https://www.rabbitmq.com/tutorials/amqp-concepts.html
Some general best practices: https://www.cloudamqp.com/blog/part2-rabbitmq-best-practice-for-high-performance.html
Routing: https://www.rabbitmq.com/tutorials/tutorial-four-python.html
Latency & throughput: https://blog.rabbitmq.com/posts/2012/05/some-queuing-theory-throughput-latency-and-bandwidth
We have multiple application environments (development, QA, UAT, etc) that need to connect to fewer provider environments through MQ. For example, the provider only has one test (we'll call it TEST1) environment to which all of the client application environments need to interact. It is imperative that each client environment only receives MQ responses to the messages sent by that respective environment. This is a high volume scenario so correlating message IDs has been ruled out.
Right now TEST1 has a queue set up and is functional, but if one of the client app's environments wants to use it the others have to be shut off so that messaging doesn't overlap.
Does MQ support a model having multiple clients connect to a single queue while preserving the client-specific messaging? If so, where is that controlled (i.e. the channel, queue manager, etc)? If not, is the only solution to set up additional queues for each corresponding client?
Over the many years I have worked with IBM MQ, I have gone back and forth on this issue. I've come to the conclusion that sharing a queue just makes life more difficult. Queues should be handed out like candy on Halloween. If an application team says that they have 10 components to their application then the MQAdmin should give them 10 queues. To the queue manager or server or CPU or hard disk, there is no difference in resource usage.
Also, use an MQ naming standard that makes sense and is easy to apply security to. i.e. for HR (Human Resource) department
HR.PAYROLL.SALARY
HR.PAYROLL.DEDUCTIONS
HR.PAYROLL.BENEFITS
HR.EMPLOYEE.DETAILS
HR.EMPLOYEE.REVIEWS
etc...
You could use a selector such as MQGET(where applname="myapp") or based on a specific user-defined property assuming the sender populates such a property but that's likely to be worse performance than any retrieval by msgid or correlid. Though you've not given any information to demonstrate that get-by-correlid is actually problematic.
And of course any difference between a test and production environment - whether it involves code or configuration - is going to be very risky.
You would not normally share a single destination queue between multiple different application types - multiple queues is far more standard.
We have a requirement to copy messages from one ActiveMQ broker to another. Here the message has to just copy and the message should exist in both broker.
I can think of a custom application that subscribes to a certain destination and read that message and re-post the messages to the destination in multiple brokers.
I do not have access to make changes in the Broker so I couldn't think of Network of Brokers option.
Is there any best practice or tools available to copy A-MQ messages from one broker to another?
Without having access to the target broker, as far as I know and I have read, I believe there is not shortcut to avoid the custom application that re-post those messages.
However, depending on the messages you want to re-post, there might be some functionalities offered by ActiveMQ that could facilitate your implementation (but they would not be for free, regarding the computational costs).
For example, in the case you want to copy ALL the messages sent through that broker to the other, then you might consider using Mirrored Queues, with a specific prefix (e.g. "copy"), that would allow you to just have a single consumer using a wildcard after that prefix (e.g. "copy.>"). That consumer would get ALL the messages sent to the broker, and it would simplify your implementation since you would just have to care about that single consumer and re-post from it. However this has costs, since as it is described in the documentation, enabling the mirrored queues will make a duplicate of each queue/topic in the system, and will post each message twice. You need to consider if this is an important inconvenient in your case, depending on the amount of messages and the available memory that your broker disposes.
In case you just wanted to copy SOME of the messages and not all, then I believe the most elegant way to handle it is by creating an abstraction of your Consumer class (or specific implementation), and use that special implementation for those queues you want to re-post. That class would be responsible of re-posting the messages to the other broker, in a way that would be transparent from the other Consumer class when using it.
I have talked above about consumers, but the same concept could apply to topics and subscribers. Hope these ideas help :)
I have to run two instances of the same application that read messages from 'queue-1' and write them back to another queue 'queue-2'.
I need my messages inside the two queues to be ordered by specific property (sequence number) which is initially added to every message by producer. As per documentation, inside queue-1 the order of messages will be preserved as messages are sent by a single producer. But because of having multiple consumers that read, process and send the processed messages to queue-2, the order of messages inside queue-2 might be lost.
So my task is to make sure that messages are delivered to queue-2 in the same order as they were read from queue-1. I have implemented re-sequencer pattern from Apache camel to re-order messages inside queue-2. The re-sequencer works fine but results to data transfer overhead as the camel routes run locally.
Thinking about doing it in a better way, I have three questions:
Does artemis inherently supports re-ordering of messages inside a
queue using a property such as sequence number.
Is it possible to run the routes inside the server? If yes, can you
give an example or give a link to the documentation?
Some artemis features such as divert (split) requires modifying
broker configuration (broker.xml file), is there a way to do them
programmatically and dynamically so I can decide when to start
diverting message? I know this can be accomplished by using camel,
but I want everything to be running in the server.
Does artemis inherently supports re-ordering of messages inside a queue using a property such as sequence number.
No. Camel is really the best solution here in my opinion.
Is it possible to run the routes inside the server? If yes, can you give an example or give a link to the documentation?
You should be able to do the same kind of thing in Artemis as in ActiveMQ 5.x using a web application with a Camel context. The 5.x doc is here.
Some artemis features such as divert (split) requires modifying broker configuration (broker.xml file), is there a way to do them programatically and dynamically so I can decide when to start diverting message?
You can use the Artemis management methods to create, modify, and delete diverts programmatically (or administratively) at runtime. However, these modifications will be volatile (i.e. they won't survive a broker restart).
Here is my use case: I am developing a trading application and i want to send incremental stock updates (bidQty etc) to active consumers instead of the whole quote and a snapshot update to a new consumer (to start with).
Now, is it possible to override any ActiveMQ's class (implementors of Topic) to achieve this behavior? Any clues on this would be helpful .
If the same is possible in any other openSource provider, please let me know.
This is NOT a case where you simply can change the implementation of topic. You should actually avoid changing the implementation of core ActiveMQ features to solve specific business requirements. Fixing bugs and adding core messaging features is another thing.
There are multiple ways to solve your use case with regular ActiveMQ features.
Separate Sync and Update channel
I would probably divide the "sync/snapshot" channel from the "incremental update" channel.
One way is to implement the "snapshot-sync" as JMS request/reply where the consumer asks the provider for a sync, then continues to rely on incremental updates pushed via the topic.
Advisory messages and Selectors
You can also implement it all using a single topic using a mix of AdvisoryMessages and JMS Selectors.
An idea (you can do this in many ways):
Introduce two message properties: MsgType and Receiver
Mark each incremental update with MsgType=inc
Mark each snapshot with some client id of the consumer, Receiver=.
Have the producer listen to advisory messages from ActiveMQ and and fire a snapshot/sync message marked with Receiver= and MsgType=snapshot when there is a new client subscribing the stock topic.
The client subscribes with a selector of something like
MsgType='inc' OR (MsgType='snapshot' AND Receiver=<me>)
This way you can trigger snapshot syncs with specific clients as well as incremental updates for all clients.
If you start think about the dynamics you already have, you can probably think of another ten or so solutions.
Retroactive Consumers
You might have some use of a Retroactive Consumer - the example actually shows a scenario similar to yours.