I'm trying to use the InMemoryBus transport in unit tests with 2 bus instances (same process). One of the bus instances should publish an event and the other one consume it.
What I need, I think, is the Shared Transport feature documented here http://masstransit-project.com/MassTransit/usage/transports.html but it seems no longer available in version 5.2.1. Is there another way to achieve the same result?
Multiple bus instances with a shared in-memory transport are no longer supported. You can use a single bus instance with multiple receive endpoints if you need to test multiple consumers in the same process.
I'm guessing that you're upgrading from a much older version of MT, since that ability was removed quite some time ago.
Related
We have multiple application environments (development, QA, UAT, etc) that need to connect to fewer provider environments through MQ. For example, the provider only has one test (we'll call it TEST1) environment to which all of the client application environments need to interact. It is imperative that each client environment only receives MQ responses to the messages sent by that respective environment. This is a high volume scenario so correlating message IDs has been ruled out.
Right now TEST1 has a queue set up and is functional, but if one of the client app's environments wants to use it the others have to be shut off so that messaging doesn't overlap.
Does MQ support a model having multiple clients connect to a single queue while preserving the client-specific messaging? If so, where is that controlled (i.e. the channel, queue manager, etc)? If not, is the only solution to set up additional queues for each corresponding client?
Over the many years I have worked with IBM MQ, I have gone back and forth on this issue. I've come to the conclusion that sharing a queue just makes life more difficult. Queues should be handed out like candy on Halloween. If an application team says that they have 10 components to their application then the MQAdmin should give them 10 queues. To the queue manager or server or CPU or hard disk, there is no difference in resource usage.
Also, use an MQ naming standard that makes sense and is easy to apply security to. i.e. for HR (Human Resource) department
HR.PAYROLL.SALARY
HR.PAYROLL.DEDUCTIONS
HR.PAYROLL.BENEFITS
HR.EMPLOYEE.DETAILS
HR.EMPLOYEE.REVIEWS
etc...
You could use a selector such as MQGET(where applname="myapp") or based on a specific user-defined property assuming the sender populates such a property but that's likely to be worse performance than any retrieval by msgid or correlid. Though you've not given any information to demonstrate that get-by-correlid is actually problematic.
And of course any difference between a test and production environment - whether it involves code or configuration - is going to be very risky.
You would not normally share a single destination queue between multiple different application types - multiple queues is far more standard.
One question on vert.x event bus scalability. I am planning to use vert.x in smart device (small form facor) application and a remote management application. Initial estimate is that there will be close to 100K smart devices and 3/4 servers hosting management application. In this case, can you please advise using event bus between the smart device and web application (in cluster mode). My primary requirement of using event bus is to send dynamic notifications originated from device to the management servers and take corrective steps in case of system failure.
I posted another query recently and one of the users pointed me that internally vert.x uses the netsockets for event bus backed by hazelcast for cluster mode discovery. If that is the case, my assumption is that the scalability will be limited by the number of sockets that can be handled by the management server. Is this right ?
Also appreciate if anyone can point me to any benchmark test done on the vert.x eventbus in terms of msg processing performance.
My primary requirement of using event bus is to send dynamic notifications originated from device to the management servers and take corrective steps in case of system failure.
No, use regular HTTP requests for this. EventBus, and indeed every concurrent two-way networking model, is fundamentally unsuitable for this use case. Absolutely do not use Hazelcast on the clients; using a SockJS EventBus bridge is possible but so error-prone that you will certainly waste more time doing that correctly than writing a simple HTTP endpoint for this heartbeat behaviour.
my assumption is that the scalability will be limited by the number of sockets that can be handled by the management server. Is this right ?
No. Your scalability will be limited by however you'll be persisting the data you receive from the device. Hazelcast's maps are fine for this (accessed via vertx.sharedData()), but it really depends if you 100% understand what you want.
Does any body have some info, links, pointer on how is cross process Eventbus communication is occurring. Per documentation I am concluding that multiple Vert.x (thus separate JVM processes) could be clustered on and communicate via Eventbus. However, there are little to none documentation on how to achieve it.
Looking into DOCs, I can see that publish/registerHandler methods take address as a String what works within a process, but I can not wrap my head around on how it works cross processes and how to register and publish to address, does it work over HTTP , TCP ? From API perspective do I need to pass port and process signature ?
Cross process communication happens via the EventBus. Multiple vertx instances can be started up and clustered to allow separate instances on the same or other machines to communicate. The low level clustering is handled by Hazelcast.The configuration is handled by the cluster.xml file in the conf folder of your vertx install. You can learn more about the format of the file by looking at the Hazelcast Docs. It is transparent to your handers and works over TCP.
You can test it by running two or more instances on your local machine once they are started with the -cluster flag. Look at the example being run, and the config changes required in How to use eventbus messaging in vertx?
I am really curious about this topic.
I will create a communication mechanism for internal systems and may also need connection to some external clients too. The internal modules are also distributed systems.
I need to create a ESB between that modules. The system should provide high performence over millions of subscribers.
publish subscribe or p2p communications are both needed,
When I first started to thinking about that implementation , I was planed to make a REST api on front and the REST api will communicate with a JMS bus .The JMS bus has an ability to provide communication between internal systems.
Unfortunately as per my investigation, using JMS can be caused so musch critical problems : performance,scalability... and looks like JMS is needless, I can create some adapters over internal modules and both can communicate with REST services.
Does anyone have any idea why should I use JMS for internal communication ?
Both REST and JMS/MQ enable communicate between remote systems (and local). You can get help based on the scenarios below:
Some Reasons for using JMS in your case:
If your producer is spitting messages at a very high rate than the consumer then the persistent messaging will help. This may also mean you are fine with the transaction/message to be processed later.
All systems are not up all the time.
You need a publish subscribe mechanism (topic).
Messages are not critical and discard old messages when load is high.
Reasons for using REST API (without any jms connected):
1. You want an immediate response that transaction is completed. Example, hotel booking etc.
2. All systems should be up all the time for processing to complete.
You would want to use JMS (or enterprise messaging) when don't have to rely on all the systems being available. So if one of your internal systems was down for some reason, then a REST api interface would fail when communicating to that system, but a JMS interface would not as you are communicating to the MOM.
For some MOM you don't have to just communicate using JMS, so you can have different runtimes communicate to the MOM.
We use IBM MQ6.1. And use IBM API to send Messages to Queue.
As we need to send huge number of messages, was wondering if we can use multithreading and send messages to the same Queue.
Any suggestions?
Absolutely. You can have as many threads as memory and CPU will allow. Each thread requires it's own connection handle if you want to see a performance improvement. If you use a single connection handle, all GET and PUT activity from multiple threads is synchronized through the connection handle. If you use many connection handles, the GET and PUT activity can occur in parallel. Please see Multithreaded programs in the WMQ Using Java manual for additional details.
Just make sure that OS kernel settings like MAXUPROCS are set to allow WMQ to run sufficient threads to receive the connections and that WMQ settings like MAXHANDS and MAXCHANNELS are bumped up to accommodate the load. Keep in mind that units of work are thread-scoped so each thread is independent of the others for syncpoint - for example, when you issue COMMIT inside a thread it only commits the messages put by that particular thread assuming it has a dedicated connection. The queues have an attribute for DEFSOPT (default share options) but this relates to how many input handles can be active. Even if you open the queue for exclusive input, you can have many threads writing to it.
Also, v6.x of WMQ will be end-of-life as of September 2011. Start planning now to get to v7 to remain on a supported version. If you are using WMQ Client connections, upgrade the client as well as the QMgr. The v7 Client works fine with a v6 QMgr and can be upgraded independently. Of course it supports only the v6 functionality when used with a v6.x QMgr. You can download the v7 client as SupportPac MQC7.