I wanna build a micro services system, and i'm choosing the message queue.
As the title, can I use Chonicle queue as a pub/sub system?
I read that Chronicle Queue can be used as a message passing between jvms on the same machine, but how can we achieve this on different machines?
If we use replication, on a gateway host machine we create a source queue and replicate it to sub service machine (sink), so we can pass message from gateway to sub service. How can we do the vice versa from sub service to the gateway?
To pass data from one machine to another you can use replication. This works the same whether you pass from a gateway to a service or from a service to a gateway.
NOTE: replication is a licensed feature for Chronicle Queue.
Related
I'm using MassTransit with Azure Service Bus as a transport. Some endpoints will live outside of our network, so I'd like to restrict the connection strings to those endpoint queues/topics while allowing the endpoints that are on our network to send to all of the other endpoints.
Is this possible? If I try to set a connectionstring like that, errors indicating the lack of permissions to a topic that I don't think I need it to access.
MassTransit requires Manage since it creates any topics/queues at startup.
If you are only sending messages to a specific queue, I have heard of some having success by ensuring the queue already exists and has the appropriate access for the credentials, but I don't know the details. In the one case I know of, they were using queue:name with GetSendEndpoint on IBus, and then calling Send.
this is what I want to achieve with IBM MQ:
I have a topic in place, and 2 different applications (A1,A2) subscribing the topic. each application has 2 instances, i.e. (A1-I1,A1-I2,A2-I1,A2-I2).
when a message (M1) is published to the topic, the message will be received by both applications A1 & A2. but within A1, only one instance (A1-I1 or A1-I2) can receive this message M1, the same goes to instances of A2.
is it possible with IBM MQ topics and JMS?
I have looked up shared subscription mentioned in the comment, but I find it (IBM MQ) has a limitation that the shared subscription can only be accessed by one JVM, i.e. two application instances can't access the the shared subscription at the same time, so I give up this purely pub/sub direction.
I managed to make it work by making the applications listen to the destination queue of subscription instead of directly subscription. I defined in the queue manager subscription SUB.A1 with destination queue of SUB.A1.QUEUE and subscription SUB.A2 with destination queue of SUB.A2.QUEUE. all instances of application A1 will listen to SUB.A1.QUEUE and A2 to SUB.A2.QUEUE. and everything works as expected.
The above setup is all applications connecting to the same queue manager. if there is a queue manager cluster of multiple QMs, just define the subscription destination queues as cluster queues and it will work as well
I'm building a system where client IoT devices will be making persistent websocket connections to a single instance of a microservice. We'll call it the "hardware gateway". End devices will be connecting to one of these service instances and may migrate between services at anytime (perhaps due to a reboot or network interruption).
Other services will be pushing notifications to these hardware clients via some hardware gateway instance. I need a way to route these requests to the specific instance that is maintaining a connection to a specific IoT device. At the moment, my solution is to maintain an external KV store where I can map an IoT device's UUID to a service instance, but that puts an extra dependency on all other services to know about this KV store. Not to mention the additional latency introduced by this query.
Maybe there's some reverse proxy that allows me to dynamically update its matching criteria? I've also looked into using a message broker like RabbitMQ, but it doesn't seem to support this use case.
There's a reasonable solution in JVM land for this: Akka.
The instances form an Akka cluster. When a device makes a websocket connection, an actor is spawned to handle the interactions over the websocket. The actor registers that it is the actor interacting with the device with a cluster sharded actor keyed by the device's ID (and likely periodically reregisters with the sharded actor). As instances are deployed, etc. the cluster rebalances. An important feature of this is that the service is stateful, but the instances deploy in a way that looks to the outside world like it's stateless: requests can go to any node.
For pushing notifications to the devices, the HTTP endpoint or message-bus consumer in the service looks up the cluster sharded actor which forwards the notification to the websocket actor (you'll want to think about whether you want at-least-once or at-most-once delivery, which will govern whether there's some portion of the cluster sharded actor which should be persistent).
I need to send messages to different Queue Managers (different QM name, host-port pairs).
What is the best way to handle this scenario? Do I need to create separate ConnectionFactory for each Queue Manager?
Use Case: Java service which is sending command message to distributed agents (FTE agents). However those agents can listen on different queues on different Queue Managers.
Yes you need to use different ConnectionFactory for each JMS server. You can reuse ConnectionFactory only for various Queues on one Message Broker.
I think you are better of connecting the different queue managers hosting the queues and accessing the remote queues via remote queue definitions created on the queue manager your JMS application is connecting to.
I guess these queue managers are already connected, if you are moving files between the FTE agents.
FTE has the concept of the Command Queue Manager. This node is designated as the one QMgr to which the application should connect when communicating with the FTE agents. If the network has been properly defined, connectivity between all FTE agents and the Command QMgr will be present.
The design is one of the ways in which FTE can enforce security. It is possible on the agent QMgrs for the channel from the command queue manager to be authorized very specifically to a subset of the queues and the messages monitored. The design provides a policy enforcement point. Since the FTE security model is weak to begin with, it is important to try to properly use the security controls that are provided.
This design also means that it is not necessary to manage application service account credentials across all the agent QMgrs.
So, yes, use the Command QMgr for its intended purpose. Have one and only one connection factory and properly design and secure the FTE network using the Command QMgr and its channels as a policy enforcement point.
Currently using WebLogic and Distributed Queues. And I know from the documentation that Distributed Queues allow you to retrieve a connection to any of the Queues across a cluster by using the Global JNDI name. It seems one of the main pieces of functionality Distributed Queue gives you is load balanced connections across multiple managed servers. So we have 4 Managed Servers (two on each physical, that communicate over multicast), and each Managed Server has an individual JMS Server which is configured to it's own Data Store.
I am 99% certain I already know the answer to this, but it appears that if you wanted to do a Consume a message off of a Queue, and that Queue exists on each Mgd Server in the Cluster, you cannot technically pull a Message off of any of the Queues (you can only pull the Message off the Queue to which you are connected to). So if I have a Message on Mgd Server 4, and I connect to Mgd Server 1, I won't see the messages on the Queue from Mgd Server 4.
So is there a way in Java EE or WLS to consume a message from all the nodes of a Queue (across the Cluster). Like a view into every instance of the Queue on each Mgd Server? It doesn't appear so and the documentation makes it seem like this is not possible, as well as this video (around minute 5):
http://www.youtube.com/watch?v=HAKixK_wp0Q
No you cannot consumer a message that is delivered to one managed server when your client is connected to another managed server of the same cluster.
Here's how it works.
When using UDT, wls provides a JNDI name that resolves internally into 4 distinct JNDI names for each of the managed server, the JMS servers on each of the managed servers are distinct.
When using the UDQ JNDI name when you post a message, it gets to one of the 4 managed servers using the algorithm you chose and other configuration done in your connection factory.
When a message consumer listens to the UDQ it gets pinned to the JMS server on one of the managed servers. It has no visibility about messages in the other servers.
Usually UDQ is used in scenarios where you want the message to be consumed concurrently by more than one managed server. You would normally deploy a MDB to the cluster, meaning the MDB will be deployed to each of the managed server and each of these will be able to consume the messages from their local JMS server.
I believe you can if your message store is config'd to use a database. If so, then I would think removing an item from the queue would remove it from the shared db table. I.e. all JMS servers are pointing to the same db instance and table. That should be pretty easy to test, too.