I have just started work on a software which uses IBM MQ for some communication.
As I understand that MQ can be used for many to one communication and one to many communication.
Lets say there are 3 Business applications A, B and C. A wants to send a message using MQ to B and another message to C but A is only using one queue Queue1.
Now my question is if we can define (in MQMD or otherwise) that a certain message is only for B NOT for C, hence only B can retrieve it from Queue1 whenever B is available. If not how can we do this if it is at all possible?
One other thing is can we make a separate queue Queue2 only for A-B communication?
It is better to use separate queues. For example use queue QA2B for application A to send messages to application B and QA2C for application A to send messages to application C. This way traffic is separated out and you can administratively restrict application B from receiving message meant for C and vice-versa.
It is possible to use just one queue wherein application A while sending messages sets a message property that says something like "Message is for B" or "Message is for C". Application B uses a selector to match the property value "Message for B" while receiving messages. Similarly application C also uses selector "Message for C" and receives messages. But note if either B or C receives messages without any selector, then messages can go to wrong hands.
Related
I need inputs on how I can design a simple notification system .
There is system "Z" which generates certain events (this is an external system), and there are two systems A and B (internal systems) interested in getting email notifications about those events. However, A is interested in type "error" and B is interested in type "failure" of events from Z. I am trying to design system N which will translate those events from Z and send them as an email to A and B. I am not sure how many queues/topics I require.
Please let me know if the below steps are valid/needed/or can be improved
System A and B subscribe to emails. If I use a queue or topic the response from N has to be published on that topic, but I want to send out an email.
System N is listening to queue "test" on which system Z is publishing "error" and "failure" messages.
System N read and translates the messages from "test" queue and send email to subscribers i.e. A and B
I am mainly not sure of step 1. How will A and B let system N know that they need email notifications?
It sounds to me like you should deploy some kind of MLM (i.e. mailing list manager) which both systems A and B can use to subscribe to one or more relevant lists (e.g. an "error" list and/or a "failure" list). Then system N can send emails to those lists as needed when system Z generates the corresponding events.
Here is a list of well-known MLM implementations.
Let's say I want to set up and event-driven architecture with services A-D where the events propagate as follows
A
/ \
B C
/
D
In other words,
(1) A publishes an event
(2) Subscribers B and C receive A's event
(3) C publishes an event
(4) Subscriber D receive's C's event
One way is to have services B and C directly listen to a queue into which A posts messages. But the issue I see with this is maintenance. Once the system becomes complicated with 1000s of subscriptions, it becomes difficult to have any visibility into how the updates are propagating.
A solution I propose to this problem is to have another service X that knows the tree in the in the first image and is responsible for directing the propagation of events according to the tree. Every service publishes its event to X and it publishes the event to the listening services. So it's kinda of a middleman like
A
|
X
/ \
B C
|
X
|
D
This also makes it easier to track the event propagation.
Are there any downsides to this (other than extra cost associating with twice as much message transferring)?
You’re thinking of events like they are implemented in a Winforms UI where the publisher sends the event directly to the subscriber. That’s not how events work in an EDA architecture. The word “event” has taken on a whole new meaning.
Before we start, you’re jumbling together the ideas of a message and an event when they really need to be kept separate. A message is a request for some action to happen, while an event is notification that something has already happened. The important distinction for this discussion is that a message publisher assumes 1 or more other processes will receive and process the message. If the message is not processed by something, downstream errors will occur. An event has no such assumption and can go unread without adversely affecting anything. Another difference is that once messages are processed they are typically thrown away, whereas events are kept for an extended period (days, or weeks).
With that in mind, the ‘X’ service you talk about already exists (please don’t build one) and is integral to the process – it’s called the bus. There are 2 types of bus; a message bus (think RabbitMQ, MSMQ, ZeroMQ, etc) or event bus (Kafka, Kinesis, or Azure Event Hub). In either case, a publisher puts a message on to the bus and subscribers get it from the bus. You may implement the bus servers as multiple physical buses, but when imagining it think of them all being the same logical bus.
The key point that’s tripping you up, and it’s a subtle difference, is thinking that the message bus has business logic indicating where messages go. The business logic of who gets what message is determined by the subscribers – the message bus is just a holding place for the messages to wait for pickup.
In your example, A publishes an event to the bus with a message type of “MT1”. B and C both tell the bus that they are interested in events of type “MT1”. When the bus receives the request from B and C to be notified of “MT1” messages, the bus creates a queue for B and a queue for C. When A publishes the message, the bus puts a copy in the “B-MT1” queue and a copy in the “C-MT1” queue. Note that the bus doesn’t know why B and C want to receive those messages, only that they’ve subscribed.
These messages sit there until processed by their respective subscribers (the processes can poll or the bus can push the messages, but the key idea is that the messages are held until processed). Once processed, the messages are thrown away.
For C to communicate with D, D will subscribe to messages of type “MT2” and C will publish them to the bus.
Constantin’s answer above has a point that this is a single point of failure, but it can be managed with standard network architecture like failover servers, local message persistence, message acknowledgements, etc.
One of your concerns is that with 1000’s of subscriptions it becomes difficult to follow the path, and you’re right. This is an inherent downside of EDA and there’s nothing you can do about it. Eventual consistency is also something the business is going to complain about, but it’s part of the beast and is actually a good thing from a technical perspective because it enables more scalability. The biggest problem I’ve found using the term Eventual Consistency is that the business thinks it means hours or days, not seconds.
BTW, This whole discussion assumes the message publishers and subscribers are different apps. All the same ideas can be applied within the same address space, just with a different bus. If you’re a .net shop look at Mediatr. For other tech stacks, there are similar solutions that I’m sure google knows about.
If your main concern is visibility into the propagation of events (which is a very valid concern for debugging and long-term application maintenance of a distributed system), you can use a correlation identifier to trace the generation of messages from the initial event through the entire chain. You don't need to build another layer of orchestration -- let your messaging platform handle that for you.
Most messaging platforms/libraries have the concept built in: e.g., NServiceBus defines a ConversationId field in the message headers, and AMQP defines a correlation-id field in the basic messaging model.
Your system should have some kind of logging that allows you to audit messages -- the correlation ID will allow you to group all messages that result from a single command/request to make debugging distributed logic much simpler.
If you set a GUID in the client requests, you can even correlate actions in the UI to the backend API, right through all the events recursively generated.
It is OK but the microservices shouldn't care how they get the messages in the first place. From their point of view the input messages just arrive. You will then be tempted to design your system to depend on some global order of events, which is hard in a distributed scalable system. Resist that temptation and design your system to relay only on local ordering of events (i.e. the ordering in an Event stream emitted by an Aggregate in Event sourcing + DDD).
One downside that I see is that the availability and the scalability may be hurt. You will then have a single point of failure for the entire system. If this fails everything fails. When it needs to be scaled up then you will have again problems as you will have distributed messaging system.
I'll try to explain this the best I can.
As I store my data that I receive from my ActiveMQ queue in several distinct locations, I have decided to build a composite Queue so I can process the data for each location individually.
The issue I am running into is that I currently have the Queue in a production environment. It seems that changing a queue named A to a composite Queue also called A having virtual destinations named B and C causes me to lose all the data on the existing Queue. It does not on start-up forward the previous messages. Currently, I am creating a new CompositeQueue with a different name, say D, which forwards data to B and C. Then I have some clunky code that prevents all connections until I have both a) updated all the producers to send to D and b) pulled the data from A using a consumer and sent it to D with a producer.
It feels rather messy. Is there any way around this? Ideally I would be able to keep the same Queue name, have all its current data sent to the composite sub-queues, and have the Queue forward only in the end.
From the description given the desired behavior is no possible as message routing on the composite queue works when messages are in-flight and not sometime later when that queue has already stored messages and the broker configuration is changed. You need to consume the past messages from the initial Queue (A I guess it is) and send them onto the destinations desired.
In Spring Integration, I have a chain of services, like this:
message -> A -> B -> C -> D -> ... -> output
This works fine. I want to make each of the services asynchronous and to make them pessimistic. Each of them will get a message, process it and send it to the next service in chain. However, it will not wait till the whole chain finishes. It will continue processing the next message and so on. Standard async here.
However, let's say service B is slower than A and that it accumulates 10k messages in its inbound channel queue and at that time the system crashes. I want to be able to restore the system by figuring out where I left and re-processing the messages. For that reason, I want each of the services to know which of the messages it processed was successfully consumed by the following service. The difference between sent vs. processed.
My idea is to do it similar to this (fancy ascii):
-> A --> B -> C -> ...
^ |
| ack |
\-----/
That is, A will send to B, B will process and when it is done successfully it will send an ack to A. A will then remove that particular message from the store, so that the next time it runs, it will not re-process it. I thought I would just put a splitter after B that will call a different method on service A (i.e. ackProcessed).
Is this how it should be done in SI or is there another way I'm missing? I'm primarily asking for a confirmation I'm not missing something supported out-of-the-box or something that will not force me to create a splitter after each of the services.
It wouldn't be a splitter; more likely a pub-sub channel and the ack would probably want to go to a different method in A (i.e. a different service-activator that references the same bean, different method; and the methods share some state).
An easier solution would be to use a persistent message channel (e.g. JMS, RabbitMQ, or a message-store-backed QueueChannel). That way the framework will take care of everything for you.
We are using IBM WebSphere MQ as JMS provider with Spring MDP (Message Driven POJO).
Is there any way in JMS where we can configure time related properties in message so that message can be consumed at particular defined time only?
For example, if I am sending three messages into queue M1, M2 and M3. Where, I can configure M2 message property let say 3 AM. And consumer side, consumer can only pick this message # 3 AM only. If time is not defined, messages should be consumed in a way that JMS Receiver does.
JMS 2.0 specification has defines Delivery Delay. With this feature a message producer can specify that a message must not be delivered until after a specified time interval. The message will be available for delivery after the specified time. But this may not help you as you want to a message to be consumed at a specified time. Typically messaging applications are designed to consume messages as soon as they are made available by the messaging provider.
If you want to process messages at a specified time only, you could create another queue "queue_3am", and schedule a reader to run exactly at 3am.
A variation is to set the timestamp as a message property. So one queue could contain messages to be processed at different points in time. The reader could use message selectors to get relevant messages only.
But you should use a "message pickup timeframe" by adding two timestamps as message properties, for eaxmple set the window to 1 or 5 minutes.
The receiver can use a message selector: A selector is a condition using message properties.
Have a look at this