I need inputs on how I can design a simple notification system .
There is system "Z" which generates certain events (this is an external system), and there are two systems A and B (internal systems) interested in getting email notifications about those events. However, A is interested in type "error" and B is interested in type "failure" of events from Z. I am trying to design system N which will translate those events from Z and send them as an email to A and B. I am not sure how many queues/topics I require.
Please let me know if the below steps are valid/needed/or can be improved
System A and B subscribe to emails. If I use a queue or topic the response from N has to be published on that topic, but I want to send out an email.
System N is listening to queue "test" on which system Z is publishing "error" and "failure" messages.
System N read and translates the messages from "test" queue and send email to subscribers i.e. A and B
I am mainly not sure of step 1. How will A and B let system N know that they need email notifications?
It sounds to me like you should deploy some kind of MLM (i.e. mailing list manager) which both systems A and B can use to subscribe to one or more relevant lists (e.g. an "error" list and/or a "failure" list). Then system N can send emails to those lists as needed when system Z generates the corresponding events.
Here is a list of well-known MLM implementations.
Related
I'm looking for the best approach as to how I can go about doing validation of a message as its enqueued in async messaging based systems.
Scenario:
Let's say we have a two services A and B where they need to interact with each other asynchronously. And we have a queue between them lets say SQS which will receive the message from A, which will be then polled by service B.
Ask:
How can I validate the message like doing schema validation as its enqueued to SQS since currently SQS doesnt have any in-built schema validation functionality like we have for JMS
Couple of options I can think of:
Have a validation layer maybe a small service sitting between A and SQS queue but not sure how feasible this will be
Use some sort of MOM like AWS Eventbridge between A and SQS queue as it has functionalities to validate schemas as well as it could act as a central location to store all the schemas
Have a rest endpoint in B that'll do the validation and have SQS sitting behind B but then this removes the async communication b/w A and B
Would appreciate any inputs on the above ask and how it could be resolved via best practices.
I'd recommend to read about the Mediator Topology of Event-Driven architecture style. From the details that you shared, it sounds to me that putting a "Mediator Service" called M for example, which will get messages from A, make the required validations, and then will send the message to SQS on its way to B - will achieve what you want.
Validation of the message payloads can occur on the "way in" or the "way out" depending on your use case and scaling needs. Most scenarios will aim to prevent invalid data getting too far downstream i.e. you will validate before putting data into SQS.
However, there are reasons you may choose to validate the message payload while reading from the queue. For example, you may have many services adding messages, those messages may have multiple "payload versions" over time, different teams could be building services (frontend and backend) etc. Don't assume everything and everyone is consistent.
Assuming that the payload data in SQS is validated and can be processed by a downstream consumer without checking could cause lots of problems and/or breaking scenarios. Always check your data in these scenarios. In my experience it's either the number one reason, or close to it, for why breaking changes occur.
Final point: with event-driven architectures the design decision points are not just about the processing/compute software services but also about the event data payloads themselves which also have to be designed properly.
Let's say I want to set up and event-driven architecture with services A-D where the events propagate as follows
A
/ \
B C
/
D
In other words,
(1) A publishes an event
(2) Subscribers B and C receive A's event
(3) C publishes an event
(4) Subscriber D receive's C's event
One way is to have services B and C directly listen to a queue into which A posts messages. But the issue I see with this is maintenance. Once the system becomes complicated with 1000s of subscriptions, it becomes difficult to have any visibility into how the updates are propagating.
A solution I propose to this problem is to have another service X that knows the tree in the in the first image and is responsible for directing the propagation of events according to the tree. Every service publishes its event to X and it publishes the event to the listening services. So it's kinda of a middleman like
A
|
X
/ \
B C
|
X
|
D
This also makes it easier to track the event propagation.
Are there any downsides to this (other than extra cost associating with twice as much message transferring)?
You’re thinking of events like they are implemented in a Winforms UI where the publisher sends the event directly to the subscriber. That’s not how events work in an EDA architecture. The word “event” has taken on a whole new meaning.
Before we start, you’re jumbling together the ideas of a message and an event when they really need to be kept separate. A message is a request for some action to happen, while an event is notification that something has already happened. The important distinction for this discussion is that a message publisher assumes 1 or more other processes will receive and process the message. If the message is not processed by something, downstream errors will occur. An event has no such assumption and can go unread without adversely affecting anything. Another difference is that once messages are processed they are typically thrown away, whereas events are kept for an extended period (days, or weeks).
With that in mind, the ‘X’ service you talk about already exists (please don’t build one) and is integral to the process – it’s called the bus. There are 2 types of bus; a message bus (think RabbitMQ, MSMQ, ZeroMQ, etc) or event bus (Kafka, Kinesis, or Azure Event Hub). In either case, a publisher puts a message on to the bus and subscribers get it from the bus. You may implement the bus servers as multiple physical buses, but when imagining it think of them all being the same logical bus.
The key point that’s tripping you up, and it’s a subtle difference, is thinking that the message bus has business logic indicating where messages go. The business logic of who gets what message is determined by the subscribers – the message bus is just a holding place for the messages to wait for pickup.
In your example, A publishes an event to the bus with a message type of “MT1”. B and C both tell the bus that they are interested in events of type “MT1”. When the bus receives the request from B and C to be notified of “MT1” messages, the bus creates a queue for B and a queue for C. When A publishes the message, the bus puts a copy in the “B-MT1” queue and a copy in the “C-MT1” queue. Note that the bus doesn’t know why B and C want to receive those messages, only that they’ve subscribed.
These messages sit there until processed by their respective subscribers (the processes can poll or the bus can push the messages, but the key idea is that the messages are held until processed). Once processed, the messages are thrown away.
For C to communicate with D, D will subscribe to messages of type “MT2” and C will publish them to the bus.
Constantin’s answer above has a point that this is a single point of failure, but it can be managed with standard network architecture like failover servers, local message persistence, message acknowledgements, etc.
One of your concerns is that with 1000’s of subscriptions it becomes difficult to follow the path, and you’re right. This is an inherent downside of EDA and there’s nothing you can do about it. Eventual consistency is also something the business is going to complain about, but it’s part of the beast and is actually a good thing from a technical perspective because it enables more scalability. The biggest problem I’ve found using the term Eventual Consistency is that the business thinks it means hours or days, not seconds.
BTW, This whole discussion assumes the message publishers and subscribers are different apps. All the same ideas can be applied within the same address space, just with a different bus. If you’re a .net shop look at Mediatr. For other tech stacks, there are similar solutions that I’m sure google knows about.
If your main concern is visibility into the propagation of events (which is a very valid concern for debugging and long-term application maintenance of a distributed system), you can use a correlation identifier to trace the generation of messages from the initial event through the entire chain. You don't need to build another layer of orchestration -- let your messaging platform handle that for you.
Most messaging platforms/libraries have the concept built in: e.g., NServiceBus defines a ConversationId field in the message headers, and AMQP defines a correlation-id field in the basic messaging model.
Your system should have some kind of logging that allows you to audit messages -- the correlation ID will allow you to group all messages that result from a single command/request to make debugging distributed logic much simpler.
If you set a GUID in the client requests, you can even correlate actions in the UI to the backend API, right through all the events recursively generated.
It is OK but the microservices shouldn't care how they get the messages in the first place. From their point of view the input messages just arrive. You will then be tempted to design your system to depend on some global order of events, which is hard in a distributed scalable system. Resist that temptation and design your system to relay only on local ordering of events (i.e. the ordering in an Event stream emitted by an Aggregate in Event sourcing + DDD).
One downside that I see is that the availability and the scalability may be hurt. You will then have a single point of failure for the entire system. If this fails everything fails. When it needs to be scaled up then you will have again problems as you will have distributed messaging system.
I've read my docs most examples are for basic use cases.
Where simply one process publish X event and another subscribe to X event.
But in my applications X is kind of variable. so lets say i've X means my user.
so i can do publish from one server event like user-ID means if i've 1000s of user connected to server so will that be Okay to publish and subscribe to so many dynamic topics, and then another 20 servers subscribe to that 1000s topics on this server.
Lets see the example.
i've 10 servers. each server with 1000 users connected. so total 10k users.
i need to send X data from each user to another user.
so i've did this.
X server publish user-ID data (1 publish user's who is connected, 1K publish)
Y server subscribe user-ID data (10k subscribe request to sent each server)
What should be optimal way of pub sub with dynamic topics so less bandwidth used among servers?
Notice::
user-ID is just an example where ID is dynamic number, and it publish some real time data which can't be stored anywhere.
In ZeroMQ subscription matching is implemented in the PUB socket with a prefix-matching trie. This is a very efficient data structure, and I would expect that 10K subscriptions and 10K msg/sec would be no problem at all.
The PUB socket only sends messages for matching subscriptions (so there is no "waste"). If a message doesn't match any subscription then the PUB socket will drop it. Matching messages are only sent to SUB sockets that have subscribed to them.
When you add or remove a subscription, the SUB socket will send a message its connected PUB socket(s). Each PUB socket will then update its topic trie.
My guess is 10k subs and 10k msgs/s is no problem, but the best thing to do would be to write some test code and try it out. Once nice thing about ZeroMQ is that it's not much work to test different architectures.
As far as I know in pyzmq API publisher can send messages to any topic
socket.send("%d %d" % (topic, messagedata))
and subscribers set a filter on these topics for topic of their interests with setsockopt
topicfilter = "10001"
socket.setsockopt(zmq.SUBSCRIBE, topicfilter)
So I think you can fully implement your plan.
although I went through the guide I cant find a way to do the following:
we have n publishers,
we have m subscribers.
Each subscriber subscribes to certain type of msg(one publisher can send more than one kind of the message, multiple pubs can emit msgs of same type).
How to create N to N(or N to 1 to N) pattern in 0MQ that does that?
Pattern? A publisher will accept any number of subscribers, and your subscribers can connect to multiple publishers. I don't see any issue here, just go ahead and do it.
In the future, perhaps you'll be better served by stating what you want to accomplish, rather than how you think it should be done. Just hooking up a bunch of subscribers to a bunch of publishers might not be the best way to go about whatever it is that you're doing.
0MQ automatically allows N to M networks; just connect each subscriber to each publisher. That can become a burden to manage, in which case you can use a simple proxy that all publishers and subscribers connect to, see http://zguide.zeromq.org/page:all#toc36.
I have done a simple example in C which uses the built in zmq forwarder device to forward messages from N publishers to N subscribers. The code is available at https://gitorious.org/hellozmq
Are there any texting services out there (like Twilio) with APIs that allow you to see what message another message is in response to? For example: I want to tell whether message 3 is in response to 1 or 2.
A->B: Do you like ice cream?
A->B: Do you like pizza?
B->A: Yes.
I'm guessing that there's just no threadedness to text messages, and this is impossible. But might as well have a canonical answer out there.... Reasons for this belief:
When you send emails to a phone number at txt.att.net, it uses a different number each time.
The texting services that I've looked at don't offer the feature
The phone UIs I've used all present it as a flat message list rather than a threaded list
Can someone confirm this?
There is no native support for this type of functionality in SMS. Message threading for SMS is inferred on the client side from context. When I send a text to person 1, if I receive a reply from person 1 within a given time-frame, I can assume that it was part of the same "conversation". (I actually implemented something like this for a chat client that I was working on)
I have also seen where the original message is included with the response, and the client can group the messages accordingly (take a look at gmail), but this could be messy if your clients were not the same.
Just a few thoughts.