I have some experience doing REST microservices architecture, but now I want ot start using kafka as a message provider in my Event drivenr architecture.
I have an issue understanding how the communication works between microservices.
Lets say I have two microservices one that deals with users and another one with messages between users.
Users can be created, and then users can send another users messages, and I am planning to do them with topics and then store them in a MongoDB.
How do I retrieve this messages in a event driven architecture?
EDIT: The example is quite simple is just for educational purpose and to practice with Apache-Kafka.
For clarity, let's call user messages instead as posts.
One possible architecture is all users create post objects, which are sent to a Kafka topic that contains information such as timestamp, sender and sendee, rather than directly as a database insert.
Out of Kafka, you write some consumer process(es) to poll all messages from the topic, then insert to Mongo (and possibly also a search engine or machine learning / text categorization model).
On another user account, they (periodically) query the database for all posts sent to them, as Kafka is not indexable by a user account and one topic per user is not a scalable design pattern.
The gist is that users aren't directly retrieving posts from Kafka, but you've buffered messages in Kafka so they can be forked into different models so that you can decouple your application from a Mongo dependency (except for having a search feature)
From events, you can separate out "posts-new", "posts-edit", and "posts-deletes" as separate topics. For more context about designing this, you can look into Event Storming
Related
i'm trying to learn Microservices by implementing a sample project, tried to pick a semi-complex one to face real world challenges in Microservice architecture.
this is a simplified version of the project flow that I designed till here:
the flow
as you can see in the image I'm trying to get the list of appointments for a specific company, but since the required data is inside different Microservices, for getting the appointments I have to follow these steps:
the API gateway (bff) will get the request from frontend that contains a token
bff will authorize the jwt token by sending it to the users ms
appointments are separated by companies ids, so before getting the appointments, I need to get the user company
company id will be sent to appointments to get the appointments for the company
appointments will check to see if actor is authorized to get the list of appointments by its role (came earlier from the user ms)
appointments will return the list of appointments
inside appointments as you can see in the entities, I do have the id of both sides (sideA, sideB)
bff will get those users details by the ids from users
inside appointments data that is returned, there is a customer_id that is the id of a customer inside the company ms so bff send another request to the company to get the customer details
inside customer details, there is an id of a project that the customer is eager to visit so, bff will send a request to get the project from the projects ms
at the end, bff will join the data and return it to the frontend application.
this is also the simplified version of entities inside Microservices:
entities
right now, i'm using composition API approach to get the data I need, but as you can see the flow is complicated, and I can't think of a way to implement pagination, since I might need to sort, filter and then paginate the data, so I think in this situation, this might be a good idea to use CQRS pattern, but the problem is since I have many situations like this, I have to implement lots of CQRS services.
I'm wondering if:
is it possible to create a single CQRS service to have all the data for read purpose, instead of CQRS for each situation?
for some situations like this, the CQRS read database will becomes almost identical to a monolith architecture db. is. this okay?!
is there any alternative way to scape the complexity of creating and managing multiple CQRSs with partial repetitive data?
CQRS will help you get all the required details in one call. A CQRS service will have multiple tables that are part of different microservices.
An example will be like "OrderViewService" will require to listen and store events from "OrderService", "DeliveryService", "AccountingService". But it wont be listening to multiple other services which are not of concern for "OrderViewService".
So the point I am making here is the database won't become so similar to the monolithic database as it would have a lot more details.
For your project you might require a single CQRS that may deliver your requirements.
As it seems your requirements have dependency on all of the microservices and so a single CQRS service could help you solve the requirements.
Also if you are concerned about the space make sure what details would be required were only be saved to the view/read CQRS database. Thereby ensuring that we are not overwhelming the db with all the details from all the services.
As application grows there can be multiple CQRS services listening to different services or a combination of services and thereby serving their responsibilities.
Reference - https://microservices.io/patterns/data/cqrs.html
I think this explains and these are my thoughts about CQRS. Let me know if you have any questions post it as comments.
Soon I'll start a project based on a Microservice Architecture and ones of the components I need to develop is a Worker Service (or Daemon).
I have some conceptual questions about this.
I need to create a worker service that send emails and sms. This worker service need the data to send this emails. Also, I need to create a micro service that allow users to create a list of emails that need to be sanded by this Worker service. But both of then need to consume data from the same database.
In my worker service I should consume a micro service resource to get the data or it's ok that this worker service have a connection to the same database that my micro service?
Or is best that my worker service also has the api endpoints to let the users create new lists of emails, add or modify configuration and all the other functionalities i need to implement? This sound like a good idea, but I'll get a component with two responsibilities, so I have some doubts about that.
Thanks in advance.
Two microservices sharing the connection to the same database is usually a bad idea. Because each service should be the owner of its own data model and no one else should access it directly. If a service needs data of the domain of another service it should get it calling the owner via API or replicating the model in a read-only way in its own dabase and update it using events for example.
However, I think that for your current use case the best option is to provide the worker with all the information that it needs to send an email, (address, subject, body, attached files...) so the only responsibility of the worker will be to send emails and not to fetch the information.
It could provide also the functionality to send emails in batches. In the end, the responsibility of the service will be only one "To send emails" but it can provide different ways to do it (single emails, batches, with attached files... etc)
I am new to microservices and trying to break up a big monolithic application into microservices. While scoping the microservice I am unable to decide whether I should go for a data duplication between services or ignore SRP by clubbing all requiring the same data into 1 service. Following is the scenario.
I have a service which receives Customer order say build a car with these parts and features. Now I have other 2 functionalities which uses the Parts and features to derive some runtime value say ;
If the order contains part A and Feature A then perform X operation. As each of these functionalities have there respective UI for configuration and runtime engine to derive the output and most of the time changes only comes in these respective function blocks, I thought of creating the separate microservices.
Creating the separate microservice would need data(Parts and Features) to be duplicated. Another option could be given each of these service uses the same data is clubbing all of them into 1, but with that I again create a big service which if goes down will stop all 3 functionalities and is against SRP. Another option could be when the data is required by the other 2 services make a call and get it from Order Service, but that is making it highly dependent and getting the data over network for each operation.
Can anyone suggest what would be ideal to do in such case.
Microservice should communicate via events using publisher/subscriber model and topics. See the tri-lateral design pattern. In a pub/sub system the producer writes the event to the topic and a broker makes it available to each subscriber, aka microservice. This is one advantage of eventing, one producer, and many consumers.
Order A has a BOM that expresses parts and features an order represents. When the order is placed, other services need that information such as the feature/part microservice and UI components and their respective datastores. For example, define two topics parts-ordered and features-ordered.
The order microservice writes to the parts and features topics respectively. The feature, parts, and UX microservices read and act upon those events.
You need to make a choice about your system of record. You can write to an orders DB before publishing the events, or let the pub/sub it. In your example, it sounds like you are talking about three different tables; order, parts, and events. That decomposes cleaning into the microservices you describe where each maintains the data it is responsible for. I'd be inclined to just use the pub/sub as your audit trail.
If you are writing in Go we have blueprints for the most common design patterns.
First, you mentioned that you are trying to convert the monolithic application into microservices. You can create/caters the microservices on basis of domain data, we can be called it domain-driven architecture.
Suppose you have the business functionality for customer data, customer order, customer order handling, and customer payment. And currently, it's part of a monolithic application. So you can create the subdomain for each functionality like Customer domain, Order domain, order handle domain, and payment domain respectively. Each domain contains several microservices depends on the business requirement.
For e.g you can check the Amazon website, In personal/customer data, you see the customer name, phone number, address, billing account information, delivery address type(office/home). In this case the under customer domain, there will be 3 microservices required(It totally depends on your domain design). One for customer(handles customer name, phone number, reference of Billing account id, reference of address id), second for Billing account(Billing account number, billing account information, reference if customer id), third for Address data(customer office address, preferable address). And for each microservice, there will be a dedicated database/buckets, Only that microservice can change/add the data. If any other microservice wants to add/update/get data, it needs to be get by calling that microservices HTTP endpoint over the network.
Updating the data in other microservice::
Now coming to your question about data duplication, Let's consider the above example.
If Customer microservice wants to store/ cache the billing account data for some purpose, that microservice can store that data in the database but again Customer microservice needs to make sure that, the current data of the billing account is always real one and not the old one. For this customer, microservice needs to listen to the event whenever there is update in billing account data, so old data in billing account gets purged and customer microservice always has the latest data of billing.
you can read here about event driven architecture.
https://en.wikipedia.org/wiki/Event-driven_architecture#:~:text=Event%2Ddriven%20architecture%20(EDA),sale%22%20to%20%22sold%22.
You can read more about this at the below links about Domain driven design.
https://www.thoughtworks.com/insights/blog/domain-driven-design-services-architecture
https://en.wikipedia.org/wiki/Domain-driven_design
This is my free book :)
https://github.com/vaquarkhan/microservices-recipes-a-free-gitbook
If you want to create microservice then need to follow microservice guideline.
Now come to real world :) really difficult to meet all microservice requirements as database has own licensing cost etc. so you can choose pragmatic microservices. You can get started with them faster and pick and choose the pieces that make sense for your team.
Design Domain driven design oriented microservice : DDD talks about problems as domains. It describes independent problem areas as Bounded Contexts and each Bounded Context correlates to a microservice.
Where to draw the boundaries is the key task when designing and defining a microservice.
DDD patterns help you understand the complexity in the domain, the domain model for each Bounded Context, you identify and define the entities, value objects, and aggregates that model your domain. You build and refine a domain model that is contained within a boundary that defines your context. And that is explicit in the form of a microservice. The components within those boundaries end up being your microservices.
https://martinfowler.com/bliki/AnemicDomainModel.html
https://github.com/vaquarkhan/Domain-driven-design
https://github.com/vaquarkhan/ddd-by-examples.github.io/blob/master/ddd-factory.pdf
Now you can create layers on top of you microservice and build complex logic using orchestration and choreography.
Example :
Gateway Customer order Application layer microservice --domain model layer microservice infrastructure layer
There are various ways to decompose an application into services.
1.Decompose by business capability
2.Decompose by subdomain
And there are various ways for data management also
SAGA
API Composition
Database per service
Please go through with the link for more details click
Suppose that I have a microservice for messaging. The microservice knows how to send emails. The service have templates of emails that have some sort of "template engine" like pugjs, and can replace data in the body of the message.
I have an user service (used for authentication/authorization for example), and a bank account service (each user have one). Between the User microservice and Bank Account microservice it's clear that we don't have to duplicate any data than de user's uuid.
But I want now to send every day a message to each user with their account statement. The Messaging microservice needs data from the User microservice and the Bank Account microservice.
Okay... This is a small case of the real world. Now I know that to have the benefits of decoupled microservices I must follow some rules:
I can't share databases between microservices
I can't make synchronous requests between microservices
Okay... I can use a broker and each time a new user is created/updated the Messaging microservice can store that data. But really, this is a stupid thing:
I don't want to have inconsistency with this data, and keeping things sync is hard
The development time and complexity of the Messaging Microservice must now consider: listen and extract the relevant data from events, keep data consistent about other domains/services, managing the saved data on database
And think about a Messaging microservice. Really I must store all the data needed to parse the templates?
I read a lot about microservices and people creating rules for their simple examples. But I never really saw a good explanation and real-world examples like above.
So how to have the microservices above without data duplication?
In your domain example I would not let the message service know anything about bank or user details. Instead the message service should just receive instructions to send messages to recipients along with the given content. I would use a dedicated scheduled job (maybe implemented as an account notification service) that performs the work of acquiring the user and account data from the corresponding services, compiles the information for the message service and instructs it to actually send the messages. This introduces another "higher level, business purpose entity/service" but allows you to keep a clear separation of concerns.
In general it will happen frequently that your "basic" domain services are used by another service that represents a specific business purpose and requires their data. Dependency in itself is not a bad thing as long as concerns are seperated clearly and interfaces versioned, changes communicated etc.
Don't forget the whole idea of microservices is for allowing teams to have dedicated responsibilities with clear interfacing. It is about organization as much as it is about architecture.
When making something like a chat application using Spring Websockets, it is useful to know who all is subscribed to any given topic. For, e.g. presence data displayed in the clients.
I know that I can implement ApplicationListener and keep my own list of "connected users", but it seems like the library must already be doing this work.
What's the recommended way to get active subscription info from the library directly (and without maintaining my own list in memory or db).
You're right, you could use ApplicationContext events, but unfortunately those events deal with user sessions events and broker events - so you won't be notified when a user subscribes to a particular topic.
You could do that when using the SimpleBrokerMessageHandler, by getting the SubscriptionRegistry. But again, the SimpleMessageBroker is not for production use.
If you're using RabbitMQ, you can get that information from its REST API.
The thing is, this is very specific to the broker implementation, so I'm wondering if a feature like that makes sense in Spring Framework. Could you open a JIRA issue to start the discussion?