Soon I'll start a project based on a Microservice Architecture and ones of the components I need to develop is a Worker Service (or Daemon).
I have some conceptual questions about this.
I need to create a worker service that send emails and sms. This worker service need the data to send this emails. Also, I need to create a micro service that allow users to create a list of emails that need to be sanded by this Worker service. But both of then need to consume data from the same database.
In my worker service I should consume a micro service resource to get the data or it's ok that this worker service have a connection to the same database that my micro service?
Or is best that my worker service also has the api endpoints to let the users create new lists of emails, add or modify configuration and all the other functionalities i need to implement? This sound like a good idea, but I'll get a component with two responsibilities, so I have some doubts about that.
Thanks in advance.
Two microservices sharing the connection to the same database is usually a bad idea. Because each service should be the owner of its own data model and no one else should access it directly. If a service needs data of the domain of another service it should get it calling the owner via API or replicating the model in a read-only way in its own dabase and update it using events for example.
However, I think that for your current use case the best option is to provide the worker with all the information that it needs to send an email, (address, subject, body, attached files...) so the only responsibility of the worker will be to send emails and not to fetch the information.
It could provide also the functionality to send emails in batches. In the end, the responsibility of the service will be only one "To send emails" but it can provide different ways to do it (single emails, batches, with attached files... etc)
Related
As part of my project, I'd like to use microservices. The application is a store website where the admin can add products and the user can order and buy them.
I envision implementing four services: admin service, user service, product service, and order service.
I had trouble with handling data between multi services but it's solved by duplicating some necessary data using message brokers.
I can do this solution between product and user and order service because I need some of the data not all of them
Now, my question is about handling admin service because in this service I need to access all of the data, for example, the admin should have a list of users and the ability to add new products or update them.
how can I handle data between these services and the admin service?
should I duplicate all data inside the admin service?
should I use Rest API?
no thats wrong. it seems you want run away from the fact. in general duplication is an anti-pattern mostly in case you describe.
the way you thinking about admin-service is wrong.
because in this service I need to access all of the data
i dont think you need to have such a service. accessing the data based on users must be handled by Identity server(oidc Oauth) which is the separated service and handle the accessing end points .
for example the product-service provides 1-return product list 2-return individual product data 3-create data. the first two can access by both user and admin but the 3rd must be accessed by admin. one of identity server duty is to identify user in case of user interaction(login) with services.
ADMIN Scenario
user-client request create product endpoint(services eg:product.service).
client-app(front end app) is configed with identity server and realize there is no require identity tokens and redirect to identity server login.
NOTE: there is also identifying the client-app itself i skipped.
user-client login and get require token that based on his claims and roles and etc.
user-client request create product endpoint with tokens included in request header
endpoint (product service) receives the request and check the header (the services also configured base on identity server and user claims)
get the user claims info.
the create-product requires admin role if its there then there we go otherwise no access.
the image uses identity server 4 . there are also several kinds and also you can implement by your self using 0AUTH and oidc protocol libraries.
so the admin just request to the certain service not getting data through the separate service for this goal.
Communication between Service:
the most struggling part of microservices is the wiring it up. the wiring is directly the consequence of your design.(recommand deep study on Domain Driven Design).
asynchronous communication :
to avoid coupling between services mostly use asynchronous communication which you pass event eg:brokers like rabbitmq and kafka..etc , redis etc. in this communication the source service who send event does not care about response and not wait for it.just it always ready to listen for any result event. for example
the inventory service creates item
123|shoe-x22|22units
and this service fire event with data 123|shoe-x22(duplicate maybe or maybe not just id) to product service to create but it does not wait for response from product service that is it created successfully or not.
as you see this scenario is unreliable in case of fault and you need handle that so in this case you have to study CAP theory,SAGA,Circuit-breaker.
synchronous communication :
in this case the service insist to have response back immediately. this push service to become more coupling. if you need performance then you can use gRPC communication other wise simple api call to the certain service. in case of gRPC i recommand using libraries like MassTransit
which also can be used for implementingf gRPC with minimum coupling.
Some of Requests need data from multiple services
if you are in such situation you have two options.
mostly microservices architecture using APIGATE WAY (EG:nginx,OCELOT,etc)
which provide reverse-proxy,load balancing,ssl terminations etc. one of its ability is to merge the multiple responses from a request.but it just merge them not changing the data structure of response.
in case of returns desire response data structure you may create an Aggregator service which itself calls other two, gathers data and wrap it in desire format and return it.
so in the end still the Domain Driven Design is the key and i think i talked tooooo much. hope help you out there.
If my company sole purpose is processing a specific payload, but there is lot of orchestration for it. Should the orchestration, be in a separate domain. Lets say, payment is what the company does, but there is a workflow service, for that payment payload? If that is in a seperate domain, how should the workflow service domain talk to payment service domain?
It's better to use Event Driven Design which powered by message services like RabitMQ (or Kafka, MSMQ, or ..). It's not recommended to speech microservice each other directly via APIs. On the other hand to aggregate, some information from multiple services you can use 2 techniques, first using a BFF (back end for frontend layer), Second use a materialized view to gather information from many services.
I m building a spring boot microservices, and i have some questions
I have an account microservice, a payment microservice, a product microservices... in these microservices, some requests sometimes need to use a mailing api, an sms sending api, or a push notification api..
What i have done now is create a microservice for mailing, microservice for sending sms and microservice for push notification.
What i can't seem to solve is how to make these microservices used only internally. for example, forbid users to directly call the mailing microservice.
before creating this question on stackoverflow, i dud myself, why i'll not put the code for sending sms in a library, and the same for sending emails and push notifications and add them to the microservice .. and when a microservice has need to use one of these apis i add the needed library .. for example i create a push notification library, and i add it to each microservice that needs to do a push notification ..
what is the best approach to integrate these mailing, sms and notification services into my microservice project, and respecting security by forbidding users to use them directly
I don't know what to do, can someone advise me?
Well it is not exactly clear to me what do you mean by "forbidding users to use them directly" but usually as it is pointed out #kavhakaran's answer you should put the security measures to prevent your services from abuses.
In that answer only network related part is focused as far as I can see. There should also be a second level which is about user authorization. That means you can/should have proper roles and authorization definitions for the services you would like to secure. And based on provided roles you can authorize the client to use the services.
That is how it works for cloud services usually as well. You will be provided an api-key in order to consume some cloud service and they will check if the api-key is authorized for the requested service etc.
You shouldn't worry about other micro services calling the mailing microservice or sms microservice in the application code. If you think about this concern, this will apply to any internal mircoservice. This concern can be handled in infrastructure level
Let me give you an example, you have a database running somewhere, does your microservice does anything to make sure, it is the only one talking to that database. The answer is no. At infrastructure level, whatever cloud infrastructure you are using, they allow to define security rules/ network policies, that lets you define who can talk to who. ie. rules for incoming traffic and rules for outgoing traffic
If they are public facing microservices, that is a different question. These are internal services
Some examples based on infrastructure
AWS SecurityGroups
AWS subnets
Kubernetes Network Policy
And also I want to add a point which may not be directly related to your question. The services in question seems to be very good candidates as asynchronous services. Then no services talk to them directly, sending services put the notifications in queue or kafka topic and these services consume from the topic. So now it is making sure only relevant services send it to queue or topic at network level
I would not recommend to use libraries for sending sms, emails and push notifications across your Microservices. This would lead to dependencies on source code level which I would try to avoid in a Microservices architecture if possible.
Concerning the architectural issues of your question:
From my experience it is a good idea to have separate services for handling notifications such as sms, email, etc. because with that you create an abstraction between your Microservices and the concrete notification infrastructure such as third party sms, email or push notification services.
Usually the core requirements to, for instance, sending an email will more or less be the same over time. But you might come into a situation where you want to exchange one third party service for another - for instance due to cost concerns, performance concerns or other reasons.
If you choose to directly communicate with the notification infrastructure from each Microservice that needs to send emails you would have to adapt all these Microservices when you switch from one email service to another, no matter if you use a shared library or each Microservices implements the communication with that service on its own.
But if you have a separate Email Microservice that is used by all your Microservices that need to send email notifications, you only have to change the Email Microservice itself to communicate with, for instance SendGrid instead of MailJet (just to name two third-party Email services). Your other Microservices aren't even concerned with that change.
Concerning the security aspects:
As it was already mentioned, if you choose to communicate with your notification services asynchronously the security aspects will be addressed on the infrastructure level by allowing the Microservices to access messaging infrastructure based on the authentication and access control mechanisms provided by the corresponding messaging services (be it RabbitMQ, Azure Service Bus, Kafka, AWS SQS, etc.)
Or if you choose to call your notification services via REST APIs from your Microservices you can look into token-based authentication via OpenID Connect (e.g. via Client credentials flow for machine-to-machine security).
One other thing to consider:
I would also think about other shared functionality that could be common to sms, email and push notification services such as user preferences - e.g. which kinds of notification does a user want to receive. This could also be some functionality you do not want all of your Microservices have to know about. So you could think of a notification service that is concernced with this kind of responsibility and would be responsible to delivery the notifications over the different kinds of channels (email, sms, push) based on the user prferences. Or you could have separate Microservice for user preferences which is than accessed by your sms, email and push notification Microservices. But there is no obvious answer to which option is better because this strongly depends on the use cases you have to deal with.
Suppose that I have a microservice for messaging. The microservice knows how to send emails. The service have templates of emails that have some sort of "template engine" like pugjs, and can replace data in the body of the message.
I have an user service (used for authentication/authorization for example), and a bank account service (each user have one). Between the User microservice and Bank Account microservice it's clear that we don't have to duplicate any data than de user's uuid.
But I want now to send every day a message to each user with their account statement. The Messaging microservice needs data from the User microservice and the Bank Account microservice.
Okay... This is a small case of the real world. Now I know that to have the benefits of decoupled microservices I must follow some rules:
I can't share databases between microservices
I can't make synchronous requests between microservices
Okay... I can use a broker and each time a new user is created/updated the Messaging microservice can store that data. But really, this is a stupid thing:
I don't want to have inconsistency with this data, and keeping things sync is hard
The development time and complexity of the Messaging Microservice must now consider: listen and extract the relevant data from events, keep data consistent about other domains/services, managing the saved data on database
And think about a Messaging microservice. Really I must store all the data needed to parse the templates?
I read a lot about microservices and people creating rules for their simple examples. But I never really saw a good explanation and real-world examples like above.
So how to have the microservices above without data duplication?
In your domain example I would not let the message service know anything about bank or user details. Instead the message service should just receive instructions to send messages to recipients along with the given content. I would use a dedicated scheduled job (maybe implemented as an account notification service) that performs the work of acquiring the user and account data from the corresponding services, compiles the information for the message service and instructs it to actually send the messages. This introduces another "higher level, business purpose entity/service" but allows you to keep a clear separation of concerns.
In general it will happen frequently that your "basic" domain services are used by another service that represents a specific business purpose and requires their data. Dependency in itself is not a bad thing as long as concerns are seperated clearly and interfaces versioned, changes communicated etc.
Don't forget the whole idea of microservices is for allowing teams to have dedicated responsibilities with clear interfacing. It is about organization as much as it is about architecture.
We are evaluating a move to microservices. Each microservice would be its own project developed in isolation. During planning, we have determined that some of the microservices will communicate with other via REST calls, pub/sub, messaging (ie. a order service needs product information from product service).
If a microservice depends on retrieving data from another microservice, how can it be run in isolation during development? For example, what happens when your order service requests product details, but there is nothing to answer that request?
What you probably need is an stub rest service. Create a webapp that takes the expected output using a path that is not part of the public api. When you invoke the public api it sends what it just received
If a microservice depends on retrieving data from another microservice, how can it be run in isolation during development?
It should be always temporally isolated from other services during development and production as well.
For example, what happens when your order service requests product details, but there is nothing to answer that request?
This is a place where design flaw reveals itself: order service should not request product details from another service. Product details should be stored in the message (event) that order service will be subscribed to. Order service should be getting this message in an asynchronous manner using publish-subscribe pattern and saving it in its own database. Data about the product will be stored in 2 places as the result of that.
Please consider reading this series of articles about microservices for more details. But in a nutshell: your services should be temporally decoupled, so when your product service is down - order service can continue its operations without interruptions. This is the key thing to understand about good distributed systems design in general.