As part of my project, I'd like to use microservices. The application is a store website where the admin can add products and the user can order and buy them.
I envision implementing four services: admin service, user service, product service, and order service.
I had trouble with handling data between multi services but it's solved by duplicating some necessary data using message brokers.
I can do this solution between product and user and order service because I need some of the data not all of them
Now, my question is about handling admin service because in this service I need to access all of the data, for example, the admin should have a list of users and the ability to add new products or update them.
how can I handle data between these services and the admin service?
should I duplicate all data inside the admin service?
should I use Rest API?
no thats wrong. it seems you want run away from the fact. in general duplication is an anti-pattern mostly in case you describe.
the way you thinking about admin-service is wrong.
because in this service I need to access all of the data
i dont think you need to have such a service. accessing the data based on users must be handled by Identity server(oidc Oauth) which is the separated service and handle the accessing end points .
for example the product-service provides 1-return product list 2-return individual product data 3-create data. the first two can access by both user and admin but the 3rd must be accessed by admin. one of identity server duty is to identify user in case of user interaction(login) with services.
ADMIN Scenario
user-client request create product endpoint(services eg:product.service).
client-app(front end app) is configed with identity server and realize there is no require identity tokens and redirect to identity server login.
NOTE: there is also identifying the client-app itself i skipped.
user-client login and get require token that based on his claims and roles and etc.
user-client request create product endpoint with tokens included in request header
endpoint (product service) receives the request and check the header (the services also configured base on identity server and user claims)
get the user claims info.
the create-product requires admin role if its there then there we go otherwise no access.
the image uses identity server 4 . there are also several kinds and also you can implement by your self using 0AUTH and oidc protocol libraries.
so the admin just request to the certain service not getting data through the separate service for this goal.
Communication between Service:
the most struggling part of microservices is the wiring it up. the wiring is directly the consequence of your design.(recommand deep study on Domain Driven Design).
asynchronous communication :
to avoid coupling between services mostly use asynchronous communication which you pass event eg:brokers like rabbitmq and kafka..etc , redis etc. in this communication the source service who send event does not care about response and not wait for it.just it always ready to listen for any result event. for example
the inventory service creates item
123|shoe-x22|22units
and this service fire event with data 123|shoe-x22(duplicate maybe or maybe not just id) to product service to create but it does not wait for response from product service that is it created successfully or not.
as you see this scenario is unreliable in case of fault and you need handle that so in this case you have to study CAP theory,SAGA,Circuit-breaker.
synchronous communication :
in this case the service insist to have response back immediately. this push service to become more coupling. if you need performance then you can use gRPC communication other wise simple api call to the certain service. in case of gRPC i recommand using libraries like MassTransit
which also can be used for implementingf gRPC with minimum coupling.
Some of Requests need data from multiple services
if you are in such situation you have two options.
mostly microservices architecture using APIGATE WAY (EG:nginx,OCELOT,etc)
which provide reverse-proxy,load balancing,ssl terminations etc. one of its ability is to merge the multiple responses from a request.but it just merge them not changing the data structure of response.
in case of returns desire response data structure you may create an Aggregator service which itself calls other two, gathers data and wrap it in desire format and return it.
so in the end still the Domain Driven Design is the key and i think i talked tooooo much. hope help you out there.
Related
I plan to set up a set of microservices with an API gateway, I am new to microservices architecture but the services I plan to add more services and keep this application highly extensible. The API gateway should manage the users and their permissions and should delegate the incoming requests to the underlying microservices. But my problem is, how can I create a relationship between the user at the gateway and an entity in a microservice.
Like in the picture above i need to figure out what is the best practice to deal with user relations in the underlying services. I want to implement all the services with laravel the gateway should use laravel\passport.
My thought was that the API gateway is responsible for authenticating the users and forwarding requests to the services behind the gateway. If the user is authenticated, he has access to the services through the gateway. But how can I provide the service with the information about the user, for example, if the user edits an item in service A, how can I store which user edited the item. What would be the approach to establish this relationship?
There are many aspects to consider when selecting an approach, so basically answering your question will mostly be giving you pointers that you can research deeper on.
Here are some approaches you should review that will greatly depend on your service:
Authentication/Authorization method for the platform as a whole
How each individual service talks to each other (sync REST calls, messaging, GraphQL, GRPC, ...)
How are individual service's secured (each service is public and does auth, every service is behind a secured network and only the gateway is public, service mesh takes care of auth, ...)
The most common auth method in REST based microservices is OAuth, with JWT tokens. I recommend that you look deeper into that.
(Now just digressing a bit to demonstrate how much this varies depending on the use case and architecture)
Taking OAuth and looking at your question, you still have different flows in OAuth that you will use according to the use case. For example, generating tokens for users will be different than for services.
Then you still need to decide which token to use in each service: will the services behind the gateway accept user tokens, or only service-to-service tokens? This has implications to the architecture that you need to evaluate.
When using user tokens you can encode the user ID in the token, and extract it from there. But if you use user tokens everywhere, then it assumes services only talk to each other as part of a user flow, and you are enforcing that through the use of a user token.
If you go with service-to-service tokens (a more common approach, I'd say) you need to pass the user ID some other way (again, this depends your chosen architecture). Thinking of REST, you can use the Headers, Request Params, Request Path, Request Body. You need to evaluate the trade-offs for each depending on the business domain of each service, which influences the API design.
If you don't use tokens at all because all your services are inside a secured network, then you still have to use some aspect of your protocol to pass the user ID (headers, parameters, etc...)
Soon I'll start a project based on a Microservice Architecture and ones of the components I need to develop is a Worker Service (or Daemon).
I have some conceptual questions about this.
I need to create a worker service that send emails and sms. This worker service need the data to send this emails. Also, I need to create a micro service that allow users to create a list of emails that need to be sanded by this Worker service. But both of then need to consume data from the same database.
In my worker service I should consume a micro service resource to get the data or it's ok that this worker service have a connection to the same database that my micro service?
Or is best that my worker service also has the api endpoints to let the users create new lists of emails, add or modify configuration and all the other functionalities i need to implement? This sound like a good idea, but I'll get a component with two responsibilities, so I have some doubts about that.
Thanks in advance.
Two microservices sharing the connection to the same database is usually a bad idea. Because each service should be the owner of its own data model and no one else should access it directly. If a service needs data of the domain of another service it should get it calling the owner via API or replicating the model in a read-only way in its own dabase and update it using events for example.
However, I think that for your current use case the best option is to provide the worker with all the information that it needs to send an email, (address, subject, body, attached files...) so the only responsibility of the worker will be to send emails and not to fetch the information.
It could provide also the functionality to send emails in batches. In the end, the responsibility of the service will be only one "To send emails" but it can provide different ways to do it (single emails, batches, with attached files... etc)
Suppose that I have a microservice for messaging. The microservice knows how to send emails. The service have templates of emails that have some sort of "template engine" like pugjs, and can replace data in the body of the message.
I have an user service (used for authentication/authorization for example), and a bank account service (each user have one). Between the User microservice and Bank Account microservice it's clear that we don't have to duplicate any data than de user's uuid.
But I want now to send every day a message to each user with their account statement. The Messaging microservice needs data from the User microservice and the Bank Account microservice.
Okay... This is a small case of the real world. Now I know that to have the benefits of decoupled microservices I must follow some rules:
I can't share databases between microservices
I can't make synchronous requests between microservices
Okay... I can use a broker and each time a new user is created/updated the Messaging microservice can store that data. But really, this is a stupid thing:
I don't want to have inconsistency with this data, and keeping things sync is hard
The development time and complexity of the Messaging Microservice must now consider: listen and extract the relevant data from events, keep data consistent about other domains/services, managing the saved data on database
And think about a Messaging microservice. Really I must store all the data needed to parse the templates?
I read a lot about microservices and people creating rules for their simple examples. But I never really saw a good explanation and real-world examples like above.
So how to have the microservices above without data duplication?
In your domain example I would not let the message service know anything about bank or user details. Instead the message service should just receive instructions to send messages to recipients along with the given content. I would use a dedicated scheduled job (maybe implemented as an account notification service) that performs the work of acquiring the user and account data from the corresponding services, compiles the information for the message service and instructs it to actually send the messages. This introduces another "higher level, business purpose entity/service" but allows you to keep a clear separation of concerns.
In general it will happen frequently that your "basic" domain services are used by another service that represents a specific business purpose and requires their data. Dependency in itself is not a bad thing as long as concerns are seperated clearly and interfaces versioned, changes communicated etc.
Don't forget the whole idea of microservices is for allowing teams to have dedicated responsibilities with clear interfacing. It is about organization as much as it is about architecture.
I am currently reading a lot about microservices but still, I don't understand some parts. I made the following draw:
Each microservice has 2 accesses:
REST: For http uses
gRPC: For intra/background communication/exchanges
If I want to login I can just send an Http Request to my Authentication service. But what about if I want to access the Stuff service that needs you to be already connected?
Let say that the user wants to display the stuff available in the database STUFF, the service Stuff will first check if the "token" of the connected user is right, by exchanging with the Authentication service, and then return the stuff or a "login requires request".
So the thing I don't understand is, if each services that needs a client already connected needs to exchange with Authentication, then it will create a huge internet traffic in order to check each user request.. So I though about make one Authentication service per service, but since I should have only one Database, then it's the database that will slow the traffic?
Also, if I understand, each micro service should be on separate servers, not the same one?
I hope I am clear, don't hesitate to ask for more details !
Thanks in advance :)
Max
Edit 1
Based on #notionquest's answer:
So it should more looks like that right?
Also, based on Peter's comment, each service can implement its own middleware (JWT as mentioned) so the API Gateway is only a "pass-through". However, I don't feel like it could be a nice for me since each service make a token check for each internal exchange, doesn't it?
For the stuff, it's easy since it checks only 1 time the token. Now, let's say that, after the user got the stuff, he choose one and wanna buy it. Then the "Buying service" will call the stuff service in order the verify the price of the item, but... It will have to check the user token since the stuff is a "on authenticated access", so it means that "Buying" service and "Stuff" service both check the token, which add an extra check.
I though about an internal guaranteed access between services but is it worth it?
Also, maybe you said to implement the middleware for each service since they have a REST access, but the API Gateway would just destroy the idea of having REST access
There are multiple solutions available for this problem. One of the solution is API Gateway pattern.
First request goes to API gateway
API Gateway authenticates & authroizes the request
Authentication is stored on cache database such as Redis, Memcache etc with expiry time on it
Saved access token is returned to client
Client can use the saved access token in the subsequent calls for the some time span (i.e. until the token is valid)
Once the token is expired, the API gateway will authenticate and share the new token to client
This solution will reduce the need to authenticate each request and improves the performance
API Gateway is the single entry point for all the services. So, you may not need separate cache for each service.
Refer the diagram in this page.
Apart from #notionquest answer, there is another approach which does not involve having an API gateway;
You can share a SESSION_SECRET among all your services, so the only task of your Authentication Service is to validate username and password against the database and then encrypt this information using SESSION_SECRET and return a jwt token. All other services won't need to interact with Authentication Service but simply check if the jwt token is valid (can be decrypted) with the SESSION_SECRET.
You then have two other options;
Store all user data you need in the token - this will increase the amount of data in transit from your client to the micro-services. This can be prohibitive depending on the size of this information
You can store only the userId, and request additional data as needed per each micro service, which depending on how often/how big your data is will generate a problem as you described.
Note that you will not always be able to use this approach but depending on your specific scenario and requirements having this architecture in mind can be useful.
Also keep in mind that rotating the SESSION_SECRET can be tricky (although necessary for security reasons). AWS has just released a service called AWS Secrets Manager, so one idea to make things simple would be to have your micro-services periodically query a service like this for the current valid SESSION_SECRET instead of having this values hardcoded or as environment variables.
We are evaluating a move to microservices. Each microservice would be its own project developed in isolation. During planning, we have determined that some of the microservices will communicate with other via REST calls, pub/sub, messaging (ie. a order service needs product information from product service).
If a microservice depends on retrieving data from another microservice, how can it be run in isolation during development? For example, what happens when your order service requests product details, but there is nothing to answer that request?
What you probably need is an stub rest service. Create a webapp that takes the expected output using a path that is not part of the public api. When you invoke the public api it sends what it just received
If a microservice depends on retrieving data from another microservice, how can it be run in isolation during development?
It should be always temporally isolated from other services during development and production as well.
For example, what happens when your order service requests product details, but there is nothing to answer that request?
This is a place where design flaw reveals itself: order service should not request product details from another service. Product details should be stored in the message (event) that order service will be subscribed to. Order service should be getting this message in an asynchronous manner using publish-subscribe pattern and saving it in its own database. Data about the product will be stored in 2 places as the result of that.
Please consider reading this series of articles about microservices for more details. But in a nutshell: your services should be temporally decoupled, so when your product service is down - order service can continue its operations without interruptions. This is the key thing to understand about good distributed systems design in general.