I am well aware of the fact that east/west, or service to service synchronous communication between services is not the gold standard, and should only be used sparingly in a microservice architecture. However, in every real world implementation of a microservice architecture, I have seen some use-cases which require it. For example, the user service is often needs to be communicated with by other services to get up the millisecond details on the user (I'm aware that event based sharing of that data is also a possibility, but in some cases that isn't always the right approach).
My question is, what is the best way to do function to function, service to service communication in a Lambda + API Gateway style architecture?
My guess is that making an http request back out on the domain name is not ideal, since it will require going back out over the internet to resolve DNS.
Is it using the SDK to do an invoke on the downstream function directly? Will this cause issue if the downstream function depends on an API Gateway Proxy Event structure?
Related
I've implemented two lambda's (let's call A and B) behind api gateway. Assume A is called from "outside" and B is being called from outside and also from A.
I've also implemented lambda Authorizer (token-based; cognito) as auth layer. Everything is working as expected.
Is there a way to bypass authorizer process for B, for calls coming from A only?
Tnx
There are multiple possibilities I have explored myself in the past for the exact same issue.
Change the calls to lambda:Invoke
Assuming you're generating some client code for your micro-services, you can create two versions of these clients:
external to call your service via HTTP API
internal to use lambda:Invoke operation straight to your micro-service.
Create a mirrored VPC-private API
This is probably feasible if you're deploying your infrastructure using CDK (or a similar alternative). Essentially, you keep your existing API where it is, and you create another internal version of it that does not have the authorizer. (Note that you may still want some sort of authorization process happening depending on the nature of your project.)
From this point on, you can pass the endpoint of your internal HTTP API to the Lambdas as environment variables and have them call that.
You can find more info about this, here. As a perk you should probably get lower latencies when talking to API Gateway as traffic through the VPC endpoints will only flow through AWS network, instead of going out on the internet and back in.
Move your workloads to ECS
This is perhaps a major change to your project, but one worth mentioning.
You can create true micro-services using ECS. You can run these services in private subnets of your VPC. In order not to have to deal with IP addresses yourself, you can explore multiple options:
have a VPC-internal Route53 Hosted Zone (more on this here). See more on ECS Service Discovery here
create Network Load Balancers in the private subnets of your VPCs and pass their endpoints to your services.
If my company sole purpose is processing a specific payload, but there is lot of orchestration for it. Should the orchestration, be in a separate domain. Lets say, payment is what the company does, but there is a workflow service, for that payment payload? If that is in a seperate domain, how should the workflow service domain talk to payment service domain?
It's better to use Event Driven Design which powered by message services like RabitMQ (or Kafka, MSMQ, or ..). It's not recommended to speech microservice each other directly via APIs. On the other hand to aggregate, some information from multiple services you can use 2 techniques, first using a BFF (back end for frontend layer), Second use a materialized view to gather information from many services.
I m building a spring boot microservices, and i have some questions
I have an account microservice, a payment microservice, a product microservices... in these microservices, some requests sometimes need to use a mailing api, an sms sending api, or a push notification api..
What i have done now is create a microservice for mailing, microservice for sending sms and microservice for push notification.
What i can't seem to solve is how to make these microservices used only internally. for example, forbid users to directly call the mailing microservice.
before creating this question on stackoverflow, i dud myself, why i'll not put the code for sending sms in a library, and the same for sending emails and push notifications and add them to the microservice .. and when a microservice has need to use one of these apis i add the needed library .. for example i create a push notification library, and i add it to each microservice that needs to do a push notification ..
what is the best approach to integrate these mailing, sms and notification services into my microservice project, and respecting security by forbidding users to use them directly
I don't know what to do, can someone advise me?
Well it is not exactly clear to me what do you mean by "forbidding users to use them directly" but usually as it is pointed out #kavhakaran's answer you should put the security measures to prevent your services from abuses.
In that answer only network related part is focused as far as I can see. There should also be a second level which is about user authorization. That means you can/should have proper roles and authorization definitions for the services you would like to secure. And based on provided roles you can authorize the client to use the services.
That is how it works for cloud services usually as well. You will be provided an api-key in order to consume some cloud service and they will check if the api-key is authorized for the requested service etc.
You shouldn't worry about other micro services calling the mailing microservice or sms microservice in the application code. If you think about this concern, this will apply to any internal mircoservice. This concern can be handled in infrastructure level
Let me give you an example, you have a database running somewhere, does your microservice does anything to make sure, it is the only one talking to that database. The answer is no. At infrastructure level, whatever cloud infrastructure you are using, they allow to define security rules/ network policies, that lets you define who can talk to who. ie. rules for incoming traffic and rules for outgoing traffic
If they are public facing microservices, that is a different question. These are internal services
Some examples based on infrastructure
AWS SecurityGroups
AWS subnets
Kubernetes Network Policy
And also I want to add a point which may not be directly related to your question. The services in question seems to be very good candidates as asynchronous services. Then no services talk to them directly, sending services put the notifications in queue or kafka topic and these services consume from the topic. So now it is making sure only relevant services send it to queue or topic at network level
I would not recommend to use libraries for sending sms, emails and push notifications across your Microservices. This would lead to dependencies on source code level which I would try to avoid in a Microservices architecture if possible.
Concerning the architectural issues of your question:
From my experience it is a good idea to have separate services for handling notifications such as sms, email, etc. because with that you create an abstraction between your Microservices and the concrete notification infrastructure such as third party sms, email or push notification services.
Usually the core requirements to, for instance, sending an email will more or less be the same over time. But you might come into a situation where you want to exchange one third party service for another - for instance due to cost concerns, performance concerns or other reasons.
If you choose to directly communicate with the notification infrastructure from each Microservice that needs to send emails you would have to adapt all these Microservices when you switch from one email service to another, no matter if you use a shared library or each Microservices implements the communication with that service on its own.
But if you have a separate Email Microservice that is used by all your Microservices that need to send email notifications, you only have to change the Email Microservice itself to communicate with, for instance SendGrid instead of MailJet (just to name two third-party Email services). Your other Microservices aren't even concerned with that change.
Concerning the security aspects:
As it was already mentioned, if you choose to communicate with your notification services asynchronously the security aspects will be addressed on the infrastructure level by allowing the Microservices to access messaging infrastructure based on the authentication and access control mechanisms provided by the corresponding messaging services (be it RabbitMQ, Azure Service Bus, Kafka, AWS SQS, etc.)
Or if you choose to call your notification services via REST APIs from your Microservices you can look into token-based authentication via OpenID Connect (e.g. via Client credentials flow for machine-to-machine security).
One other thing to consider:
I would also think about other shared functionality that could be common to sms, email and push notification services such as user preferences - e.g. which kinds of notification does a user want to receive. This could also be some functionality you do not want all of your Microservices have to know about. So you could think of a notification service that is concernced with this kind of responsibility and would be responsible to delivery the notifications over the different kinds of channels (email, sms, push) based on the user prferences. Or you could have separate Microservice for user preferences which is than accessed by your sms, email and push notification Microservices. But there is no obvious answer to which option is better because this strongly depends on the use cases you have to deal with.
We are evaluating a move to microservices. Each microservice would be its own project developed in isolation. During planning, we have determined that some of the microservices will communicate with other via REST calls, pub/sub, messaging (ie. a order service needs product information from product service).
If a microservice depends on retrieving data from another microservice, how can it be run in isolation during development? For example, what happens when your order service requests product details, but there is nothing to answer that request?
What you probably need is an stub rest service. Create a webapp that takes the expected output using a path that is not part of the public api. When you invoke the public api it sends what it just received
If a microservice depends on retrieving data from another microservice, how can it be run in isolation during development?
It should be always temporally isolated from other services during development and production as well.
For example, what happens when your order service requests product details, but there is nothing to answer that request?
This is a place where design flaw reveals itself: order service should not request product details from another service. Product details should be stored in the message (event) that order service will be subscribed to. Order service should be getting this message in an asynchronous manner using publish-subscribe pattern and saving it in its own database. Data about the product will be stored in 2 places as the result of that.
Please consider reading this series of articles about microservices for more details. But in a nutshell: your services should be temporally decoupled, so when your product service is down - order service can continue its operations without interruptions. This is the key thing to understand about good distributed systems design in general.
I have this webservice at work. For that webservice our department have developed a client to consume the webservice.
What we want to prevent is, that they develop any other client to consume it.
Is there any algorithm, practice that we can improve in our client and webservice communication to validate that the consuming client is our application?
I have an idea, that we can develop an encrypting algorithm about the time (5 seconds of grace) that must match with the calculated on server.
But i want to be sure, there is no best practice for that... or if it still a good idea...
(sorry about my english)
I think part of the point of web services (SOAP or REST based, for example) is to publish an interface that will let your service potentially be used by a variety of client implementations (interoperability is one of the motivations for web services).
If you want to lock your service to be used by your client, the only benefit of having it as a "web service" is probably the tools and libraries with which you've implemented it. You may want to consider whether this was worthwhile (it's possible indeed).
If you distribute your client application, chances are that whatever protection mechanism that may ensure the requests come from that client will have to be embedded with this client. Therefore whatever secret mechanism you embed will probably be only obfuscated to a certain point, but breakable by more advanced users.
What you are looking for is known as "authentication".
You need application authentication. For Web based application and services try looking into 2-legged OAuth. In OAuth You give out an id and a secret for every client app which accesses your service and every message is signed for extra security.