I'm trying to send error messages from Lambda to DLQ. I can see error messages in cloudwatch logs but unable to see any entry in DLQ. I followed below steps-
1- Created Lambda Functions.
2- Added API gateway as trigger.
3- Created a queue in SQS as standard.
4- Assigned SQS full access policy to Lambda Role.
5- Configured Asynchronous invocation in Lambda Function and defined my queue which i created in step 3.
Do I need to do anything additional on API gateway or Lambda.
Related
I have two AWS accounts, we'll call them A and B.
Account A hosts a rabbitMQ broker in AmazonMQ.
Account B has a lambda function that performs some actions and ultimately needs to send some messages to the queues in account A.
Currently, some consoles outside of AWS are sending messages to the queues, for security reasons Account A uses a subnet with specific ACL rules allowing only traffic from these consoles to reach the queues.
I'm unsure how to allow this lambda to send messages to the queues in account A, while this lambda must continue to live in account B. The lambda connects via MassTransit and the amqp endpoint, the same way the non-AWS consoles connect.
If I create a rule allowing all traffic I am able to send messages successfully. so I'm fairly certain the problem lies somewhere in these ACL rules. I've tried putting my lambda into a VPC and whitelisting its CIDR blocks in the ACL inbound rules with no success. I also tried a peering connection between the two accounts to no success.
is it possible to do a post processing (besides transformation) of response in API Gateway?
For instance POST call to API Gateway goes to my Rest API and after response is return, API Gateway publishes an event (call lambda or SNS in the background).
Thanks
At present, there is no direct option for this in Api-Gateway. However, you can create a JSON message and push it to SNS topic which can call AWS Lambda or can be pushed into SQS for later process. Pushing into SNS can be done using AWS SDK.
I want to use AWS SQS for communication between my microservices (and later possibly SNS). Each microservice can have multiple instances up.
Currently I'm trying to implement the Request/Response pattern of message queues.
As I understand it, the normal way is to have one request queue, and pass a unique response queue per service instance.
The consuming service will process the message and send the response to the given response queue. Thus, the response will always be returned to the correct instance of the requesting service.
My problem now comes with Cloudfoundry.
How it should work:
Service A needs to request data from Service B.
There is one queue named A-request-B.
Service A starts with 6 instances.
Every instance creates its own queue: B-response-A-instance[x]
Every request from an instance of A sends their response queue name in the request so response is routed to the correct queue.
This is the only way I know to guarantee that the response from B gets to the correct instance of A.
This doesn't work as Cloudfoundry doesn't allow the "create-queue" call from SQS, even if I can connect to the SQS instance to send and receive messages.
The only way to create a queue is via the command line.
So I would have to create these 6 response-queues manually beforehand.
And if I start a 7th instance of A, it will fail as it doesn't have its own response queue.
I also tried using the SQS temporary queues, but they also work by creating queues dynamically which is not possible in Cloudfoundry.
I'm currently stuck with SQS, so switching to kafka/rabbitmq or something else is not possible.
Is there any other way to pass a response to the matching service instance? Or is there another way to create queues in cloud foundry?
Summary from comments above...
This doesn't work as Cloudfoundry doesn't allow the "create-queue" call from SQS
Cloud Foundry doesn't really care what messaging system you're using, unless you're using a Marketplace service to create it. In that case, Cloud Foundry will work on your behalf to create a service instance. It does this by talking to a service broker, which does the actual creation of the service instance and user credentials.
In your case, Cloud foundry handles creating the credentials to the AWS SQS through the AWS Service Broker. Unfortunately, the credentials the broker gives you don't have the permission to create queues. The creds are only allowed to send and receive messages for the specific queue that was created by the broker.
There's not a lot you can do about this, but there's a couple options:
Don't use the Marketplace service. Instead, just go to AWS directly, create an IAM user, create your SQS resources, and give the IAM user permissions to them.
Then create a user provided service with the credentials and information for the resources you created. You can bind the user provided service to your apps just like a service created by the AWS Service broker. You'd lose the convenience of using the broker, but you won't have to jump through the hoops you listed when scaling up/down your app instances.
You could create a service instance through the broker, then create a service key. The service key is a long-lived set of credentials so you could then go into AWS, look up the IAM user associated with that service key and adjust the permissions so that you can create queues.
You would then need to create a user provided service, like the first option, insert the credentials and information for your service key and bind the user provided service to any apps that you'd like to use that service.
Don't delete the service key, or your modified user will be removed and your user provided service will stop working.
Hope that helps!
Per link! here, Azure Functions Service Bus trigger lets you listen on Azure Service Bus. We are currently using mostly AWS for our cloud services. And we are working with vendor who has real time notifications using Azure service bus. I would like to know if there is anyway to connect to service bus using lambda. Anytime there is a new message on the bus, we would like our AWS lambda to invoke and take it from there.
It's not possible. However you can use Azure functions (Azure serverless offering) triggered by Azure Service bus to consume the messages.
If you really want cross vendor trigger then you need to consume azure service bus message, convert the message into http payload and trigger AWS lambda with Http payload that has message contents.
Cloudwatch Event Rule: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Rule.html
You specify your event source -- a supported Service and an action/API call and the targets and setting up the required IAM setting(Lambda permission etc. if you create from IaC tools like terraform..) And you are good to go!
Then as long as Cloudwatch event rule is up, all the events that falls into the rule you specify will trigger your lambda.
Event rule can also be used a "cron schedule" for lambda, which I have been using. I did encounter some delay very rarely tho.
Update: to make it as real time as possible, you would need to make modification to your vendor's Azure account to enable some message pushing to your AWS endpoint(an API gateway), which I assume is a NO. Other than that, a AWS self contained solution is to set up a cloudwatch event rule to pull your vendor's Azure HTTP endpoint every 1 minutes and store the stuff in your own SQS queues.
Currently I have a frontend service that makes calls to a notification service. The frontend service does not depend on the response from the notification service. I want the frontend service to make a call to the notification, but not wait for a response from the notification service. Could do this within the frontend service itself? Do I have to use some messaging service that can 'proxy' the calls between the two services?
You need a queue, the queue can be independent service (RabbitMq, Hornet, etc) o you can use a queue inside your notification service.
Independent Queue, in this case you can use a message broker in the middle of the services, the frontendservices send the message to the queue and the notification service read the message from the queue and then send it.
Internal Queue, you can have in your notification service an internal queue, like the queues in SpringIntegration. Whe the service receive a message inmediately put it on the queue and return an OK code. The frontendservice doesn't need to wait until the notification is send. Then the notification service process the messages in its queue on its own pace.
You could go for asynchronous execution. Just add #EnableAsync to your application class and annotate the method calling the notification service with #Async.