I'm fairly new to the Serverless Framework but I was able to achieve a lot of the bits and pieces of our aspired infrastructure using the framework and the serverless-lift plugin.
The serverless-lift plugin allows me to easily create worker queues with workers and DLQs for each of them. That's exactly what we need as it allows us to redrive the failed jobs. See this great article https://medium.com/serverless-transformation/serverless-queues-and-workers-designing-lift-d870afdba867 (under "A production-ready approach")
The Serverless Framework can create Lambda functions triggered by the EventBridge - that's also amazing.
However, as seen in the diagram we'd like the EventBridge to put the events - according to rules - into the respective queues. That bit seems to be missing or I haven't found it yet.
My question now:
Can I, and if so how can I, have the EventBridge be the central piece in our infrastructure, serving events to the worker queues?
From what I read I'd be able to define resources via CloudFormation notation (see below). Since Serverless is so much simpler and can easily be handled by our developers, I'd like to avoid using CF if possible.
Another approach could be to have intermediate Lambda functions putting the event into the queues but it's adding unnecessary complexity.
Thanks in advance!
For those interested, here's the CF code:
# serverless-list constructs
constructs:
my_queue:
type: queue
worker:
name: my-dev-my_queue_worker
handler: src/functions/hello-dlq/handler.main
# CloudFormation
resources:
Resources:
EventBridge:
Type: AWS::Events::EventBus
Properties:
Name: my-dev-event-bus
EventBridgeSQSRule:
Type: AWS::Events::Rule
DependsOn:
- EventBridge
Properties:
Name: my-dev-event-bus-rule1
Description: Test Event Bus
EventBusName: my-dev-event-bus
EventPattern:
source:
- test.v1
State: ENABLED
Targets:
- Arn:
Fn::GetAtt:
- ${construct:my_queue.queueArn}
- Arn
Id: SQSQueueTestV1
EventBridgeSQSSendPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: sqs:SendMessage
Resource: arn:aws:sqs:${aws:region}:${aws:accountId}:*
Queues:
- ${construct:my_queue.queueUrl}
Related
A SQS triggered Lambda is configured like this in a SAM template:
MyFunction:
Type: AWS::Serverless::Function
Properties:
Handler: ....
CodeUri: ....
Policies:
- AWSXrayWriteOnlyAccess
- AWSLambdaBasicExecutionRole
- SQSPollerPolicy:
QueueName: !GetAtt MyQueue.QueueName
Events:
TheSQSEvent:
Type: SQS
Properties:
Queue: !GetAtt MyQueue.Arn
I'm expecting this to create an execution role to which only 3 policies are attached. However, when created, the role also has the AWSLambdaSQSQueueExecutionRole attached to it.
How do I prevent this extra policy from being attached? It's undesirable since, for example, it grants sqs:DeleteMessage to all resources ("Resource": "*").
Update
This seems like a bug to me (since it violates the principal of least privilege). AWS team, on the other hand, stated that this is expected and they might introduce a feature to address it (https://github.com/aws/serverless-application-model/issues/2118).
i have the following lambda configuration:
MyFunc:
handler: my_handler
timeout: 60
role: myrole
events:
- stream:
type: dynamodb
arn: <<dynamo_db_stream_arn>
startingPosition: LATEST
maximumRetryAttempts: 3
destinations:
onFailure: <sqs_queue_arn>
enabled: True
Yet, when deploying, i don't see that the onFailure is even rendered in the cloudformation template.
i've set it up as said in this documentation:
https://serverless.com/framework/docs/providers/aws/events/streams/
Any idea what i'm missing?
==========================
So, completing Snickers3192's answer - I actually am not sure what's wrong with the configuration above, as serverless should support it, but eventually what i did is created the stream handler in another resource, so basically my serverless looks like that:
functions:
MyFunc:
handler: my_handler
timeout: 60
role: myrole
resources:
Resources:
MySourceMapping:
Type: AWS::Lambda::EventSourceMapping
DependsOn: MyFuncLambdaFunction
Properties:
EventSourceArn: <dynamo_db_stream_arn>
FunctionName: MyFunc
MaximumRetryAttempts: 3
StartingPosition: LATEST
DestinationConfig:
OnFailure:
Destination: <sqs_queue_qrn>
I think you're just missing "arn:"
Here's what worked for us.
maximumRetryAttempts: 10
maximumRecordAgeInSeconds: 300
bisectBatchOnFunctionError: true
destinations:
onFailure:
arn:
Fn::GetAtt:
- fileReducerDeadLetterQueue
- Arn
type: sqs
Even though I like serverless framework, I don't recommend using it for anything other than developing Lambda functions, I wouldn't even use the http event for creating an API gateway. Stick to the unix philosophy do one thing good, that's how I feel serverless should stick to that, not try and become another terraform or something, it isn't.
So create your Lambda functions in serverless and that's it. Do the other stuff else where. If the resource can be managed effectively in Cloudformation AWS::Lambda::EventSourceMapping, then you can use that. If it makes sense to put it at the bottom of the serverless.yml in resources: you can do that, but if not then let it have it's own template.
There is quite a number of permissions needed for setting up your lambda for DynamoDB streams, I wouldn't trust serverless to do that for you. A proper AWS prod setup as well might not let some external tool create iam roles as well.
As soon as you differ the slightest little bit from the serverless default cloudformation template, you'll have problems, probably you are spending hours right now, on a tool which was supposed to save you time, therefore defeating it's purpose. I suggest making more stacks than less, and using conventions when one stack needs a Lambda in another, this is actually more manegable as when one thing fails you can still update other stacks, and swap stacks as you change, you can't do that if you stick it all in a serverless.yml.
I'm using Serverless Framework and have multiple services which are attempting to use the same SQS queue. I can successfully make the resource in the first service but the second one is missing the lambda trigger when deployed to AWS. Hardcoding the ARN ID will successfully make the trigger so I can only assume I have something wrong with my syntax/indentation, but it's very similar to how I'm exporting/importing my API Gateway details and I'm just not seeing it.
I have an SQS Queue set up and exported from my first service like this:
resources:
- Resources:
InitializeAuthenticationQueue:
Type: "AWS::SQS::Queue"
Properties:
QueueName: "InitializeAuthenticationQueue"
- Outputs:
InitializeAuthenticationQueueArnId:
Value:
Fn::GetAtt:
- InitializeAuthenticationQueue
- Arn
Export:
Name: ${self:provider.stage}-InitializeAuthenticationQueueQueueArnId
In my second service I am attempting to use the SQS ARN ID as a trigger for a function, like this:
functions:
authenticationIntialize:
handler: myHandlerFile.myHandler
events:
- sqs:
arn:
'Fn::ImportValue': ${self:provider.stage}-InitializeAuthenticationQueueArnId
I've also tried this to see if I have my indentation wrong:
functions:
authenticationIntialize:
handler: myHandlerFile.myHandler
events:
- sqs:
arn:
'Fn::ImportValue': ${self:provider.stage}-InitializeAuthenticationQueueArnId
Feel like I'm missing something obvious on this one, but I've been stuck on it way too long. Anyone able to help me spot the obvious?
What errors do you get? What does the generated .serverless/cloudformation-template-update-stack.json have for the export and import values?
I usually find it easier to use the internal Serverless CloudFormation property reference. So where you are trying to import the SQS ARN do this:
${cf:STACK_NAME.InitializeAuthenticationQueueArnId}
where STACK_NAME is the name of the CloudFormation stack generated by the Serverless deployment this has the SQS ARN output. Using this method the way you reference the value to import is via the CloudFormation key and not the export name (which is admittedly confusing)
I am creating a AWS serverless application with SAM. Basically what I would like to achieve is to use API Gateway's different stages (dev/test/prod) to invoke various Lambda functions alias (dev/test/prod).
I am totally stucked, I would like to know what are the strategies people have taken to shift lambda traffics, eg. from LambdaA:dev to LambdaA:prod?
I have tried to use "AutoPublishAlias", but in SAM AutoPublishAlias you can't have more then one alias in a single cloudformation stack, so that makes traffic shifting impossible.
Before using a single stack, I have also used Canary Deployment, it works ok when I separate lambda into multiple envrs (ie. dev-lambaA, test-lambdaA, prod-lambdaA) managed by different cloudformation stack. But I would like to reduce the number of lambda functions by only have lambdas reside in a single stack.
What you can do is add the following to your template.yaml file:
Resources:
ProductionAPI:
Type: AWS::Serverless::Api
Properties:
StageName: PRD
DefinitionUri: ./prdswagger.yaml
DevelopmentAPI:
Type: AWS::Serverless::Api
Properties:
StageName: DEV
DefinitionUri: ./devswagger.yaml
And use the swagger files to create your endpoints. At every endpoint add an x-amazon-apigateway-integration to the correct lambda version that you are targeting.
x-amazon-apigateway-integration:
httpMethod: "POST"
type: aws_proxy
uri: "arn:aws:apigateway:eu-central-1:lambda:path/2015-03-31/functions/arn:aws:lambda:eu-central-1:[account_nr]:function:[myfunctionname]:PRD/invocations"
passthroughBehavior: "when_no_match"
I'm using AWS CloudFormation to build a stack for a microservice. My AWS CloudFormation template creates resources like: a Lambda function, an SNS topic and API Gateway.
This microservice does some work and publishes messages to the SNS topic. Other microservices subscribe to this topic.
The problem I'm facing is that when I upgrade my microservice's CloudFormation template (sometimes I need to redeploy it, and recreate all resources), the SNS topic changes its ARN. Hence, all microservices that use this topic need to change as well.
I think I could create a separate CloudFormation template for the SNS topic (I have more than one per microservice).
Will this be a good approach?
If not, what's the recommended way?
As per AWS CloudFormation docs, there are 3 properties available for an SNS topic, and only a change to the third property (i.e. TopicName) would lead to a replacement of an SNS resource (and leading to a generation of a new ARN).
Since you're nuking the resources in CloudFormation stack and then re-creating them (you mentioned that that's how the serverless framework you're using works), you can preserve the SNS topic ARN by specifying the TopicName attribute for the SNS topic in your template (it's only when one doesn't specify a TopicName is when CloudFormation comes up and assigns a random name, otherwise it uses the name that the user specifies).
That being said, your scenario does seem a little unnatural to me. If you aren't already managing subscriptions to that topic via CloudFormation, I'd recommend doing that, and it should automatically create SNS subscriptions on the replaced SNS topic for you, if at all SNS topic ARN changes.
Ref:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sns-topic.html
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-sns-subscription.html
#p.magalhaes - Ideally you should not be deleting the stack and creating it again, rather you should be updating the cloudformation stack. And if you update your stack it wont delete SNS topic as there won't be any change in that. However if you have a specific need of deleting and creating the stack you can create separate stacks for all of your components using nested stack (Reference - https://aws.amazon.com/blogs/devops/use-nested-stacks-to-create-reusable-templates-and-support-role-specialization/ )
If you need to dynamically link to your SNS topic's ARN, you should create an environmental variable within your Lambda resource and reference your topic's ARN, like so:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
MyTopic:
Type: AWS::SNS::Topic
MyLambdaFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.handler
Runtime: nodejs8.10
Code: ./path/to/index.js
Role: [Role ARN here]
MemorySize: 128
Timeout: 3
Environment:
Variables:
SNS_TOPIC: !Ref MyTopic
If multiple stacks need to access your SNS Topic, I suggest creating a separate stack for your SNS topic and then putting the topic ARN in other templates as a parameter, and similarly reference that parameter, like so:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
MyTopicArn:
Type: String
Resources:
MyLambdaFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.handler
Runtime: nodejs8.10
Code: ./path/to/index.js
Role: [Role ARN here]
MemorySize: 128
Timeout: 3
Environment:
Variables:
SNS_TOPIC: !Ref MyTopicArn
This way, if you have to replace your SNS topic, you can just update the other stacks' parameters when you deploy them next.