Listening to remote AWS SQS from local using serverless - aws-lambda

I want to execute the lambda function locally , on SQS event which is on my AWS account. I have defined the required event but this not getting triggered.
How can this be achieved?
I am able to send the messages to the same queue using cron event from my local.
Here are few things I tried... but didnt work for me .
functions:
account-data-delta-test:
handler: functions/test/data/dataDeltatestGenerator.handler
name: ${self:provider.stage}-account-data-delta-test
description: account delta update - ${self:provider.stage}-account-data-delta-test
tags:
Name: ${self:provider.stage}-account-data-delta-test
# keeping 5 minute function timeout just in case large volume of data.
timeout: 300
events:
- sqs:
arn:
Fn::GetAtt: [ testGenerationQueue, Arn ]
batchSize: 10
----------
Policies:
- PolicyName: ${self:provider.stage}-test-sqs-policy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- sqs:ReceiveMessage
- sqs:DeleteMessage
- sqs:GetQueueAttributes
- sqs:ChangeMessageVisibility
- sqs:SendMessage
- sqs:GetQueueUrl
- sqs:ListQueues
Resource: "*"
---------------
---
Resources:
testGenerationQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: ${self:provider.stage}-account-test-queue
VisibilityTimeout: 60
Tags:
-
Key: Name
Value: ${self:provider.stage}-account-test-queue
-------------
const sqs = new AWS.SQS({
region: process.env.REGION,
});
exports.handler = async (event) => {
console.error('------------ >>>>CRON:START: Test delta Job run.', event);
log.error('------------ >>>>CRON:START: Test delta Job run.', event);
});

You can't trigger your local Lambda function from your remote context because they haven't nothin in common.
I suppose your goal is to test the logic of Lambda function, if so you have two options.
Option 1
A faster way could be invoke function locally using sam local invoke. In this way, you could provide this command some argument, one of those arguments is the event source (i.e. the event information that SQS will send to the Lambda as soon this is triggered).
sam local invoke -e sqs.input.json account-data-delta-test
and your sqs.input.json would look like this (generate using sam local generate-event sqs receive-message)
so you will actually test your Lambda locally.
Pros: is fast
Cons: You still have to test the trigger when you will deploy on AWS
Option 2
In a second scenario you will sacrifice the bind between a queue and Lambda. You have to trigger your function at fix interval and explicitly use the ReceiveMessage in your code.
Pro: you can read a real message from a real queue.
Con: you have to invoke function at regular interval and this is not handy.

Related

Is it possible to set EventBridge ScheduleExpression value from SSM in Serverless

I want to schedule one lambda via AWS EventBridge. The issue is I want to read the number value used in ScheduledExpression from SSM variable GCHeartbeatInterval
Code I used is below
heartbeat-check:
handler: groupconsultation/heartbeatcheck.handler
description: ${self:custom.gitVersion}
timeout: 15
memorySize: 1536
package:
include:
- groupconsultation/heartbeatcheck.js
- shared/*
- newrelic-lambda-wrapper.js
events:
- eventBridge:
enabled: true
schedule: rate(2 minutes)
resources:
Resources:
GCHeartbeatInterval:
Type: AWS::SSM::Parameter
Properties:
Name: /${file(vars.js):values.environmentName}/lambda/HeartbeatInterval
Type: String
Value: 1
Description: value in minute. need to convert it to seconds/milliseconds
Is this possible to achieve in serverless.yml ?
Reason for reading it from SSM is, it's a heartbeat service and the same value will be used by FE to send a heartbeat in set interval. BE lambda needs to be triggerred after 2x heartbeat interval
It turns out it's not possible. Only solution to it was to pass the variable as a command line argument. something like below.
custom:
mySchedule: ${opt:mySchedule, 1} # Allow overrides from CLI
...
schedule: ${self:custom.mySchedule}
...
resources:
Resources:
GCHeartbeatInterval:
Type: AWS::SSM::Parameter
Properties:
Name: /${file(vars.js):values.environmentName}/lambda/HeartbeatInterval
Type: String
Value: ${self:custom.mySchedule}
With the other approach, if we make it work we still have to redeploy the application as we do need to redeploy in this case also.

Send message to sqs while catch error in stepfunctions

i am using serverless framework with serverless-step-functions plugin. I want to check any errors in my stepfunction workflow and send this error to sqs queue.
Currently I want to pass all input as message to the queue(MessageBody: $). But if I get the data from the queue, message is $ (dollar sign) and not actual input. How can I send to queue the error message from the previous step?
States:
state1:
Type: Task
Resource:
Fn::GetAtt: [function1, Arn]
Next: state2
Catch:
- ErrorEquals: [States.ALL]
Next: sendErrorToDLQ
ResultPath: $.error
state2:
Type: Task
Resource:
Fn::GetAtt: [function2, Arn]
Next: done
Catch:
- ErrorEquals: [ States.ALL ]
Next: sendErrorToDLQ
ResultPath: $.error
sendErrorToDLQ:
Type: Task
Resource: arn:aws:states:::sqs:sendMessage
Parameters:
QueueUrl:
Ref: ServiceDeadLetterQueue
MessageBody: $ # <== how to pass input to sqs message
Next: fail
fail:
Type: Fail
done:
Type: Succeed
I have got the same when connecting SNS. As per the AWS doc, we have to follow the below structure to send the parameters
"MessageBody.$": "$"
Reference: https://docs.aws.amazon.com/step-functions/latest/dg/connect-sqs.html

In CloudFormation, how do I target a Lambda alias in Events::Rule

I'm trying to trigger a Lambda:alias (the alias is key here) on a schedule. The following code errors out with
"SampleLambdaLiveAlias is not valid. Reason: Provided Arn is not in
correct format. (Service: AmazonCloudWatchEvents; Status Code: 400;
Error Code: ValidationException;"
How do I properly target the lambda:alias in CloudFormation? I've tried !Ref, !Sub and just the logical name.
My custom-resource approach to retrieving the latest lambda version appears to be a necessary evil of setting up the "live" alias because AWS maintains old lambda versions, even after you delete the lambda and stack AND a valid version is required for a new alias. If anyone knows a more elegant approach to that problem, please see: how-to-use-sam-deploy-to-get-a-lambda-with-autopublishalias-and-additional-alises
SampleLambdaFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: SampleLambda
AutoPublishAlias: staging
CodeUri: src/
Handler: SampleLambda.handler
MemorySize: 512
Runtime: nodejs12.x
Role: !GetAtt SampleLambdaRole.Arn
SampleLambdaLiveAlias:
Type: AWS::Lambda::Alias
Properties:
FunctionName: !Ref SampleLambdaFunction
FunctionVersion: !GetAtt SampleLambdaGetMaxVersionFunction.version
Name: live
SampleLambdaFunctionScheduledEvent:
Type: AWS::Events::Rule
Properties:
State: ENABLED
ScheduleExpression: rate(1 minute) # same as cron(0/1 * * * ? *)
Description: Run SampleLambdaFunction once every 5 minutes.
Targets:
- Id: EventSampleLambda
Arn: SampleLambdaLiveAlias
Your error is in the last line of the piece of configuration you shared. In order to get the resource ARN you need to use Ref intrinsic function such as, !Ref SampleLambdaLiveAlias:
SampleLambdaFunctionScheduledEvent:
Type: AWS::Events::Rule
Properties:
State: ENABLED
ScheduleExpression: rate(1 minute) # same as cron(0/1 * * * ? *)
Description: Run SampleLambdaFunction once every 5 minutes.
Targets:
- Id: EventSampleLambda
Arn: !Ref SampleLambdaLiveAlias
Be aware that Ref intrinsic function may return different things for different types of resources. For Lambda alias it returns the ARN, just what you need.
You can check the official documentation for more detail.

How to specify AutoPublishAlias at runtime of 'sam package/deploy'?

I am wrapping our AWS SAM deployment in Jenkins as part of our CI/CD pipeline. I only want to add the "live" alias to the lambdas when we are merging for example, yet I want "branch builds" to be without an alias. This allows developers to test the code in AWS without it being "live". Other than sed replacing part of the template.yaml before I run "sam package/deploy" is there some other way to accomplish this?
--UPDATE--
It looks like I can use Parameters to create environments in my lambda, but I don't know how to toggle between them. This would look like:
Parameters:
MyEnv:
Description: Environment of this stack of resources
Type: String
Default: testing
AllowedValues:
- testing
- prod
Then I can reference this w/:
Environment:
Variables:
ENV: !Ref: MyEnv
If someone knows how to toggle this parameter at runtime that solves my problem.
I got this working. My template.yaml:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sams-app
Globals:
Function:
Timeout: 3
Parameters:
Stage:
Type: String
Description: Which stage the code is in
Default: test
AllowedValues:
- test
- prod
Resources:
HelloWorldSQSFunction:
Type: AWS::Serverless::Function
Properties:
Role: arn:aws:iam::xxxxxxxxxxxx:role/service_lambda_default1
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python3.7
AutoPublishAlias: !Ref Stage
DeploymentPreference:
Type: AllAtOnce
Environment:
Variables:
STAGE: !Ref Stage
Outputs:
HelloWorldSQSFunction:
Description: "Hello World SQS Lambda Function ARN"
Value: !GetAtt HelloWorldSQSFunction.Arn
My lambda code:
import json
import os
def lambda_handler(event, context):
stage = os.environ['STAGE']
print(f"My stage is: {stage}")
return {
"statusCode": 200,
}
And to run it locally (I'm using Cloud9):
DEVELOPER:~/environment/sams-app $ sam local invoke --parameter-overrides Stage=prod
Invoking app.lambda_handler (python3.7)
Fetching lambci/lambda:python3.7 Docker container image......
Mounting /home/ec2-user/environment/sams-app/hello_world as /var/task:ro,delegated inside runtime container
START RequestId: 85da81b1-ef74-1b7d-6ad0-a356f4aa8b76 Version: $LATEST
My stage is: prod
END RequestId: 85da81b1-ef74-1b7d-6ad0-a356f4aa8b76
REPORT RequestId: 85da81b1-ef74-1b7d-6ad0-a356f4aa8b76 Init Duration: 127.56 ms Duration: 3.69 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 22 MB
{"statusCode":200}
One thing to note is that this will cause your "sam validate" to fail. For info on that, see: https://github.com/awslabs/serverless-application-model/issues/778
Special thanks to JLarky for the comment on this thread: aws-sam-local environment variables

InvalidResourceException when trying to "ref" an event in a function

I have a SAM template file that is throwing errors while doing sam build: [InvalidResourceException('MyFunction', "Type of property 'Events' is invalid.")]
First off, at the top of my file (at the same level as Globals) I have this event (the idea is to define a CloudWatch schedule that fires every 15 minutes and invokes a lambda):
Events:
Type: Schedule
Properties:
Schedule: rate(15 mins)
name: InvokeEvery15MinutesSchedule
Description: Invoke the target every 15 mins
Enabled: True
And here's what the function looks like:
MyFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: ./path-to-code
Events:
- !Ref InvokeEvery15MinutesSchedule
I was doing this because I saw earlier that the following syntax is valid:
Globals:
Function:
Layers:
- !Ref Layer1
- !Ref Layer1
So, I thought that if I define an event at the top level and reference it inside the lambda, it will work. I want to keep it outside of the Lambda declaration because I want this to apply to several functions.
Can someone help with what I'm doing wrong?
"Events" is a lambda source object that defines the events that trigger this function. The object describing the source of events which trigger the function.
Try this:
MyFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: ./path-to-code
Events:
RateSchedule:
Type: Schedule
Properties:
Schedule: rate(15 mins)
Name: InvokeEvery15MinutesSchedule
Description: Invoke the target every 15 mins
Enabled: True

Resources