Why does this AWS Lambda function keep looping from this EventBridge rule for AWS ECS? - aws-lambda

I have an EventBridge rule where it invokes a Lambda function (target) when one of the ECS Tasks starts (status: RUNNING). The Lambda function does something then at the end it is supposed to stop the ECS Task.
I have the following EventBridge rule:
{
"source": ["aws.ecs"],
"detail-type": ["ECS Task State Change"],
"detail": {
"clusterArn": ["<cluster-arn>"],
"lastStatus": ["RUNNING"],
"desiredStatus": ["RUNNING"]
}
}
And it invokes the Lambda function.
The following is a simplified version of the Lambda function:
import { ECSClient, StopTaskCommand } from "#aws-sdk/client-ecs";
export const handler = async (event, context, callback) => {
// event has the EventBridge event capture from the AWS ECS Task
// Do something
const ecs = new ECSClient({ region: "<region>" })
var taskArn = event.detail.containers[0].taskArn;
var stopTask = new StopTaskCommand({
cluster: "<cluster-arn>",
reason: "<reason>",
task: taskArn
});
try {
const data = await ecs.send(stopTask);
} catch (error) {
console.log(error)
}
}
When I start the ECS Task, the Lambda function is invoked and starts to run. After it finishes what it is supposed to do, it then goes to stop the ECS Task which invoked it. I use the aws-sdk v3 for this where I get the taskArn from the event parameter of the Lambda function. The Lambda function can successfully stop the ECS Task (I receive a 200 HTTP response code from result of the send command). However, the Lambda function is then invoked again and it repeats this forever (I checked the CloudWatch logs for the function).
I am not sure why the Lambda function starts up again as, from what I can tell, the EventBridge rule shouldn't trigger it.

Is not a problem with your Lambda Function. Is the desired behavior of your ECS Service!
Your ECS Service has a property "desired count", which instructs the service to meet the desired state.
If your service indicates that it should have 1 replica, then if you kill the task, the service will try to launch a new one until it meets the desired quantity. For this reason you are seeing that it returns to have activity over and over again.
It is probably not the correct approach to kill the service.
What I would do is change the "desired count" to zero, without using the lambda function.

Related

Event bus name not registering when attempting to connect eventbridge and lambda using terraform

I am attempting to create an Eventbridge that will get notifications from Datadog, and trigger a lambda function to store the notifications to an S3 bucket. This is all going to be done through Terraform.
The following is the code I have written:
###################################
# Eventbridge Integration to Lambda
###################################
data "aws_cloudwatch_event_source" "datadog_event_source" {
name_prefix = var.spog_event_bus_name # aws.partner/datadog.com/my_eventbus
}
resource "aws_cloudwatch_event_bus" "datadog_event_bus" {
name = data.aws_cloudwatch_event_source.datadog_event_source.name
event_source_name = data.aws_cloudwatch_event_source.datadog_event_source.name
}
resource "aws_cloudwatch_event_rule" "spog_cloudwatch_rule" {
name = "spog_cloudwatch_rule"
event_bus_name = aws_cloudwatch_event_bus.datadog_event_bus.name
event_pattern = <<EOF
{
"account": [
"${var.aws_account_id}"
]
}
EOF
}
resource "aws_cloudwatch_event_target" "spog_cloudwatch_event_target" {
rule = aws_cloudwatch_event_rule.spog_cloudwatch_rule.name
target_id = aws_lambda_function.write_datadog_events.function_name
arn = aws_lambda_function.write_datadog_events.arn
}
resource "aws_lambda_permission" "spog_allow_cloudwatch" {
statement_id = "AllowExecutionFromCloudWatch"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.write_datadog_events.function_name
principal = "events.amazonaws.com"
source_arn = aws_cloudwatch_event_rule.spog_cloudwatch_rule.arn
}
Here, the write_datadog_events is the lambda function to store the notifications.
The build succeeds, but when I try to apply the plan, I get an error saying that "validationException: EventBus name starting with 'aws.' is not valid". From inspecting the aws console, it seems that the actual eventbridge and rule are created successfully, but the event bus name is not registered properly on the eventbridge. The event bus name only says default, and I thought by changing the event_bus_name value within aws_cloudwatch_event_rule, it would not be default, but I was wrong.
Can anyone help me out with this. The lambda function itself is not wrong (since I ran a test case on it), it seems that the core issue is eventbridge not registering my event bus name. Also, although the rule is generated, the event bus is never generated.
Thanks for your help.

AWS lambda missing events from dynamodb stream

I am consistently missing lambda triggers from a dynamodb stream. I am mapping the dynamodb stream to the lambda function with event_source_mapping resource.
resource "aws_lambda_event_source_mapping" "history" {
event_source_arn = "${data.aws_dynamodb_table.auditlogDynamoDB.stream_arn}"
function_name = "${module.cx-clientcomm-history-lambda.lambda_function_arn}"
starting_position = "LATEST"
maximum_retry_attempts = 0
maximum_record_age_in_seconds = 604800 // adding to explicitly set. QA plan was providing invalid value
#(Optional) An Amazon SQS queue or Amazon SNS topic destination for failed records.
destination_config {
on_failure {
destination_arn = "${data.aws_sns_topic.cx-clientcomm-sns-slack.arn}"
}
}
}
It appears when there are entries in dynamodb are coming it very close to each other, some are not picked up by the event_source_mapping and not sent to the lambda.
Is there a way to make sure this doesn't happen? This needs to be very reliable

"Unsupported Media Type" using serverless offline

I'm working on a small serverless offline assignment and I got error Unsupported Media Type when tried to invoke one lambda function in another.
I found a solution but when I tried to applied to my project was not working:
here in the link all the details. cloud anyone help me with that
https://github.com/dherault/serverless-offline/issues/1005#issue-632401297
there are three possible solutions.
Make sure that the lambda_A have the same port and host where the lambda_B is running.
Lambda_A:
const { Lambda } = require('aws-sdk');
const lambda = new Lambda({
region: 'us-east-1',
endpoint: 'http://localhost:3000',
});
module.exports.main = async (event, context) => {
// invoke
}
Lambda_B: Is running on http://localhost:3000
You have configured out serverless-offline in twice functions.
https://www.serverless.com/plugins/serverless-offline#usage-with-invoke
Lambda_A or Lambda_B have correctly stage?. Remember to use sls offline --stage local in both functions.

CloudFormation: The runtime parameter of nodejs6.10 is no longer supported for creating or updating AWS Lambda functions

I'm trying to update a cloud formation template with a few lambda functions in it. The last version of the template was deployed a few years ago, and all the lambda functions currently have a runtime of node6.10.
I have updated the runtime for all functions to node10.x, but when I deploy the template, I get the following message:
The runtime parameter of nodejs6.10 is no longer supported for creating or updating AWS Lambda functions
I've created a change set, and reviewed it, and it includes an update to the runtime property for each lambda function, however Cloud Formation seems to be ignoring it.
Is there something I'm missing?
Context:
I assume that you encountered this issue - (you got "nodejs version not supported error" message after you tried to amplify push followed by amplify add auth)
Go to amplify -> backend -> auth -> cognito -> click cognito cloudformation ->
search for "Runtime: node"
change it to "Runtime: nodejs8.10" - whatever latest recommended in error message
re-run
$ amplify push
Unfortunately, I found I had to update the runtime of all functions in a template outside of Cloud Formation, to get the stacks to deploy. I used this script:
const AWS = require('aws-sdk')
const lambda = new AWS.Lambda(...)
main().catch(err => {
console.error(err)
process.exit(1)
})
async function main() {
const functions = await getFunctions()
await Promise.all(
functions
// filter only functions you want to update
.filter(...)
.filter(x => x.Runtime !== 'nodejs10.x')
.map(updateFunction)
)
}
async function updateFunction(func) {
await lambda
.updateFunctionConfiguration({
FunctionName: func.FunctionName,
Runtime: 'nodejs10.x'
})
.promise()
console.log(`function updated: ${func.FunctionName}`)
}
async function getFunctions() {
let marker
let functions = []
do {
const result = await lambda
.listFunctions({
Marker: marker
})
.promise()
functions = [...functions, ...result.Functions]
marker = result.NextMarker
} while (marker)
return functions
}

Is it possible to trigger a lambda on creation from CloudFormation template

I tried creating a set of lambdas using cloudformation. I want the lambdas to get triggered once they are created. I saw at various blogs to create a trigger to s3 or sns but none seems to be a option to trigger lambda once it has been created. Any options?
Yes, it is possible. Here are a few options:
Manually create an SNS Topic. Add an AWS::SNS::Subscription to your stack with the lambda function as the Endpoint and the SNS topic as the TopicArn. On stack creation/update, configure Stack Event Notifications to be sent to this SNS topic.
(See Setting AWS CloudFormation Stack Options for documentation on how to do this when using the AWS Console to create your stack, or use the equivalent option like --notification-arns if creating/updating your stack using the AWS CLI or other AWS SDK.)
Add a Custom Resource referencing a Lambda function to be called on creation.
If you need the Lambda function to be called after some specific Resource is created, add a DependsOn attribute on the Custom Resource referencing the Resource you want to make sure is created first before the function is called.
In order for the Custom Resource to create successfully (and not cause a failure/rollback in your stack), you will need to adapt your Lambda function to support the CloudFormation request/response format (see Custom Resource Reference).
This option will call the Lambda function while the stack status is still CREATE_IN_PROGRESS, because the Custom Resource is part of the stack itself.
The Lambda function will also be called again when the stack (and associated Custom Resource) is deleted. This will need to be handled by your Lambda function correctly, or your stack could get stuck in the DELETE_FAILED state.
Add the Lambda function reference to a Stack Output, then write a simple script that performs the stack creation and then manually invokes the Lambda function afterwards.
by yl.
The following just works great !
It invokes a lambda as a part of deployment:
LambdaFunction2:
Type: AWS::Lambda::Function
Properties:
FunctionName: caller
Code:
ZipFile: |
import boto3, json
import cfnresponse
def handler(event, context):
print('EVENT:[{}]'.format(event))
lambda_client = boto3.client('lambda')
test_event = '{"name":"test1"}'
lambda_client.invoke(
FunctionName='target1',
InvocationType='Event',
Payload=test_event,
)
responseValue = 120
responseData = {}
responseData['Data'] = responseValue
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData)
Handler: index.handler
Role:
arn:aws:iam::11111111111:role/mylambda-role
Runtime: python3.7
Timeout: 60
Primerinvoke:
Type: AWS::CloudFormation::CustomResource
DependsOn: LambdaFunction2
Version: "1.0"
Properties:
ServiceToken: !GetAtt LambdaFunction2.Arn
For who looking the similar workaround.
CloudWatch is able to capture API calls of CloudFormation, which is "CreateStack", "UpdateStack" and "DeleteStack", stack states like "Create_complete" or "Complete_Rollback" are uncapturable, which means such state changes not be able to trigger lambda.
The workaround is SNS, stacks are able to send notifications to SNS (In advance settings when you creating stack) and SNS can choose to trigger lambda, however, you can't choose for specific states. So, lambda function takes the job to find out what state in "Message" content of an event. Everyone, just coding.
I know this is a bit old- but a solution could also be too use CommandRunner as a resource type in your template.
https://aws.amazon.com/blogs/mt/running-bash-commands-in-aws-cloudformation-templates/.
You can run virtually any shell command. Add a DependsOn attribute to your CommandRunner type and run a shell script:
aws lambda invoke --function-name my-function --invocation-type RequestRespone --payload '{ "name": "Bob" }'
Improving on Kyr's answer, because it lacks two important things:
how to pass paramaters to the Lambda you invoke
how to treat UPDATE and DELETE on your Stack (his solution would cause CloudFormation to crash on delete)
Here is the revised and improved code:
LambdaInvoker:
DependsOn: ## important, add stuff here you need to existe BEFORE the lambda is called
Type: AWS::Lambda::Function
Properties:
FunctionName: YourLambdaName
Description: 'Lambda invoke wrapper for Custom CFN actions'
Code:
ZipFile: !Sub |
import boto3, json
import cfnresponse
def handler(event, context):
print('EVENT:')
print(event)
if event['RequestType'] == "Create":
lambda_client = boto3.client('lambda')
cfn_event = {
"param1" : "${Param1}",
"param2" : "${Param2}"
}
lambda_client.invoke(
FunctionName='scm-custom-cfn-actions',
InvocationType='Event',
Payload=json.dumps(cfn_event)
)
responseValue = 120
responseData = {}
responseData['Data'] = responseValue
cfnresponse.send(event, context, cfnresponse.SUCCESS,
responseData, 'scm-cfn-customresource-id')
Handler: index.handler
Role: YourLambdaRoleARN
Runtime: python3.7
Timeout: 5
You have the option to notify to a SNS topic, and you may build a lambda that listens to the topic, so the workflow would be: Cloudformation launch -> SNS Topic -> Lambda.
The following template should invoke the lambda :
"InvokeLambda" : {
"Type": "Custom::InvokeLambda",
"Version" : "1.0",
"Properties" : {
"ServiceToken": {
"Fn::GetAtt": ["InitFunction","Arn"]
}
}
},

Resources