Schedule AWS-Lambda with Java and CloudWatch Triggers - spring-boot

I am new to AWS and AWS-Lambdas. I have to create a lambda function to run a cron job in every 10 minutes. I am planning to add a Cloudwatch trigger to trigger the same in every 10 minutes but without any event. I looked up on the internet and found that some event needs to be there to get it running.
I need to get some clarity and leads on below 2 points:
Can I schedule a job using AWS-Lambda with cloudwatch triggering the same in span of 10 minutes without any events.
How do one need to make it interact with MySQL databases that have been hosted on AWS.
I have my application built on SpringBoot running on multiple instances with a shared database (single source of truth). I have devised everything stated above using Spring's in-built scheduler and proper synchronisation on DB level using locks but because of the distributed nature of instances, I have been advised to do the same using lambdas.

You need to pass ScheduledEvent object to the handleRequest() of the lambda.
handleRequest(ScheduledEvent event, Contex context)
Configure a cron job that runs every 10 minutes in your cloudwatch template (if using cloudformation). This will make sure to trigger your lambda after every 10 min.
Make sure to add below mentioned dependency to your pom.
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-events</artifactId>
<version>2.2.5</version>
</dependency>
Method 2:
You can specify something like this in your cloudformation template. This will not require any argument to be passed to your handler(), incase you do not require any event related information. This will automatically trigger your lambda as per your cron job.
"ScheduledRule": {
"Type": "AWS::Events::Rule",
"Properties": {
"Description": "ScheduledRule",
"ScheduleExpression": {
"Fn::Join": [
"",
[
"cron(",
{
"Ref": "ScheduleCronExpression"
},
")"
]
]
},
"State": "ENABLED",
"Targets": [
{
"Arn": {
"Fn::GetAtt": [
"LAMBDANAME",
"Arn"
]
},
"Id": "TargetFunctionV1"
}
]
}
},
"PermissionForEventsToInvokeLambdaFunction": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Ref": "NAME"
},
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceArn": {
"Fn::GetAtt": [
"ScheduledRule",
"Arn"
]
}
}
}
}

If you want to run a cronjob from cloudwatch event is the only option.
If you don't want to use cloudwatch events then go ahead with EC2 instance. But EC2 will cost you more than the cloudwatch event.
Note: Cloudwatch events rule steup is just like defining cronjob in crontab in any linux system, nothing much. In linux serevr you will define everything as a RAW one but here its just an UI based one.

Related

Lambda function won't run in parallel via Statemachine step functions

I am sure I am missing some basic info but my setup is as follows:
I have 1 lambda function and I have 2 State Machines.
Each state machine is calling the same lambda function twice in sequence (30 sec pause between calls)
Then I have rules setup to trigger the state machines every minute. Below is what each of my state machines look like, each lambda invoke is for the same function.
Each state machine is passing different params to the lambda function so I am trying to get to a situation where my Lambda function is called every 30 seconds with 1 set of params (from statemachine 1), and the same lambda function is called with a different set of params (via statemachine 2) every 30 seconds.
Looking at the lambda function logs it looks like the state machines will not run the lambda function until the other state machine has completed its entire execution (ie calling the lambda function twice). I would expect that the two state machines would run independently of each other and there would be no timing dependency between them?
Is there some limitation because they are all calling the same lambda function? Or is there some setup issue or is this just how it works?
Thanks!
From the documetation:
When you invoke a Lambda function, the execution will wait for the function to complete. However, it is possible to call Lambda asynchronously using the InvocationType parameter, as seen in the following example
To avoid waiting for oneLambda function to end and continue your step function you have to set the InvocationType to Event in the parameters. However, if your Lambda functions are completely independent from one another using the Parallel execution type may be a better option for you.
{
"Comment": "Parallel Example.",
"StartAt": "LookupCustomerInfo",
"States": {
"LookupCustomerInfo": {
"Type": "Parallel",
"End": true,
"Branches": [
{
"StartAt": "LookupAddress",
"States": {
"LookupAddress": {
"Type": "Task",
"Resource":
"arn:aws:lambda:us-east-1:123456789012:function:AddressFinder",
"End": true
}
}
},
{
"StartAt": "LookupPhone",
"States": {
"LookupPhone": {
"Type": "Task",
"Resource":
"arn:aws:lambda:us-east-1:123456789012:function:PhoneFinder",
"End": true
}
}
}
]
}
}
}

How to trigger lambda once Cloudformation stack is created

I have to trigger a lambda function once specific stack is created.
I have created the below CloudWatch event rule and associated the target to that lambda function but it is not triggering the lambda.
{
"source": [
"aws.cloudformation"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"cloudformation.amazonaws.com"
],
"eventName": [
"CreateStack"
],
"stackName": [
"sql-automate-04-08"
]
}
}
Please let me know if i am missing anything here.
This doesn’t work using CloudWatch Event Rules because the CloudFormation stack’s lifecycle events don’t reflect individual API calls.
However, you can configure CloudFormation to send stack events to an Amazon SNS topic via its NotificationARNs property. An AWS Lambda function subscribed to that topic can then filter and process the events.
This EventBridge Rule has worked for me:
{
"source": ["aws.cloudformation"],
"detail-type": ["CloudFormation Stack Status Change"],
"region": ["us-east-1"],
"detail": {
"status-details": {
"status": ["CREATE_COMPLETE"]
}
}
}

ECS provisions multiple servers but never runs the task

I have an ECS cluster where the capacity provider is an auto-scaling group of ec2 servers with a Target Tracking scaling policy and Managed Scaling turned on.
The min capacity of the cluster is 0, the max is 100. The instance types it's employing are c5.12xlarge.
I have a task that uses 4 x vCPUs and 4 GiB memory. When I run a single instance of that task on that cluster, ECS very slowly auto scales the group to > 1 servers (usually 2 to begin with, and then eventually adds a third one - I've tried multiple times), but never actually runs the task and stays in a state of PROVISIONING for ages and ages before I get annoyed and stop the task.
Here is a redacted copy of my task description:
{
"family": "my-task",
"taskRoleArn": "arn:aws:iam::999999999999:role/My-IAM-Role",
"executionRoleArn": "arn:aws:iam::999999999999:role/ecsTaskExecutionRole",
"cpu": "4 vCPU",
"memory": 4096,
"containerDefinitions": [
{
"name": "my-task",
"image": "999999999999.dkr.ecr.us-east-1.amazonaws.com/my-container:latest",
"essential": true,
"portMappings": [
{
"containerPort": 12012,
"hostPort": 12012,
"protocol": "tcp"
}
],
"mountPoints": [
{
"sourceVolume": "myEfsVolume",
"containerPath": "/mnt/efs",
"readOnly": false
}
]
}
],
"volumes": [
{
"name": "myEfsVolume",
"efsVolumeConfiguration": {
"fileSystemId": "fs-1234567",
"transitEncryption": "ENABLED",
"authorizationConfig": {
"iam": "ENABLED"
}
}
}
],
"requiresCompatibilities": [
"EC2"
],
"tags": [
...
]
}
My questions are:
Why, if I'm running a single task that would easily run on once instance, is it scaling the group to at least 2 servers?
Why does it never just deploy and run my task?
Where can I look to see what the hell is going on with it (logs, etc)?
So it turns out that, even if you set an ASG to be the capacity provider for an ECS cluster, if you haven't set the User Data up in the launch configuration for that ASG to have something like the following:
#!/bin/bash
echo ECS_CLUSTER=my-cluster-name >> /etc/ecs/ecs.config;echo ECS_BACKEND_HOST= >> /etc/ecs/ecs.config;
then it will never make a single instance available to your cluster. ECS will respond by continuing to increase the desired capacity of the ASG.
Personally I feel like this is something that ECS should ensure happens without your knowledge. Maybe there's a good reason why not.

Trigger lambda on ECR PutImage event

I'm having trouble understanding why a Cloudwatch event rule is not firing.
I've followed this related question and did the following.
Created a Cloudtrail which sends events to a Cloudwatch log
Created the following CloudWatch event rule:
{
"detail-type": [
"AWS write API Call via CloudTrail"
],
"source": [
"aws.ecr"
],
"detail": {
"eventSource": [
"ecr.amazonaws.com"
],
"eventName": [
"PutImage"
]
}
}
Created a lambda to be invoked by this rule.
I can verify that in my Cloudwatch log group (set up to accept events from Cloudtrail) I am seeing the PutImage event. However, the lambda never fires and the rule metrics show that it is never triggered. I am assuming at this point the rule must be faulty (I would expect to see the rule triggered even if the lambda is faulty) but I can't see what additional logic is required. Is it necessary to link my event to a particular log group?

Can a lambda in an AWS Step Function know the name of the step it is in?

For a lambda executed within a step function, I kind of expected that I could get the name of the current step from the lambda context, but it doesn't seem to be that simple.
Is there any way to get the name of the current step in a lambda that is executed within a Step Function?
UPDATE: as of 05/23/2019 this answer is outdated, since AWS introduced a way to access current step within a step function, see the accepted answer.
Looks like you are right, the current step doesn't get exposed through the context variable.
So, the information that would allow you to identify what stage is the state machine currently in, should be passed from the previous step (i.e. from the previous lambda). This seems to be the most correct option.
Or, as a workaround, you could try inserting pass states before calling your lambda functions to pass an id that could help you to identify the current stage.
Suppose you have two steps in your state machine:
"Step1Test": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxx:function:step1test",
"Next": "Step2Test"
},
"Step2Test": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxx:function:step2test",
"End": true
}
Here is how you can provide your lambda functions with current step id passed via event.stepId
"Step1TestEnter": {
"Type": "Pass",
"Next": "Step1Test",
"Result": "Step1Test",
"ResultPath": "$.stepId"
},
"Step1Test": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxx:function:step1test",
"Next": "Step2TestEnter"
},
"Step2TestEnter": {
"Type": "Pass",
"Next": "Step2Test",
"Result": "Step2Test",
"ResultPath": "$.stepId"
},
"Step2Test": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:xxxxxxxxxx:function:step2test",
"End": true
}
AWS Step Functions released Context Object where you can access information about your execution.
You can use it to send the execution arn to your lambda.
https://docs.aws.amazon.com/step-functions/latest/dg/input-output-contextobject.html
Based on the documentation, I was using Parameters and $$ to pass the step function context object into my lambda function. To test and see if it was working, I thought I could go to the step function console, start a new execution and see the context object being passed into the step function on the "Step Input" tab. To my dismay, it wasn't displayed there. I added some diagnostic logging to the lambda function serializing the input to JSON and logging out to CloudWatch. Cloudwatch logs showed that the context object was being passed in.
Anyway, thought I would post this here to maybe help someone avoid the time I spent trying to figure this one out. It gets passed in, just doesn't show up in the step function console.
I highly recommend when using step functions to specify some sort of key in the step function configuration. For my step functions I always provide:
"ResultPath": "$",
"Parameters": {
"source": "StepFunction",
"type": "LAMBDA_METHOD_SWITCH_VALUE",
"payload.$": "$"
},
And have each call to lambda use the type field to determine what code to call. I have found this to be much easier to implement.

Resources