Running a Cloudformation project on localhost with DynamoDB, Lambda, and API Gateway resources using Localstack - aws-lambda

I have a CloudFormation project with the following configs.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
team-up-interactions
Resources:
handler:
Type: AWS::Serverless::Function
Properties:
Description: Team Up interaction handler logic
CodeUri: ./handler/
Handler: handler.handler
Runtime: nodejs14.x
EventInvokeConfig:
MaximumRetryAttempts: 0
Timeout: 60
Environment:
Variables:
dbStorage: !Ref dbStorage
dbSearch: !Ref dbSearch
Architectures:
- x86_64
Policies:
- AWSLambdaBasicExecutionRole
Events:
HttpPost:
Type: Api
Properties:
Path: '/interaction-endpoint'
Method: post
dbStorage:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
-
AttributeName: "id"
AttributeType: "S"
KeySchema:
-
AttributeName: "id"
KeyType: "HASH"
BillingMode: "PAY_PER_REQUEST"
dbSearch:
Type: AWS::OpenSearchService::Domain
Properties:
EngineVersion: 'OpenSearch_1.1'
ClusterConfig:
InstanceCount: '1'
InstanceType: 't3.small.search'
EBSOptions:
EBSEnabled: true
VolumeSize: '20'
VolumeType: 'gp2'
These configs should create a Lambda, a DynamoDB table, and an OpenSearch Domain.
I am using LocalStack's samlocal command to run the project.
samlocal local start-api --host 0.0.0.0 --warm-containers EAGER
I have the following code that attempts to connect to the DynamoDB and is failing.
try {
const client = new DynamoDBClient({
endpoint: "http://localhost:4566"
});
await client.send(new UpdateItemCommand({
TableName: process.env.dbStorage,
Key: marshall({id: "test"}),
ExpressionAttributeValues: marshall({":test": "hello world"}),
UpdateExpression: "SET test = :test",
}));
//
const x = await client.send(new GetItemCommand({
TableName: process.env.dbStorage,
Key: marshall({id: "test"}),
}))
console.log("JSON.stringify(x)");
console.log(JSON.stringify(x));
} catch (ex) {
console.log("JSON.stringify(ex)")
console.log(JSON.stringify(ex))
}
This is the error.
{
"errno": -111,
"code": "ECONNREFUSED",
"syscall": "connect",
"address": "127.0.0.1",
"port": 4566,
"$metadata": {
"attempts": 1,
"totalRetryDelay": 0
}
}
I also tried creating the client without any parameter and that failed too.
const client = new DynamoDBClient();
This is the error.
{
"name": "UnrecognizedClientException",
"$fault": "client",
"$metadata": {
"httpStatusCode": 400,
"requestId": "VAHE0KG1D1UTE9AQNAREF9RJHFVV4KQNSO5AEMVJF66Q9ASUAAJG",
"attempts": 1,
"totalRetryDelay": 0
},
"__type": "com.amazon.coral.service#UnrecognizedClientException",
"message": "The security token included in the request is invalid."
}
How can I get LocalStack to create the DynamoDB resource from the configurations and/or how do I access those resources?
Update
I am using this endpoint and it seems to be able to connect the client to localstack.
const client = new DynamoDBClient({
endpoint: "http://host.docker.internal:4566",
});
I get the following error when doing a GetItemCommand
ResourceNotFoundException: Cannot do operations on a non-existent table
at deserializeAws_json1_0ResourceNotFoundExceptionResponse (/var/task/node_modules/#aws-sdk/client-dynamodb/dist-cjs/protocols/Aws_json1_0.js:3036:23)
at deserializeAws_json1_0GetItemCommandError (/var/task/node_modules/#aws-sdk/client-dynamodb/dist-cjs/protocols/Aws_json1_0.js:1729:25)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async /var/task/node_modules/#aws-sdk/client-dynamodb/node_modules/#aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
at async /var/task/node_modules/#aws-sdk/client-dynamodb/node_modules/#aws-sdk/middleware-signing/dist-cjs/middleware.js:11:20
at async StandardRetryStrategy.retry (/var/task/node_modules/#aws-sdk/client-dynamodb/node_modules/#aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
at async /var/task/node_modules/#aws-sdk/client-dynamodb/node_modules/#aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
at async module.exports (/var/task/interaction/instant/help/help.js:31:19)
at async Runtime.exports.handler (/var/task/handler.js:94:16) {
'$fault': 'client',
'$metadata': {
httpStatusCode: 400,
requestId: '3ad8e2c9-be76-449d-bc86-4b529f3ca45b',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
__type: 'com.amazonaws.dynamodb.v20120810#ResourceNotFoundException'
}

Related

attach authorizer to api gateway V2 route in aws cloudformation

How to attach authorizer to api gateway V2 route in aws cloudformation?
I am using Api Gateway v2 and cloudformation.
In AWS console it is just one click of one button "Attach Authorization" in "Routes" section
I am using simple authorizer:
My cloudformation looks like this:
Authorizer:
Type: 'AWS::ApiGatewayV2::Authorizer'
Properties:
ApiId: !Ref ApiGateway
AuthorizerPayloadFormatVersion: 2.0
AuthorizerResultTtlInSeconds: 5
AuthorizerType: REQUEST
AuthorizerUri: !Join
- ''
- - 'arn:'
- !Ref 'AWS::Partition'
- ':apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- 'arn:aws:lambda:'
- !Ref 'AWS::Region'
- ':'
- !Ref 'AWS::AccountId'
- :function:${stageVariables.AuthorizerFunctionName}
- /invocations
EnableSimpleResponses: true
IdentitySource:
- '$request.header.Authorization'
Name: !Sub ${ProjectName}-gateway-authorizer
MyRoute:
Type: AWS::ApiGatewayV2::Route
Properties:
ApiId: !Ref ApiGateway
AuthorizationType: CUSTOM
AuthorizerId: !Ref Authorizer
RouteKey: 'POST /posts/all'
Target: !Join
- /
- - integrations
- !Ref PostsLambdaIntegrationGet
Authorizer lambda body:
import json
# import jwt
def lambda_handler(event, context):
print('*********** The event is: ***************')
print(event)
print('headers is:')
print(event['headers'])
print('headers Authorization is:')
# !!!!! DONWCASE by postam or api !!!!! "A" -> "a"
print(event['headers']['authorization'])
if event['headers']['authorization'] == 'abc123':
response = {
"isAuthorized": True,
"context": {
"anyotherparam": "values"
}
}
else:
response = {
"isAuthorized": False,
"context": {
"anyotherparam": "values"
}
}
print('response is:')
print(response)
return response
BTW I do not see this option in cli apigatewayv2 cli documentation too:

How to pass the Arn of a Step function to another function in a CloudFormation stack?

I created a step function in a CloudFormation stack, and I need to call that function from another function in the stack once it's deployed. I have no idea how to do that, and I'm getting circular dependencies.
Basically I'm trying to pass an environment variable to the function that's the ARN of the step function.
Here's the CloudFormation code :
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
reports stack
Globals:
Function:
Timeout: 120
MemorySize: 2048
Runtime: python3.9
Environment:
Variables:
STEP_FUNCTION: !GetAtt Research.Arn ### =====> how do i do that ?
Parameters:
ProjectName:
Description: Name of the Project
Type: String
Default: Project
Resources:
MyApi:
Type: AWS::Serverless::Api
Properties:
StageName: Prod
TracingEnabled: true
Cors:
AllowMethods: "'DELETE,GET,HEAD,OPTIONS,POST,PUT'"
AllowHeaders: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
AllowOrigin: "'*'"
List:
DependsOn: VideoResearch
Type: AWS::Serverless::Function
Properties:
CodeUri: functions/
Handler: get.get
Events:
List:
Type: Api
Properties:
RestApiId: !Ref MyApi
Path: /reports
Method: GET
Policies:
(...)
################ STEP FUNCTION ################
Research:
Type: AWS::Serverless::Function
Properties:
CodeUri: functions/
Handler: runResearch.research
Policies:
(...)
StepFunctionToHandleResearchRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
- !Sub states.${AWS::Region}.amazonaws.com
Action: "sts:AssumeRole"
Path: "/"
Policies:
(...)
#
StepFunctionToHandleVideosResearch:
Type: "AWS::StepFunctions::StateMachine"
Properties:
DefinitionString: !Sub |
{
"StartAt": "Research",
"States": {
"Research": {
"Type": "Task",
"Resource": "${VideoResearch.Arn}",
"End": true
}
}
}
RoleArn: !GetAtt [ StepFunctionToHandleResearchRole, Arn ]
Outputs:
(...)
# I also tried to export the arn of the function
In my function's code I have :
stepFunctionARN = os.environ['STEP_FUNCTION']
#Anthony B. posted the answer in the comments : the problem was related to the fact I added the dependency as global variables, it created a circular dependency (API=> local function <= API looking for function's ARN => doesn't work).
If you remove the global variable and add a local variable, with a "DependsOn: Research" than everything works.
API=>created => creates first function, get Arn => than creates other functions and provide ARN

Using lambda authorizer in serverless function exposed through API gateway

I have written a lambda authorizer which returns response in this format -
{
principalId: 123345,
policyDocument: {
"Version": "2012-10-17",
"Statement": [
{
"Action": "execute-api:Invoke",
"Effect": "Allow",
"Resource": "arn:aws:execute-api:us-east-1:123456789012:ivdtdhp7b5/verifyToken-stage/GET/"
}
]
}
}
Note- My authorizer's name is verifyToken
In my different microservice(i.e Activity Logs) serverless.yml file
I am calling it like this -
service: activity-logs
frameworkVersion: '2'
resources:
Resources:
GatewayResponseUnauthorized:
Type: 'AWS::ApiGateway::GatewayResponse'
Properties:
ResponseParameters:
gatewayresponse.header.Access-Control-Allow-Origin: "'*'"
gatewayresponse.header.Access-Control-Allow-Headers: "'*'"
gatewayresponse.header.Access-Control-Allow-Methods: "'*'"
gatewayresponse.header.Access-Control-Allow-Credentials: "'true'"
ResponseType: UNAUTHORIZED
RestApiId:
Ref: 'ApiGatewayRestApi'
provider:
name: aws
runtime: nodejs12.x
lambdaHashingVersion: 20201221
region: us-east-1
stage: ${opt:stage}
functions:
getActivityLogs:
handler: handler.getActivityLogs
environment:
NODE_ENV: ${opt:env}
timeout: 800
events:
- http:
path: /{user_id}
method: get
authorizer: arn:aws:lambda:us-east-1:123456789012:function:auth-${opt:stage}-verifyToken
cors: true
vpc:
securityGroupIds:
- sg-xxxxxxxx
- sg-xxxxxxxx
- sg-xxxxxxxx
- sg-xxxxxxxx
- sg-xxxxxxxx
subnetIds:
- subnet-xxxxxxxxxxxxxxxxx
- subnet-xxxxxxxxxxxxxxxxx
I have checked my authorizer separately, it is working and returning 200 with the above response. Similarly, my microservice is working without authorizer. But when authorizer is enabled in getActivityLogs, it is not letting my activity-logs execute a single line
I was using callback for returning response in async function and as per this doc, https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html
we can simply return our response if a function is async.

Terraform API gateway integration with Swagger (Localstack)

i created a simple crud API lambda using mongoDb and nodeJs and it's working fine. so i passed to another step
I'm currently creating an AWS API Gateway with terraform by using an open api spec from a yaml file.
this is my swagger file:
info:
description: "API Ankan-v2"
version: "V0.0.1"
title: "ANKAN V2 API"
host: "localhost:4567"
basePath: "/restapis/api/{apiId}/test/_user_request_"
tags:
- name: "user"
description: "Operations about user"
schemes:
- "http"
# - "https"
paths:
/documents/{documentId}:
get:
tags:
- "document"
summary: "Find document by ID"
description: "Returns a single document"
operationId: "readOneDocument"
produces:
- "application/json"
parameters:
- name: "documentId"
in: "path"
description: "Id of document"
required: true
type: "string"
responses:
200:
description: "successful operation"
schema:
$ref: "#/definitions/Document"
404:
description: "document not found"
x-amazon-apigateway-integration:
uri: ${get_lambda_arn}
passthroughBehavior: "WHEN_NO_MATCH"
httpMethod: "POST"
type: "AWS_PROXY"
options:
consumes:
- "application/json"
produces:
- "application/json"
parameters:
- name: "documentId"
in: "path"
description: "Id of document"
required: true
type: "string"
responses:
200:
description: "200 response"
schema:
$ref: "#/definitions/Document"
x-amazon-apigateway-integration:
responses:
default:
statusCode: "200"
responseParameters:
method.response.header.Access-Control-Allow-Methods: "'DELETE,GET,HEAD,OPTIONS,PATCH,POST,PUT'"
method.response.header.Access-Control-Allow-Headers: "'Content-Type,Authorization,X-Amz-Date,X-Api-Key,X-Amz-Security-Token'"
method.response.header.Access-Control-Allow-Origin: "'*'"
requestTemplates:
application/json: "{\"statusCode\": 200}"
passthroughBehavior: "when_no_match"
type: "mock"
x-amazon-apigateway-any-method:
produces:
- "application/json"
parameters:
- name: "documentId"
in: "path"
description: "Id of document"
required: true
type: "string"
responses: {}
x-amazon-apigateway-integration:
responses:
default:
statusCode: "200"
uri: ${get_lambda_arn}
passthroughBehavior: "WHEN_NO_MATCH"
httpMethod: "POST"
cacheNamespace: "1rnijn"
cacheKeyParameters:
- "method.request.path.documentId"
type: "AWS"
definitions:
Document:
type: "object"
required:
- "documentName"
- "documentPath"
- "miniature"
- "status"
- "date"
properties:
_id:
type: "string"
readOnly: true
documentName:
type: "string"
documentPath:
type: "string"
miniature:
type: "string"
status:
type: "string"
enum:
- "VALID"
- "ERROR"
- "ONLOAD"
date:
type: "string"
format: "date"
xml:
name: "Document"
in my terraform i removed the integration and the method because i added them in my swagger .
so i need to import the swagger file in aws_api_gateway_rest_api with body = data.template_file.swagger.rendered
resource "aws_api_gateway_rest_api" "api" {
name = "api-gateway"
description = "document api"
body = data.template_file.swagger.rendered
}
data "template_file" "swagger" {
template = jsonencode("${path.module}/../../../../shared/swagger.yaml")
vars = {
get_lambda_arn = local.get_lambda_arn
}
}
output "json" {
value = data.template_file.swagger.rendered
}
locals {
get_lambda_arn = aws_lambda_function.lambda_document.invoke_arn
}
resource "aws_api_gateway_resource" "documents" {
rest_api_id = aws_api_gateway_rest_api.api.id
parent_id = aws_api_gateway_rest_api.api.root_resource_id
path_part = "documents"
depends_on = [aws_api_gateway_rest_api.api]
}
resource "aws_api_gateway_resource" "document_by_id" {
parent_id = aws_api_gateway_resource.documents.id
path_part = "{documentId}"
rest_api_id = aws_api_gateway_rest_api.api.id
}
resource "aws_api_gateway_deployment" "ApiDeloyment" {
rest_api_id = aws_api_gateway_rest_api.api.id
stage_name = "INT"
depends_on = [aws_api_gateway_rest_api.api]
}
resource "aws_lambda_function" "lambda_document" {
function_name = "lambda_document"
handler = "lambda.handler"
role = "arn:aws::iam::123456:role/irrelevant"
runtime = "nodejs10.x"
filename = "${path.module}/lambda.zip"
source_code_hash = filebase64sha256("${path.module}/lambda.zip")
}
After terraform apply i got this message in terraform logs
terraform apply --auto-approve
module.backend_api.module.document.aws_api_gateway_rest_api.api: Creating...
module.backend_api.module.document.aws_lambda_function.lambda_document: Creating...
module.backend_api.module.document.aws_lambda_function.lambda_document: Creation complete after 5s [id=lambda_document]
module.backend_api.module.document.data.template_file.swagger: Refreshing state...
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [10s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [20s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [30s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [40s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [50s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [1m0s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [1m10s elapsed]
and my docker-compose logs show this
localstack | 2020-01-13T07:37:56:ERROR:localstack.services.generic_proxy: Error forwarding request: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last):
localstack | File "/opt/code/localstack/localstack/services/generic_proxy.py", line 242, in forward
localstack | path=path, data=data, headers=forward_headers)
localstack | File "/opt/code/localstack/localstack/services/apigateway/apigateway_listener.py", line 53, in forward_request
localstack | data = data and json.loads(to_str(data))
localstack | File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
localstack | return _default_decoder.decode(s)
localstack | File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
localstack | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
localstack | File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
localstack | raise JSONDecodeError("Expecting value", s, err.value) from None
localstack | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
PS : i tried with JSON but same problem
is there any solution for this? i want to integrate swagger with api gateway using terraform in localstack
This is an issue with the moto library. Under the hood, local stack uses moto to create its resources, including mock api gateway. Unfortunately moto does not currently support the body parameter for putting a swagger file definition as part of api gateway deployment. So even if you give a valid swagger file and api gateway configuration (ie one that works in aws) localstack will not be able to build it.

How do I cloudform an API gateway resource with a lambda proxy integration

I've been trying to work out how to express (in cloudformation) an API Gateway Resource that has a Lambda function integration type using the Lambda Proxy integration.
This is easy to do in the AWS console as there is a check box that you can select:
However there is no corresponding field in the AWS::ApiGateway::Method CloudFormation resource (it should be in the Integration property).
How can I configure this in cloudformation?
The Integration type should be set to AWS_PROXY. An example snippet of a method from a working YAML CloudFormation template is below.
ProxyResourceAny:
Type: AWS::ApiGateway::Method
Properties:
AuthorizationType: NONE
HttpMethod: ANY
ResourceId:
Ref: ProxyResource
RestApiId:
Ref: API
Integration:
Type: AWS_PROXY
IntegrationHttpMethod: POST
Uri: !Sub
- arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Arn}/invocations
- Arn:
Fn::GetAtt:
- RestorerLambda
- Arn
It's worth saying how a I figured this out...
After scratching my head for a while I examined the output of the aws apigateway get-method CLI command for a method that was configured this way using the console. That gave me the following JSON and I realised that the checkbox might be encoded into the type. I tested my assumption and came up with the CloudFormation above.
{
"apiKeyRequired": false,
"httpMethod": "ANY",
"methodIntegration": {
"integrationResponses": {
"200": {
"responseTemplates": {
"application/json": null
},
"statusCode": "200"
}
},
"passthroughBehavior": "WHEN_NO_MATCH",
"cacheKeyParameters": [],
"uri": "arn:aws:apigateway:eu-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:eu-west-1:XXXXXXXXX:function:Shildrew-Restorer-Play-Lambda/invocations",
"httpMethod": "POST",
"cacheNamespace": "64bl3tgw4g",
"type": "AWS_PROXY"
},
"requestParameters": {},
"authorizationType": "NONE"
}
I have solved this same issue by simple changing the
Integration:
Type: AWS_PROXY
To
Integration:
Type: AWS
The cloud formation documentation currently is scarce and the API gateway cloudformation documentation doesn't match up to what can be seen on the console which hinders anyone who is trying to resolve an issue.
Hope this helps!
We faced this exact issue. We are using Ansible for our Infrastructure. Could apply to CLI or Cloudformation or even the SDK
The solution to our problem was to make sure that the Lambda policy was defined in a granular manner for the endpoint verbs in API Gateway for the lambda you are attempting to use.
For instance, We had multiple routes. Each route(or sets of routes) needs its own lambda policy defined that allows lambda:InvokeFunction. This is defined in the Lambda Policy module for Ansible. With this, the lambda trigger was enabled automatically.
There are two ways to achieve this:
Method 1: By defining the "Lambda", "API Gateway", "API Resource" and "API Methods". Linking the Lambda using the URI statement under "API Method".
MyLambdaFunction:
Type: "AWS::Lambda::Function"
Properties:
Description: "Node.js Express REST API"
FunctionName: "get_list_function" (The name in AWS console)
Handler: lambda.handler
Runtime: nodejs12
MemorySize: 128
Role: <ROLE ARN>
Timeout: 60
apiGateway:
Type: "AWS::ApiGateway::RestApi"
Properties:
Name: "example-api-gw"
Description: "Example API"
ProxyResource:
Type: "AWS::ApiGateway::Resource"
Properties:
ParentId: !GetAtt apiGateway.RootResourceId
RestApiId: !Ref apiGateway
PathPart: '{proxy+}' OR "a simple string like "PetStore"
apiGatewayRootMethod:
Type: "AWS::ApiGateway::Method"
Properties:
AuthorizationType: NONE
HttpMethod: ANY
Integration:
IntegrationHttpMethod: POST
Type: AWS_PROXY
IntegrationResponses:
- StatusCode: 200
Uri: !Sub >-
arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyLambdaFunction.Arn}/invocations
ResourceId: !Ref ProxyResource
RestApiId: !Ref "apiGateway"
METHOD 2: Define "API Gateway" and "Lambda". In Lambda definitions, call Events of type API.
CoreApi:
Type: 'AWS::Serverless::Api'
Properties:
StageName: dev
Name: SaaSAPI
EndpointConfiguration: Regional
Cors:
AllowMethods: '''POST, GET, OPTIONS, PATCH, DELETE, PUT'''
AllowHeaders: '''Content-Type, X-Amz-Date, Authorization, X-Api-Key, x-requested-with'''
AllowOrigin: '''*'''
MaxAge: '''600'''
MyLambdaFunction:
Type: "AWS::Lambda::Function"
Properties:
Description: "Node.js Express REST API"
FunctionName: "get_list_function"
Handler: lambda.handler
Runtime: nodejs12
MemorySize: 128
Role: <ROLE ARN>
Timeout: 60
Events:
ProxyResourceA:
Type: Api
Properties:
Path: /Departments
Method: GET
RestApiId: !Ref CoreApi
ProxyResourceB:
Type: Api
Properties:
Path: /Departments (This becomes the resource in API Gateway)
Method: POST
RestApiId: !Ref CoreApi
(You can link multiple methods with the same lambda, provide a unique name to each event, like ProxyResoruceA & ProxyResourceB)

Resources