Getting "internal server error" on passing binary data to AWS Lambda function deployed using serverless framework and apigw-binary plugin - aws-lambda

what I'm trying
Passing binary data via Lambda integration in API gateway. Lambda returns back text.
issue
The function returns desired output when API gateway is configured from console. To implement it using serverless framework I installed serverless-apigw-binary plugin. The required binary types show up in API gateway>settings>binary media types. However on calling API I get "internal server error". The function works properly on application/json type input. After enabling-disabling lambda proxy integration and adding mappings via console, I get correct output.
serverless.yml file
org: ------
app: ---------
service: ---------
frameworkVersion: ">=1.34.0 <2.0.0"
plugins:
- serverless-python-requirements
- serverless-offline
- serverless-apigw-binary
provider:
name: aws
runtime: python3.7 #fixed with pipenv
region: us-east-1
memorySize: 128
timeout: 60
profile: ----
custom:
pythonRequirements:
usePipenv: true
useDownloadCache: true
useStaticCache: true
apigwBinary:
types: #list of mime-types
- 'application/octet-stream'
- 'application/zip'
functions:
main:
handler: handler.main
events:
- http:
path: ocr
method: post
integration: lambda
request:
passThrough: WHEN_NO_TEMPLATES
template:
application/zip: '
{
"type": "zip",
"zip": "$input.body",
"lang": "$input.params(''lang'')",
"config": "$input.params(''config'')",
"output_type": "$input.params(''output_type'')"
}'
application/json: '
{
"type": "json",
"image": $input.json(''$.image''),
"lang": "$input.params(''lang'')",
"config": "$input.params(''config'')",
"output_type": "$input.params(''output_type'')"
}'
application/octet-stream: '
{
"type": "img_file",
"image": "$input.body",
"lang": "$input.params(''lang'')",
"config": "$input.params(''config'')",
"output_type": "$input.params(''output_type'')"
}'
handler.py
def main(event, context):
# do something on event and get txt
return txt
edit
I compared swagger definitions and found this
1. API generated from console(working)
paths:
/ocr:
post:
consumes:
- "application/octet-stream"
produces:
- "application/json"
responses:
API generated from serverless framework
paths:
/ocr:
post:
consumes:
- "application/x-www-form-urlencoded"
- "application/zip"
- "application/octet-stream"
- "application/json"
responses:
produces: - "application/json" is missing. How do I add it in serverless?

Related

Terraform API gateway integration with Swagger (Localstack)

i created a simple crud API lambda using mongoDb and nodeJs and it's working fine. so i passed to another step
I'm currently creating an AWS API Gateway with terraform by using an open api spec from a yaml file.
this is my swagger file:
info:
description: "API Ankan-v2"
version: "V0.0.1"
title: "ANKAN V2 API"
host: "localhost:4567"
basePath: "/restapis/api/{apiId}/test/_user_request_"
tags:
- name: "user"
description: "Operations about user"
schemes:
- "http"
# - "https"
paths:
/documents/{documentId}:
get:
tags:
- "document"
summary: "Find document by ID"
description: "Returns a single document"
operationId: "readOneDocument"
produces:
- "application/json"
parameters:
- name: "documentId"
in: "path"
description: "Id of document"
required: true
type: "string"
responses:
200:
description: "successful operation"
schema:
$ref: "#/definitions/Document"
404:
description: "document not found"
x-amazon-apigateway-integration:
uri: ${get_lambda_arn}
passthroughBehavior: "WHEN_NO_MATCH"
httpMethod: "POST"
type: "AWS_PROXY"
options:
consumes:
- "application/json"
produces:
- "application/json"
parameters:
- name: "documentId"
in: "path"
description: "Id of document"
required: true
type: "string"
responses:
200:
description: "200 response"
schema:
$ref: "#/definitions/Document"
x-amazon-apigateway-integration:
responses:
default:
statusCode: "200"
responseParameters:
method.response.header.Access-Control-Allow-Methods: "'DELETE,GET,HEAD,OPTIONS,PATCH,POST,PUT'"
method.response.header.Access-Control-Allow-Headers: "'Content-Type,Authorization,X-Amz-Date,X-Api-Key,X-Amz-Security-Token'"
method.response.header.Access-Control-Allow-Origin: "'*'"
requestTemplates:
application/json: "{\"statusCode\": 200}"
passthroughBehavior: "when_no_match"
type: "mock"
x-amazon-apigateway-any-method:
produces:
- "application/json"
parameters:
- name: "documentId"
in: "path"
description: "Id of document"
required: true
type: "string"
responses: {}
x-amazon-apigateway-integration:
responses:
default:
statusCode: "200"
uri: ${get_lambda_arn}
passthroughBehavior: "WHEN_NO_MATCH"
httpMethod: "POST"
cacheNamespace: "1rnijn"
cacheKeyParameters:
- "method.request.path.documentId"
type: "AWS"
definitions:
Document:
type: "object"
required:
- "documentName"
- "documentPath"
- "miniature"
- "status"
- "date"
properties:
_id:
type: "string"
readOnly: true
documentName:
type: "string"
documentPath:
type: "string"
miniature:
type: "string"
status:
type: "string"
enum:
- "VALID"
- "ERROR"
- "ONLOAD"
date:
type: "string"
format: "date"
xml:
name: "Document"
in my terraform i removed the integration and the method because i added them in my swagger .
so i need to import the swagger file in aws_api_gateway_rest_api with body = data.template_file.swagger.rendered
resource "aws_api_gateway_rest_api" "api" {
name = "api-gateway"
description = "document api"
body = data.template_file.swagger.rendered
}
data "template_file" "swagger" {
template = jsonencode("${path.module}/../../../../shared/swagger.yaml")
vars = {
get_lambda_arn = local.get_lambda_arn
}
}
output "json" {
value = data.template_file.swagger.rendered
}
locals {
get_lambda_arn = aws_lambda_function.lambda_document.invoke_arn
}
resource "aws_api_gateway_resource" "documents" {
rest_api_id = aws_api_gateway_rest_api.api.id
parent_id = aws_api_gateway_rest_api.api.root_resource_id
path_part = "documents"
depends_on = [aws_api_gateway_rest_api.api]
}
resource "aws_api_gateway_resource" "document_by_id" {
parent_id = aws_api_gateway_resource.documents.id
path_part = "{documentId}"
rest_api_id = aws_api_gateway_rest_api.api.id
}
resource "aws_api_gateway_deployment" "ApiDeloyment" {
rest_api_id = aws_api_gateway_rest_api.api.id
stage_name = "INT"
depends_on = [aws_api_gateway_rest_api.api]
}
resource "aws_lambda_function" "lambda_document" {
function_name = "lambda_document"
handler = "lambda.handler"
role = "arn:aws::iam::123456:role/irrelevant"
runtime = "nodejs10.x"
filename = "${path.module}/lambda.zip"
source_code_hash = filebase64sha256("${path.module}/lambda.zip")
}
After terraform apply i got this message in terraform logs
terraform apply --auto-approve
module.backend_api.module.document.aws_api_gateway_rest_api.api: Creating...
module.backend_api.module.document.aws_lambda_function.lambda_document: Creating...
module.backend_api.module.document.aws_lambda_function.lambda_document: Creation complete after 5s [id=lambda_document]
module.backend_api.module.document.data.template_file.swagger: Refreshing state...
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [10s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [20s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [30s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [40s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [50s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [1m0s elapsed]
module.backend_api.module.document.aws_api_gateway_rest_api.api: Still creating... [1m10s elapsed]
and my docker-compose logs show this
localstack | 2020-01-13T07:37:56:ERROR:localstack.services.generic_proxy: Error forwarding request: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last):
localstack | File "/opt/code/localstack/localstack/services/generic_proxy.py", line 242, in forward
localstack | path=path, data=data, headers=forward_headers)
localstack | File "/opt/code/localstack/localstack/services/apigateway/apigateway_listener.py", line 53, in forward_request
localstack | data = data and json.loads(to_str(data))
localstack | File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
localstack | return _default_decoder.decode(s)
localstack | File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
localstack | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
localstack | File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
localstack | raise JSONDecodeError("Expecting value", s, err.value) from None
localstack | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
PS : i tried with JSON but same problem
is there any solution for this? i want to integrate swagger with api gateway using terraform in localstack
This is an issue with the moto library. Under the hood, local stack uses moto to create its resources, including mock api gateway. Unfortunately moto does not currently support the body parameter for putting a swagger file definition as part of api gateway deployment. So even if you give a valid swagger file and api gateway configuration (ie one that works in aws) localstack will not be able to build it.

OpenAPI and Lambda integration within API Gateway

When I use CloudFormation to deploy an API with a Lambda integration, I'm able to dynamically link a Lambda function to an API method using standard CloudFormation syntax like !Ref and !GetAtt:
SomeMethod:
Type: AWS::ApiGateway::Method
Properties:
HttpMethod: PUT
Integration:
Type: AWS_PROXY
IntegrationHttpMethod: POST
# this is defined during deployment
Uri: !Join ["", ["arn:aws:apigateway:", !Ref "AWS::Region", ":lambda:path/2015-03-31/functions/", !GetAtt LambdaFunction.Arn, "/invocations"]]
IntegrationResponses:
- StatusCode: 200
ResourceId: !Ref APIResource
Now when I want to reference an external swagger file to define my API, which I can do via the BodyS3Location property on the AWS::ApiGateway::RestApi object, I can't understand how to dynamically link the defined methods to a Lambda function.
API as Lambda Proxy describes how to statically link a method to a Lambda function, i.e.
"x-amazon-apigateway-integration": {
"credentials": "arn:aws:iam::123456789012:role/apigAwsProxyRole",
"responses": { ... },
# this is statically defined
"uri": "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:123456789012:function:Calc/invocations",
"httpMethod": "POST",
"requestTemplates": { ... },
"type": "aws"
}
But if my Lambda function is part of the same CloudFormation stack (which it is and should be), and I want to deploy multiple instances, how can I dynamically integrate my API with Lambda? I can't use !Ref or !GetAttr because I'm outside of CloudFormation's context.

AWS SAM template/cloudformation No integration defined for method (Service: AmazonApiGateway

I am trying to deploy a lambda function and API gateway . I create a .net core web API project with AWS CLI . Deploying only the function and creating the API gateway and resource manually on aws web console does work.
If I do include the API gateway in the template, after doing SAM package deploying through web console or CLI I get the following error:
"No integration defined for method (Service: AmazonApiGateway; Status Code: 400; Error Code: BadRequestException; Request ID: ....)"
Is anything wrong or missing here?
SAM package command:
sam package --template-file sam-profile.yaml --output-template-file serverless-output.yaml --s3-bucket testapp-fewtfvdy-lambda-deployments
SAM Template:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
ProfileFunction:
Type: AWS::Serverless::Function
Properties:
Handler: testapp.Profile.NetCoreVS::testapp.Profile.NetCoreVS.LambdaEntryPoint::FunctionHandlerAsync
Runtime: dotnetcore2.0
MemorySize : 128
Timeout : 5
CodeUri: bin/Release/netcoreapp2.0/publish
Events:
ProfileAny:
Type: Api
Properties:
RestApiId: !Ref ProfileApiGateway
Path: /profile/v1
Method: GET
ProfileApiGateway:
DependsOn: ProfileFunction
Type: 'AWS::Serverless::Api'
Properties:
StageName: Prod
DefinitionUri: './swagger.yaml'
swagger.yaml:
---
swagger: '2.0'
info:
version: v1
title: ProfileAPI
paths:
"/profile/v1":
get:
tags:
- Values
operationId: ProfileV1Get
consumes: []
produces:
- text/plain
- application/json
- text/json
parameters: []
responses:
'200':
description: Success
schema:
type: array
items:
type: string
definitions: {}
.net core method:
[Route("profile/v1")]
public class ValuesController : Controller
{
// GET api/values
[HttpGet]
public IEnumerable<string> Get()
{
return new string[] { "value1", "value2","value_new3" };
}
}
Your swagger definition is missing x-amazon-apigateway-integration.
This should provide that integration layer for you:
---
swagger: '2.0'
info:
version: v1
title: ProfileAPI
paths:
"/profile/v1":
get:
tags:
- Values
operationId: ProfileV1Get
consumes: []
produces:
- text/plain
- application/json
- text/json
parameters: []
responses:
'200':
description: Success
schema:
type: array
items:
type: string
x-amazon-apigateway-integration:
httpMethod: post
type: aws_proxy
uri:
Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProfileFunction.Arn}/invocations
definitions: {}
Note that the httpMethod for x-amazon-apigateway-integration is always POST, since API Gateway always makes POST requests to Lambda regardless of what method your API route is using.

how to generate swagger document with tags using serverless-aws-documentation plugin for serverless

I am using serverless-aws-documentation plugin to auto-generate swagger-doc. Followed all the steps provided at : https://github.com/9cookies/serverless-aws-documentation. Under documentation key I am defining tags but it is not getting generated in the output swagger doc. Following is the sample handler :
functions:
get_tickets:
handler: handler.ticket_handler.get_tickets
events:
- http:
path: tickets
method: get
cors: true
documentation:
tags:
- private
- admin
summary: "Get list of ticket"
description: "This ticket will provide you list of tickets"
I want to segrigate APIs depending on the tags, but not able to achieve it. Thanks in advance for the help.
Try to add the serverless-aws-documentation plugin in the serverless.yml
plugins:
- serverless-aws-documentation
Add the infor and models documentation in the custom section:
custom:
myStage: ${opt:stage, self:provider.stage}
profiles:
dev: road-we-go
prod: road-we-
documentation:
info:
version: "1.0.0"
name: "Example API"
description: "Example API description"
termsOfService: "https://example.com/terms-of-service"
contact:
name: "Example API"
url: "https://example.com"
email: "dev#example.com"
licence:
name: "Licensing"
url: "https://example.com/licensing"
models:
-
name: "StoreAudioSuccess"
description: "Model for store audio"
contentType: "application/json"
schema: ${file(swagger/audios/storeResponse.
Add the function documentation:
If you want to add the custom models like RequestStore and StoreAudioSuccess check the serverless-aws-documentation documentation and the json-schema docs
functions:
Update:
handler: src/functions/update.handler
timeout: 30
memory: 128
events:
- http:
method: put
private: true
cors: true
path: api/v1/update
documentation:
summary: "Update "
description: "Update a record"
tags:
- "Users"
requestModels:
"application/json": "RequestStore"
methodResponses:
-
statusCode: "200"
responseModels:
"application/json": "StoreUserSuccess"
To download the swagger documentation you need to run this command:
First you need to deploy you project
sls downloadDocumentation --outputFileName=swagger.json
Which version are you using?
According to their latest documentation https://github.com/9cookies/serverless-aws-documentation, you need to provide tags as follows i.e. within double quotes.
documentation:
tags:
- "private"
- "admin"

How do I cloudform an API gateway resource with a lambda proxy integration

I've been trying to work out how to express (in cloudformation) an API Gateway Resource that has a Lambda function integration type using the Lambda Proxy integration.
This is easy to do in the AWS console as there is a check box that you can select:
However there is no corresponding field in the AWS::ApiGateway::Method CloudFormation resource (it should be in the Integration property).
How can I configure this in cloudformation?
The Integration type should be set to AWS_PROXY. An example snippet of a method from a working YAML CloudFormation template is below.
ProxyResourceAny:
Type: AWS::ApiGateway::Method
Properties:
AuthorizationType: NONE
HttpMethod: ANY
ResourceId:
Ref: ProxyResource
RestApiId:
Ref: API
Integration:
Type: AWS_PROXY
IntegrationHttpMethod: POST
Uri: !Sub
- arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Arn}/invocations
- Arn:
Fn::GetAtt:
- RestorerLambda
- Arn
It's worth saying how a I figured this out...
After scratching my head for a while I examined the output of the aws apigateway get-method CLI command for a method that was configured this way using the console. That gave me the following JSON and I realised that the checkbox might be encoded into the type. I tested my assumption and came up with the CloudFormation above.
{
"apiKeyRequired": false,
"httpMethod": "ANY",
"methodIntegration": {
"integrationResponses": {
"200": {
"responseTemplates": {
"application/json": null
},
"statusCode": "200"
}
},
"passthroughBehavior": "WHEN_NO_MATCH",
"cacheKeyParameters": [],
"uri": "arn:aws:apigateway:eu-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:eu-west-1:XXXXXXXXX:function:Shildrew-Restorer-Play-Lambda/invocations",
"httpMethod": "POST",
"cacheNamespace": "64bl3tgw4g",
"type": "AWS_PROXY"
},
"requestParameters": {},
"authorizationType": "NONE"
}
I have solved this same issue by simple changing the
Integration:
Type: AWS_PROXY
To
Integration:
Type: AWS
The cloud formation documentation currently is scarce and the API gateway cloudformation documentation doesn't match up to what can be seen on the console which hinders anyone who is trying to resolve an issue.
Hope this helps!
We faced this exact issue. We are using Ansible for our Infrastructure. Could apply to CLI or Cloudformation or even the SDK
The solution to our problem was to make sure that the Lambda policy was defined in a granular manner for the endpoint verbs in API Gateway for the lambda you are attempting to use.
For instance, We had multiple routes. Each route(or sets of routes) needs its own lambda policy defined that allows lambda:InvokeFunction. This is defined in the Lambda Policy module for Ansible. With this, the lambda trigger was enabled automatically.
There are two ways to achieve this:
Method 1: By defining the "Lambda", "API Gateway", "API Resource" and "API Methods". Linking the Lambda using the URI statement under "API Method".
MyLambdaFunction:
Type: "AWS::Lambda::Function"
Properties:
Description: "Node.js Express REST API"
FunctionName: "get_list_function" (The name in AWS console)
Handler: lambda.handler
Runtime: nodejs12
MemorySize: 128
Role: <ROLE ARN>
Timeout: 60
apiGateway:
Type: "AWS::ApiGateway::RestApi"
Properties:
Name: "example-api-gw"
Description: "Example API"
ProxyResource:
Type: "AWS::ApiGateway::Resource"
Properties:
ParentId: !GetAtt apiGateway.RootResourceId
RestApiId: !Ref apiGateway
PathPart: '{proxy+}' OR "a simple string like "PetStore"
apiGatewayRootMethod:
Type: "AWS::ApiGateway::Method"
Properties:
AuthorizationType: NONE
HttpMethod: ANY
Integration:
IntegrationHttpMethod: POST
Type: AWS_PROXY
IntegrationResponses:
- StatusCode: 200
Uri: !Sub >-
arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyLambdaFunction.Arn}/invocations
ResourceId: !Ref ProxyResource
RestApiId: !Ref "apiGateway"
METHOD 2: Define "API Gateway" and "Lambda". In Lambda definitions, call Events of type API.
CoreApi:
Type: 'AWS::Serverless::Api'
Properties:
StageName: dev
Name: SaaSAPI
EndpointConfiguration: Regional
Cors:
AllowMethods: '''POST, GET, OPTIONS, PATCH, DELETE, PUT'''
AllowHeaders: '''Content-Type, X-Amz-Date, Authorization, X-Api-Key, x-requested-with'''
AllowOrigin: '''*'''
MaxAge: '''600'''
MyLambdaFunction:
Type: "AWS::Lambda::Function"
Properties:
Description: "Node.js Express REST API"
FunctionName: "get_list_function"
Handler: lambda.handler
Runtime: nodejs12
MemorySize: 128
Role: <ROLE ARN>
Timeout: 60
Events:
ProxyResourceA:
Type: Api
Properties:
Path: /Departments
Method: GET
RestApiId: !Ref CoreApi
ProxyResourceB:
Type: Api
Properties:
Path: /Departments (This becomes the resource in API Gateway)
Method: POST
RestApiId: !Ref CoreApi
(You can link multiple methods with the same lambda, provide a unique name to each event, like ProxyResoruceA & ProxyResourceB)

Resources