ElasticsearchClientException: Request failed to execute. Call: Status code 403 from: POST /index/ - elasticsearch

I am having a problem executing a lambda that makes calls to a public ES in the context of AWS. This lambda is executed by HTTP API Gateway events. Everything seems to indicate that it is a permissions issue of the lambda trying to access the ES but my serverless configuration seems valid:
service: my-service
provider:
name: aws
runtime: dotnetcore2.1
stage: ${opt:stage, "Development"}
region: ${opt:region, "us-east-1"}
iamRoleStatements:
- Effect: "Allow"
Action:
- "lambda:invokeFunction"
Resource: "*"
- Effect: "Allow"
Action:
- es:ESHttpPost
- es:ESHttpPut
- es:ESHTTPGet
Resource: "arn:aws:es:us-east-1:account:domain/domain-name/*"
environment:
ES_ENDPOINT: ${file(appsettings.${self:provider.stage}.json):ES_ENDPOINT}
STAGE: ${self:provider.stage}
REGION: ${self:provider.region}
apiKeys:
- myservice-api-key
package:
individually: true
functions:
myservice_api:
handler: com.myservice::com.myservice.LambdaEntryPoint::FunctionHandlerAsync
package:
artifact: bin/release/netcoreapp2.1/myservice.zip
events:
- http:
path: /{proxy+}
method: ANY
cors: true
private: true
The error that is logging me is the following:
Invalid NEST response built from a unsuccessful (403) low level call on POST
OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 403 from: POST /index/_search?typed_keys=true
Package Versions:
NEST 7.4.1

I finally solved the problem. The internal Exception I found was: "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.". I solved the issue by changing the version of Elasticsearch.Net.Aws from 6.0.0 to 7.0.4.

Related

serverless offline start hot reload not working

I am unable to hot reload when using serverless offline start.
Here is my serverless.yml file
service: function-with-environment-variables
frameworkVersion: ">=3.0.0 <4.0.0"
provider:
name: aws
runtime: nodejs16.x
plugins:
- serverless-offline
functions:
register:
handler: handler.register
events:
- http:
path: /register
method: post
login:
handler: handler.login
events:
- http:
path: /login
method: post
verify:
handler: handler.verify
events:
- http:
path: /verify
method: post
I have also tried using sls offline start still facing same error.
Here is the output of serverless --version
`Running "serverless" from node_modules
Framework Core: 3.24.1 (local) 3.24.1 (global)
Plugin: 6.2.2
SDK: 4.3.2
Try using this command to start your server:
serverless offline start --reloadHandler
reloadHandler Reloads handler with each request. More info here:
https://github.com/dherault/serverless-offline/issues/864#issuecomment-1190950178

How to debug and run multiple lambdas locally

I would like to build .NET HTTP API using aws lambdas. These lambdas will be called by UI and some other systems via api gateway. Obviously in local environment I would like to run/debug these.
What I have tried:
a) Using the mock tool that comes with AWS Visual Studio templates. You can call individual lambdas but I couldn't figure out how I can call them from e.g. postman using normal rest calls. I don't know how mock tool makes those calls as chrome/firefox doesn't show them.
b) Using sam local start-api. Here is what I did:
sam --version
SAM CLI, version 1.22.0
sam init (choose aws quick start template, package type Image and amazon/dotnet5.0-base as base image)
I can build the solution with sam build, run it wit sam local start-api and I can browse to http://localhost:3000/hello and it works. Problem is that I would need to do build in VS + do those steps every time I change code. Also no easy way to attach debugger.
So what is the recommended way to do this? I know you can run whole .NET web api inside lambda but that doesn't sound like a good technical solution. I am assuming I am not the first person building HTTP api using lambdas.
It might be worth considering running a lambda-like environment in Docker.
While including the dotnet tools you need might not be feasable in actual Lambda, It might be feasible to either include them in a Docker image, or bind mounted to a docker container. These images from lambci can help with that: https://hub.docker.com/r/lambci/lambda/
You can use sam local
https://github.com/thoeni/aws-sam-local
Create API with API gateway example
Resources:
ApiGatewayToLambdaRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: ['sts:AssumeRole']
Effect: Allow
Principal:
Service: ['apigateway.amazonaws.com']
Version: '2012-10-17'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaRole
- arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs
ApiGateway:
Type: AWS::Serverless::Api
Properties:
StageName: test
EndpointConfiguration: REGIONAL
DefinitionBody:
swagger: "2.0"
info:
title: "TestAPI"
description: TestAPI description in Markdown.
paths:
/create:
post:
x-amazon-apigateway-integration:
uri:
!Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyLambda.Arn}/invocations
credentials: !GetAtt ApiGatewayToLambdaRole.Arn
responses: {}
httpMethod: POST
type: aws
x-amazon-apigateway-request-validators:
Validate query string parameters and headers:
validateRequestParameters: true
validateRequestBody: false
LambdaRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: ['sts:AssumeRole']
Effect: Allow
Principal:
Service: [lambda.amazonaws.com]
Version: '2012-10-17'
Path: /
Policies:
- PolicyName: CodeBuildAccess
PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- logs:*
- lambda:*
- ec2:CreateNetworkInterface
- ec2:DescribeNetworkInterfaces
- ec2:DeleteNetworkInterface
Effect: Allow
Resource: "*"
Version: '2012-10-17'
MyLambda:
Type: AWS::Serverless::Function
Properties:
Role: !GetAtt LambdaRole.Arn
Handler: myfunctionname.lambda_handler
CodeUri: ./src/myfunctionname
Events:
SCAPIGateway:
Type: Api
Properties:
RestApiId: !Ref ApiGateway
Path: /create
Method: POST
...
Build :
Time sam build --use-container --template backend/template.yam
Invoke Lambda Locally:
The command to invoke Lambda locally is sam local invoke and -e flag is used to specify the path to the Lambda event.
$ sam local invoke -e event.json
When it is run, it will look something like this:
$ sam local invoke MyLambda -e event.json
2021-04-20 11:11:09 Invoking index.handler
2021-04-20 11:11:09 Found credentials in shared credentials file:
~/.aws/credentials
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-invoke.html
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-start-api.html
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-debugging.html
https://github.com/ashiina/lambda-local

serverless-offline: remote Lambda custom authorizer for API Gateway

I have declared an HTTP Proxy method on my API Gateway resource using CloudFormation syntax (detailed in this earlier post.)
I am trying to attach a custom authorizer, of type "Request", which uses an existing AWS Lambda function. The serverless docs have a decent example of setting up custom authorizers using CloudFormation. Meanwhile, serverless-offline clearly supports (source) request-scoped custom authorizers.
I have closely cross-referenced the relevant AWS CloudFormation documentation for AWS::ApiGateway::Resource and AWS::ApiGateway::Authorizer, together with the related serverless docs, in creating the below serverless template. But so far, I am not getting the expected output or behavior from serverless-offline.
Resources are created without authorizers
When I boot up the below serverless configuration, no authorizers are loaded for my resource. I don't see what is wrong or missing in my serverless.yml file. Do you have any tips? Thanks in advance!
➜ serverless-offline-attempt git:(master) ✗ npm start
> # start /Users/freen/src/apig/serverless-offline-attempt
> ./node_modules/serverless/bin/serverless offline
Serverless: Starting Offline: dev/us-east-1.
Serverless: Routes defined in resources:
Serverless: ANY /upstream/{proxy*} -> http://upstream.company.cool/{proxy}
Serverless: Offline listening on http://localhost:3000
serverless.yml
The below template file includes the APIG resources and the authorizer configuration.
service: company-apig
provider:
name: aws
stage: dev
runtime: python2.7
plugins:
- serverless-offline
custom:
serverless-offline:
resourceRoutes: true
resources:
Resources:
# Parent APIG RestApi
ApiGatewayRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
Name: company-apig
Description: 'The main entry point of the APIG'
# Shared Authorizers
AuthorizeCompanyJWTAccessToken:
Type: AWS::ApiGateway::Authorizer
Properties:
Name: AuthorizeCompanyJWTAccessToken
Type: REQUEST
RestApiId:
Ref: ApiGatewayRestApi
AuthorizerUri:
Fn::Join:
- ""
-
- "arn:aws:apigateway:"
- Ref: "AWS::Region"
- ":lambda:path/2015-03-31/functions/"
- "arn:aws:lambda:us-east-1:123456789012:function:jwt-tokens-staging-AccessTokenAuthorizer"
- "/invocations"
# Resource /upstream
UpstreamResource:
Type: AWS::ApiGateway::Resource
Properties:
ParentId:
Fn::GetAtt:
- ApiGatewayRestApi
- RootResourceId
PathPart: 'upstream'
RestApiId:
Ref: ApiGatewayRestApi
# Resource /upstream/{proxy+}
UpstreamProxyPath:
Type: AWS::ApiGateway::Resource
Properties:
ParentId:
Ref: UpstreamResource
PathPart: '{proxy+}'
RestApiId:
Ref: ApiGatewayRestApi
# Method ANY /upstream/{proxy+}
UpstreamProxyAnyMethod:
Type: AWS::ApiGateway::Method
Properties:
AuthorizationType: CUSTOM
AuthorizerId:
Ref: AuthorizeCompanyJWTAccessToken
HttpMethod: ANY
Integration:
IntegrationHttpMethod: ANY
Type: HTTP_PROXY
Uri: http://upstream.company.cool/{proxy}
PassthroughBehavior: WHEN_NO_MATCH
MethodResponses:
- StatusCode: 200
ResourceId:
Ref: UpstreamProxyPath
RestApiId:
Ref: ApiGatewayRestApi
Related:
Shared Lambda authorizer setup in Serverless Framework (only loosely; the OP seeks a CloudFormation solution, but the answer does not use CF)

How to create a lambda permission for a custom websocket request authorizer with CloudFormation for API Gateway?

I've been trying to create a lambda permission for a websockets request authorizer for API Gateway and the $connect route. The AWS documenation mentions nothing about creating the proper lambda permission for a websocket authorizer. I keep getting a 500 error when I try to connect to my custom authorizer.
Since the AWS documenation (https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api-lambda-auth.html) does not mention anything about this permission, I assume one would have to create a lambda permission just like a TOKEN authorizer.
The custom authorizer was created with my CloudFormation script as follows:
# ***************************************************************
# API Gateway Websocket Authorizer
# ***************************************************************
WebsocketAuthorizer:
Type: 'AWS::ApiGatewayV2::Authorizer'
DependsOn: Lambda
Properties:
Name: WebsocketAuthorizer
ApiId:
Fn::ImportValue:
!Sub ${Env}-${AWS::Region}-altosignal-global-websockets
AuthorizerType: REQUEST
AuthorizerCredentialsArn:
Fn::ImportValue:
!Sub ${Env}-${AWS::Region}-global-iamprocesscommandsfromapigateway-arn
AuthorizerUri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Lambda.Arn}/invocations
IdentitySource:
- route.request.querystring.token
When I create an authorizer manually using the console, I get this prompt at the end:
This manually created authorizer works once I hook it up with the $connect endpoint.
So, it is this permission I'm trying to set in my CloudFormation script. I have tried the following settings but it does not work. I keep getting a 500 error:
LambdaPermission:
Type: AWS::Lambda::Permission
DependsOn:
- Lambda
- WebsocketAuthorizer
Properties:
Action: lambda:*
FunctionName: !GetAtt Lambda.Arn
Principal: apigateway.amazonaws.com
SourceArn: !Sub
- arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${apiId}/${Env}/$connect
- apiId:
Fn::ImportValue:
!Sub ${Env}-${AWS::Region}-altosignal-global-websockets
Does anybody know the proper lambda permission settings for a custom authorizer for a websocket API Gateway $connect endpoint?
You definitely need a Lambda Permission. This is what I have for setting the permission and it works well. The only difference I can see here is that the SourceArn is not given. This will hopefully give you a place to start.
Permission:
Type: "AWS::Lambda::Permission"
DependsOn:
- "WebsocketApi"
Properties:
Action: "lambda:InvokeFunction"
FunctionName: !GetAtt Lambda.Arn
Principal:
Fn::Join:
- ""
- - "apigateway."
- Ref: "AWS::URLSuffix"
I am using samcli and have this as authorizer permission and this works for me.
AuthorizerFunctionPermission:
Type: AWS::Lambda::Permission
DependsOn:
- WebSocketApi
Properties:
Action: lambda:InvokeFunction
FunctionName: arn:aws:lambda:AWS_REGION:YOUR_ACCOUT_ID:function:AUTHORIZER_FUNCTION_NAME
Principal: apigateway.amazonaws.com
Just substitute AWS_REGION, YOUR_ACCOUNT_ID, and AUTHORIZER_FUNCTION_NAME

AWS SAM/Cloudformation "Malformed integration at path' error

I get the following error when trying to deploy lambda and API gateway configuration
"malformed integration at path /profile/v1. (Service:
AmazonApiGateway; Status Code: 400; Error Code: BadRequestException"
What can cause this error and how can it be resolved.
swagger: '2.0'
info:
version: v1
title: ProfileAPI
paths:
"/profile/v1":
get:
tags:
- Values
operationId: ProfileV1Get
consumes: []
produces:
- text/plain
- application/json
- text/json
parameters: []
responses:
'200':
description: Success
schema:
type: array
items:
type: string
x-amazon-apigateway-integration:
httpMethod: post
type: aws_proxy
uri:
Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ProfileFunction.Arn}/invocations
definitions: {}
Got a final working setup working with AWS Support that y'all might be interested in:
When we reference the cloudformation resource details in the external swagger template, we do not get the resource details and hence receive the above error. For example: “ Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${LambdaFunction.Arn}/invocations “ will not work when you try to create API gateway integration endpoint uri in swagger definition using resource : "LambdaFunction.Arn” (which is the CloudFormation resource).
In order to resolve these issues, I made the below changes in the cloudformation template:
To reference the swagger file in the cloudformation template, I uploaded the swagger template in the s3 bucket and then used the below definition. I used :
ZazzoAPI:
Type: AWS::Serverless::Api
Properties:
StageName: Prod
Variables:
LambdaFunctionName: !Ref LambdaFunction
#Configure API settings and options
MethodSettings: [{
LoggingLevel: "INFO",
MetricsEnabled: True ,
HttpMethod: "GET",
ResourcePath: "/"
}]
DefinitionBody:
'Fn::Transform':
Name: 'AWS::Include'
Parameters:
Location: "s3://s3.code-deploy/swagger_SAM.yaml"
The AWS::Include transform lets you create a reference to a transform snippet in an Amazon S3 bucket. It allows to reference the cloudformation resource details in an external swagger file. You can refer to the documentation at [1] for more details regarding "AWS::Include” Transform.
I then checked the swagger template and could see that you are using shorthand notations for specifying the integration uri. However, "AWS::Include” does not currently support using shorthand notations for YAML snippets as mentioned in the documentation [2]. Therefore, I used the intrinsic function "Fn::Sub" and was able to reference the required cloudformation parameters in the swagger template.
Previous definition:
uri: !Sub "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${LambdaFunction.Arn}/invocations”
New definition:
uri:
Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${LambdaFunction.Arn}/invocations"
References:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/create-reusable-transform-function-snippets-and-add-to-your-template-with-aws-include-transform.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/create-reusable-transform-function-snippets-and-add-to-your-template-with-aws-include-transform.html#aws-include-transform-remarks
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-apigateway-stage-methodsetting.html

Resources