cannot import name 'HTTPException' from 'urllib3.connection' - aws-lambda

I am building an application on AWS Lambda using serverless framework.
And trying to import requests library through requirements.txt.
But, it didn't work. came out "cannot import name 'HTTPException' from 'urllib3.connection'" error message.
I can't understand why it doesn't work.
please help.
serverless.yml
service: test-app
plugins:
- serverless-offline
- serverless-package-external
- serverless-python-requirements
custom:
stage: ${opt:stage, self:provider.stage}
pythonRequirements:
dockerizePip: false
slim: true
provider:
name: aws
runtime: python3.7
stage: dv
region: ap-northeast-2
timeout: 10
memorySize: 128
stackName: ${self:service}
variableSyntax: "\\${((?!AWS)[ ~:a-zA-Z0-9._'\",\\-\\/\\(\\)]+?)}"
profile: test-profile
role: arn:aws:iam::1234:role/role-test
environment:
domainPrefix: 'kic'
moduleName: 'deptest2'
phasePrefix: ${self:custom.stage}
projectPrefix: ‘han’
regionPrefix: 'an2'
apiName: api-an2-dv-${self:service}
vpc:
securityGroupIds:
- sg-001
subnetIds:
- subnet-001
- subnet-002
functions:
dep2:
handler: dep2_handler.dep2_handler
name: lmd-an2-dv-${self:service}-deptest2
requirements.txt
-i https://pypi.python.org/simple
requests==2.22.0
dep2_handler.py
import requests
def dep2_handler(event, context):
try:
print(event)
except Exception:
print('fail to handle event data: {}'.format(event))
return

I think the error message was a bit misleading.
As per my comment, there was an offending line in requirements.txt:
The line -i https://pypi.python.org/simple needed to be removed.

Related

AWS lambda SAM deploy error - Template format error: Unresolved resource dependencies

I have am trying to deploy an aws lambda function using the SAM cli. I have some layers defined in the sam template. Testing locally using sam local start-api works quite well. The but deploying using the sam deploy --guided command throws the following error
Error: Failed to create changeset for the stack: sam-app, ex: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state: For expression "Status" we matched expected path: "FAILED" Status: FAILED. Reason: Template format error: Unresolved resource dependencies [arn:aws:lambda:us-west-1:338231645678:layer:ffmpeg:1] in the Resources block of the template
The SAM template is as follows
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
video-processor-functions
Functions to generate gif and thumbnail from uploaded videos
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
Tracing: Active
Resources:
VideoProcessorFunctions:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: src/
Handler: app.lambdaHandler
Runtime: nodejs14.x
# timeout in seconds - 2 minutes
Timeout: 120
Layers:
- !Ref VideoProcessorDepLayer
- !Ref arn:aws:lambda:us-west-1:338231645678:layer:ffmpeg:1
Architectures:
- x86_64
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
VideoProcessorDepLayer:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: mh-video-processor-dependencies
Description: Dependencies for sam app [video-processor-functions]
ContentUri: dependencies/
CompatibleRuntimes:
- nodejs14.17
LicenseInfo: 'MIT'
RetentionPolicy: Retain
Outputs:
# ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
# Find out more about other implicit resources you can reference within SAM
# https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
VideoProcessorFunctions:
Description: "Generate GIF and Thumnail from Video"
Value: !GetAtt VideoProcessorFunctions.Arn
VideoProcessorFunctionsIamRole:
Description: "Implicit IAM Role created for MH Video Processor function"
Value: !GetAtt VideoProcessorFunctionsRole.Arn
Any ideas what i'm doing wrong?
You are trying to reference a logical id that doesn’t exist.
Just put the layer's ARN of 'ffmpeg':
Layers:
- !Ref VideoProcessorDepLayer
- arn:aws:lambda:us-west-1:338231645678:layer:ffmpeg:1

Passing StateMachine ARN through enviroment variable using Serverless.yml

While testing I am getting the following message:
Serverless Warning --------------------------------------
A valid service attribute to satisfy the declaration
'self:resources.Outputs.StateMachine.Value' could not be found.
excerpts of the serverless.yml
name: aws
runtime: nodejs12.x
stage: ${opt:stage, 'dev'}
region: us-east-1
lambdaHashingVersion: 20201221
versionFunctions: false
environment:
statemachine_arn: ${self:resources.Outputs.StateMachine.Value}
State Functions
stepFunctions:
stateMachines:
updateMetric:
name: updateMetric
definition:
Comment: "A state machine to update data"
StartAt: getMetric
........
........
Outputs
resources:
- ${file(./resources/dynamodb-table.yml)}
- Outputs:
StateMachine:
Value:
Ref: UpdateMetric
All I am trying to pass the ARN of the state machine so that I can start the stateMachine
from a lamda function.
It seems my reference in outputs section not working or I am not calling it properly here:
environment:
statemachine_arn: ${self:resources.Outputs.StateMachine.Value}

An error occurred: IamRoleLambdaExecution - Maximum policy size of 10240 bytes exceeded for role

using serverless-plugin-split-stacks in serverless.yml and getting this error
An error occurred: IamRoleLambdaExecution - Maximum policy size of 10240 bytes exceeded for role Vkonnect-dev-ap-south-1-lambdaRole (Service: AmazonIdentityManagement; Status Code: 409; Error Code: LimitExceeded; Request ID: 51920d55-4b81-4b6c-99f1-d9f0ba087cc2; Proxy: null).
when i use serverless-plugin-custom-roles i get this error
The CloudFormation template is invalid: Circular dependency between resources: [GenerateOtpDocLambdaPermissionApiGateway, DoctorUnderscorelistLambdaPermissionApiGateway .......]
serverless.yml
service: Vkonnect #Name of your App
provider:
name: aws
runtime: nodejs14.x # Node JS version
memorySize: 128
timeout: 10
stage: dev
region: ap-south-1 # AWS region
deploymentBucket:
name: vkonnectlayers
# iamRoleStatements:
# - Effect: "Allow"
# Action:
# - "s3:*"
# Resource:
# NOTE you can't refer to the LogicalID of S3Bucket, otherwise
# there will be a circular reference in CloudFormation
iamRoleStatements:
- Effect: "Allow"
Action:
- "cloudformation:*"
- "codecommit:*"
- "apigateway:*"
- "execute-api:Invoke"
- "execute-api:ManageConnections"
- "cloudformation:DescribeStacks"
- "cloudformation:ListStackResources"
- "cloudwatch:ListMetrics"
- "cloudwatch:GetMetricData"
- "ec2:DescribeSecurityGroups"
- "ec2:DescribeSubnets"
- "ec2:DescribeVpcs"
- "kms:ListAliases"
- "iam:GetPolicy"
- "iam:GetPolicyVersion"
- "iam:GetRole"
- "iam:GetRolePolicy"
- "iam:ListAttachedRolePolicies"
- "iam:ListRolePolicies"
- "iam:ListRoles"
- "lambda:*"
- "logs:DescribeLogGroups"
- "states:DescribeStateMachine"
- "states:ListStateMachines"
- "tag:GetResources"
- "xray:GetTraceSummaries"
- "xray:BatchGetTraces"
Resource:
- "*"
- "arn:aws:apigateway:*::/*"
- "arn:aws:events:*:*:rule/codecommit*"
- "arn:aws:logs:*:*:log-group:/aws/lambda/*"
plugins:
- serverless-offline
- serverless-layers
- serverless-plugin-split-stacks
- serverless-plugin-custom-roles
# - serverless-nested-stack
package:
individually: true
exclude:
- ./**
custom:
splitStacks:
perFunction: false
perType: false
perGroupFunction: true
nestedStackCount: 5
serverless-layers: # All Layers
- moment:
name: moment
excludeDevDependencies: false
individually: true
dependenciesPath: ./layers/moment-layer/package.json
package:
patterns:
- /**
- "!node_modules/**"
first create
IAM role
in your aws account with full access to the service that u want then do following
serverless.yaml
provider:
name: aws
runtime: nodejs14.x
memorySize: 128
timeout: 5
stage: prod
region: us-east-1 # AWS region
versionFunctions: false
deploymentBucket:
name: XXXXXX
iam:
role: arn:aws:iam::XXXXXX:role/full //your role arn
plugins:
- serverless-offline
- serverless-layers
- serverless-plugin-split-stacks
package:
individually: true
exclude:
- ./**
For policy size limit error:
AWS has a limit on policy size. Check this article for reference: https://aws.amazon.com/premiumsupport/knowledge-center/iam-increase-policy-size/
For circular dependency error:
Check this AWS blog: https://aws.amazon.com/blogs/infrastructure-and-automation/handling-circular-dependency-errors-in-aws-cloudformation/
AWS is setting limit on few of the resources like IAM, S3 etc. Resources should not exceed whatever the limit is set. You can submit a request to AWS Support to increase the limit.
Before that, you can go to service quota in AWS to know the limit for AWS resources. Based on that you can take a call to submit a request to AWS or follow the above document to reduce the size.

How to specify AutoPublishAlias at runtime of 'sam package/deploy'?

I am wrapping our AWS SAM deployment in Jenkins as part of our CI/CD pipeline. I only want to add the "live" alias to the lambdas when we are merging for example, yet I want "branch builds" to be without an alias. This allows developers to test the code in AWS without it being "live". Other than sed replacing part of the template.yaml before I run "sam package/deploy" is there some other way to accomplish this?
--UPDATE--
It looks like I can use Parameters to create environments in my lambda, but I don't know how to toggle between them. This would look like:
Parameters:
MyEnv:
Description: Environment of this stack of resources
Type: String
Default: testing
AllowedValues:
- testing
- prod
Then I can reference this w/:
Environment:
Variables:
ENV: !Ref: MyEnv
If someone knows how to toggle this parameter at runtime that solves my problem.
I got this working. My template.yaml:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sams-app
Globals:
Function:
Timeout: 3
Parameters:
Stage:
Type: String
Description: Which stage the code is in
Default: test
AllowedValues:
- test
- prod
Resources:
HelloWorldSQSFunction:
Type: AWS::Serverless::Function
Properties:
Role: arn:aws:iam::xxxxxxxxxxxx:role/service_lambda_default1
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python3.7
AutoPublishAlias: !Ref Stage
DeploymentPreference:
Type: AllAtOnce
Environment:
Variables:
STAGE: !Ref Stage
Outputs:
HelloWorldSQSFunction:
Description: "Hello World SQS Lambda Function ARN"
Value: !GetAtt HelloWorldSQSFunction.Arn
My lambda code:
import json
import os
def lambda_handler(event, context):
stage = os.environ['STAGE']
print(f"My stage is: {stage}")
return {
"statusCode": 200,
}
And to run it locally (I'm using Cloud9):
DEVELOPER:~/environment/sams-app $ sam local invoke --parameter-overrides Stage=prod
Invoking app.lambda_handler (python3.7)
Fetching lambci/lambda:python3.7 Docker container image......
Mounting /home/ec2-user/environment/sams-app/hello_world as /var/task:ro,delegated inside runtime container
START RequestId: 85da81b1-ef74-1b7d-6ad0-a356f4aa8b76 Version: $LATEST
My stage is: prod
END RequestId: 85da81b1-ef74-1b7d-6ad0-a356f4aa8b76
REPORT RequestId: 85da81b1-ef74-1b7d-6ad0-a356f4aa8b76 Init Duration: 127.56 ms Duration: 3.69 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 22 MB
{"statusCode":200}
One thing to note is that this will cause your "sam validate" to fail. For info on that, see: https://github.com/awslabs/serverless-application-model/issues/778
Special thanks to JLarky for the comment on this thread: aws-sam-local environment variables

How to show lambda function return values in Outputs of Cloudformation

I am looking for an option to list results of AWS::Lambda::Function in Outputs of Cloudformation.
Following is the snippet of cloudformation template for AWS:Lambda::Function
Resources:
AthenaLambdaFunction:
Type: 'AWS::Lambda::Function'
DeletionPolicy: Delete
DependsOn:
- IamRoleLambdaForAthena
Properties:
Code:
ZipFile: |
import boto3
import botocore
import os
ath = boto3.client('athena')
def handler(event, context):
outputBucket = os.environ.get("outputBucket")
QSTRING = 'select * from tableName limit 10'
response = ath.start_query_execution(QueryString=str(QSTRING), ResultConfiguration={'OutputLocation': outputBucket})
s3BucketOut = output_bucket + response['ResponseMetadata']['RequestId']
return s3BucketOut
Handler: index.handler
Runtime: python3.6
MemorySize: 128
Role: !GetAtt IamRoleLambdaForAthena.Arn
Timeout: 30
Environment:
Variables:
outputBucket: !Ref OutputS3Bucket
I want to show the value retuned by lambda function s3BucketOut in Outputs of Cloudformation. Something like below (off course, the code below doesn't work).
Outputs:
LambdaFunctionOutput:
Value: !Ref AthenaLambdaFunction.s3BucketOut
Description: Return Value of Lambda Function
Any suggestions please. TIA
You are half-way through it. With your code, you created the AWS Lambda function that you want to run. Now you need to make this function run on CloudFormation and capture its value. Note that you need to make small changes on your code to allow the value to be captured by CloudFormation.
The full code will be similar to this:
Resources:
AthenaLambdaFunction:
Type: 'AWS::Lambda::Function'
DeletionPolicy: Delete
DependsOn:
- IamRoleLambdaForAthena
Properties:
Code:
ZipFile: |
import boto3
import botocore
import os
import cfnresponse # this needs to be imported for replying to CloudFormation
ath = boto3.client('athena')
def handler(event, context):
outputBucket = os.environ.get("outputBucket")
QSTRING = 'select * from tableName limit 10'
response = ath.start_query_execution(QueryString=str(QSTRING), ResultConfiguration={'OutputLocation': outputBucket})
s3BucketOut = output_bucket + response['ResponseMetadata']['RequestId']
responseData = {} # added
responseData['S3BucketOut'] = s3BucketOut # added
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData) # return modified
Handler: index.handler
Runtime: python3.6
MemorySize: 128
Role: !GetAtt IamRoleLambdaForAthena.Arn
Timeout: 30
Environment:
Variables:
outputBucket: !Ref OutputS3Bucket
S3BucketOutInvocation:
Type: Custom::S3BucketOut
Properties:
ServiceToken: !GetAtt AthenaLambdaFunction.Arn
Region: !Ref "AWS::Region"
Outputs:
LambdaFunctionOutput:
Value: !GetAtt S3BucketOutInvocation.S3BucketOut
Description: Return Value of Lambda Function
References:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-code.html
What you can do is create what is called a "Lambda-backed Custom Resource" You would use this during Stack creation to get bits of information at creation type.
Further information can be found here
AWS Lambda-backed Custom Resources

Resources