I have a VPC enabled Lambda function which attempts to launch an EC2 using a launch template. The EC2 launch step (run_instances) fails with the below generic network error.
Calling the invoke API action failed with this message: Network Error
I can launch an instance successfully directly from the launch template, so I think everything is fine with the launch template. I have configured the following in the launch template
Amazon Machine Image ID
Instance type
Key Pair
A network interface (ENI) which I had created before using a specific (VPC, Subnet, Secutity Group) combo.
IAM role
The Lambda function includes the below code-
import json
import boto3
import time
def lambda_handler(event, context):
ec2_cl = boto3.client('ec2')
launch_temp = {"LaunchTemplateId": "<<Launch Template ID>>"}
resp_ec2_launch = ec2_cl.run_instances(MaxCount=1, MinCount=1, LaunchTemplate=launch_temp, SubnetId="<<Subnet ID>>")
Few things on the Lambda function-
I have used the subnet in the run_instances() call because this is not the default vpc/subnet.
The function is setup with the same (VPC, Subnet, Secutity Group) combo as used in the launch template
The execution role is setup to be the same IAM role as used in the launch template
The function as you see needs access only to the EC2, internet access is not needed
I replaced the run_instances() with describe_instance_status (using the instance id created directly from the launch template) and got the same error.
The error is a network error, so I assume all is fine (atleast as of now) with the privileges granted to the IAM role. I'm sure there would be a different error, if the IAM role missed any policies.
Can someone indicate what I might be missing?
It appears that the problem is with your AWS Lambda function being able to reach the Internet, since the Amazon EC2 API endpoint is on the Internet.
If a Lambda function is not attached to a VPC, it has automatic access to the Internet.
If a Lambda function is attached to a VPC and requires Internet access, then the configuration should be:
Attach the Lambda function only to private subnet(s)
Launch a NAT Gateway in a public subnet
Configure the Route Table on the private subnets to send Internet-bound traffic (0.0.0.0/0) through the NAT Gateway
It appears that your VPC does not have an Internet Gateway, but it does have a VPC Endpoint for EC2.
Therefore, to try and reproduce your situation, I did the following:
Created a new VPC with one subnet but no Internet Gateway
Added a VPC Endpoint for EC2 to the subnet
Created a Lambda function that would call DescribeInstances() and attached the Lambda function to the subnet
Opened the security group on the VPC Endpoint and Lambda function to allow all traffic from anywhere (hey, it's just a test!)
My Lambda function:
import json
import boto3
def lambda_handler(event, context):
ec2 = boto3.client('ec2',region_name='ap-southeast-2')
print(ec2.describe_instances())
The result: The Lambda function successfully received a response from EC2, with a list of instances in the region. No code or changes were required.
Related
I am trying to trigger code pipeline from lambda using below link got the lambda python code as well.
https://aws.amazon.com/blogs/devops/adding-custom-logic-to-aws-codepipeline-with-aws-lambda-and-amazon-cloudwatch-events/
But somehow while running i am getting exception as
Connect timeout on endpoint URL "https://codepipeline.ap-southeast-2.amazonaws.com/"
I have opened all traffic using security group attached to lambda.
Please suggest what else to check here.
Thanks
Sharad
You are running your Lambda function in a VPC (as evidenced by the fact that you said it has a security group attached). A Lambda function in a VPC cannot access anything outside the VPC without a route to a NAT Gateway. A Lambda function in a VPC never gets a public IP assigned to it, so it can never use a VPC Internet Gateway directly. Thus to access anything outside your VPC, such as the AWS API to trigger a CodePipeline run, the Lambda function needs to be deployed only in subnets of your VPC that have a route to a NAT Gateway.
The alternative would be to add a VPC Endpoint for the specific AWS Service you want to access.
I am trying to launch an EMR cluster using AWS Lambda code written with boto3 and python. The Lambda is able to launch the cluster when there is no VPC configuration associated it. However, as soon as I add the VPC config it fails to launch the cluster and errors out and does not provide any error message.
I am trying to launch the lambda inside a default VPC and it has 3 public subnets and a default security group. I have checked the route table in the VPC is associated with an internet gateway and it is attached to the VPC.
The execution role provides full access to the cloudwatch elasticmapreduce and ec2 actions.
Any help in resolving this school boy error will be much appreciated.
hello I want to call a api which is hoisted in Apache server on AWS EC2 server from my lambda function.
I have configured VPC for my lambda function according to my EC2 server instance with same security groups and same zone ,but still no luck.
Thanks in advance.
It appears that your situation is:
You want to make an HTTP/S API call from an AWS Lambda function to an Amazon EC2 instance
The Lambda function has been configured to connect to the same VPC as the EC2 instance
The Lambda function is reporting that it is unable to connect to the EC2 instance
I would recommend:
Create a Security Group (let's call it Lambda-SG) in the same VPC. It does not need any rules.
Configured the Lambda function to use the Security Group (in the Network section)
Create another Security Group (let's call it EC2-SG) and associate it with the EC2 instance. Add a rule that permits inbound access on the desired port, with the source set to ID of Lambda-SG.
That is the EC2-SG is saying that it will accept inbound connects from Lambda-SG on the desired port.
Merely putting resources in the same security group does not necessarily mean they can communicate. There needs to be a specific rule that permits the inbound access. The above method is the best way to do so.
The EC2 instance in question is in private subnet, and does not have Internet access enabled through NAT by design.
Currently, my Lambda is in a public subnet connected to the same VPC.
When I try to invoke Lambda, I receive Connection to lambda.us-east-1.amazonaws.com timed out.
Amazon EC2 instances cannot "access a Lambda function", but they can certainly call the AWS Lambda API to invoke a Lambda function.
However, the API endpoints for AWS services all reside on the Internet. Therefore, calling an API requires access to the Internet. (Two exceptions to this are currently Amazon S3 and Amazon DynamoDB, which can use a VPC Endpoint to access the API endpoints.)
Therefore, to invoke the Lambda function from an Amazon EC2 instance in a private subnet, you will need a NAT Gateway and a private route table configured to send Internet-bound traffic to the NAT Gateway.
The fact that the Lambda function is connected to your public subnet is irrelevant for invoking a Lambda function. It simply means that the Lambda function, when running, can access resources within the VPC. (In fact, Lambda functions are normally connected to private subnets rather than public subnets, since they can only access the Internet if their traffic is routed through a NAT Gateway.)
When you try to invoke the Lambda function it doesn't matter where the Lambda function is at all. That's because you aren't connecting directly to the Lambda function when you are invoking it. You are connecting to the AWS API to invoke the Lambda function. The only way you would be able to do this from an EC2 instance in a private subnet would be through a NAT Gateway.
I am running AWS Lambda functions in a VPC.
And during the course of the project I have hit problems because:
no access to my database - had to solve this somehow
no access to AWS SES - had to find workaround
no access to AWS SQS -removed all queuing functionality from Lambda functions
no access to external Internet - still don't know how to implement ReCapthca
without Internet access
no access to AWS Cognito - cannot get
information about logged in users
I COULD implement a NAT gateway in the VPC but what is the point of serverless if I have to run a NAT server instance? That's not serverless.
So finally AWS has worn me down and I have decided to give up on running my AWS Lambda functions in a VPC - without endpoints for Internet proxying and the various AWS services its just too hard.
SO my question is - what is the downside/disadvantage of running my AWS Lambda functions with no VPC?
If you need access to resources within a VPC, then run your AWS Lambda function within a VPC. If you do not require this access, then do not run it within a VPC.
If you require Internet access, then you should connect your Lambda functions to a Private Subnet and use a NAT Gateway, which is a fully-managed NAT so you can remain serverless. It will solve the problems you listed.
AWS has provided a reference document for Lambda deployments: Serverless Application Lens, AWS Well-Architected Framework. In it they provide the following decision tree:
The only major downside noted is that a Lambda outside of a VPC cannot directly access private resources within a VPC.
One reason to create a Lambda in a VPC would be that you have a specific IP or IP range for it. This could be the case if a system just accepts calls from a specific IP which would need to be whitlistet for it.
Fix IP for Lambda function is discussed here: Is there a way to assign a Static IP to a AWS Lambda without VPC?
Downside of not having Lambda in VPC: Not having specific IP / IP-range for your Lambda function.
In the end I stayed with the VPC but I added an EC2 instance into the VPC and ran TinyProxy on it. I then configured my AWS Lambda functions with the environment variable:
HTTPS_PROXY https://ip-10-0-1-53.eu-west-1.compute.internal:8888
boto3 picked up the environment variable and sent all requests to the proxy. This seems to work fine without the complexity of a NAT gateway.