I have a beautiful Lambda setup with Cloudwatch Events triggering the functions on a schedule.
I am starting to move my Lambda functions into VPC, but I am having a dickens of a time getting Cloudwatch Events to properly trigger Lambda functions. They continually time out.
Is there a way to use CW Events to trigger VPC Lambda functions? If not, is there a canonical solution to triggering Lambda functions on schedule?
This issue isn't related to Cloudwatch Events. The source of the function trigger doesn't modify how Lambda actually invokes the function.
The problem is almost certainly that your functions don't currently work inside your VPC, and this in turn because you have not configured the VPC as needed for Internet access -- and most of the various AWS service APIs are accessed via an Internet connection. A NAT device is required, because your Lambda containers have only private IP addresses.
Important
If your Lambda function needs Internet access, do not attach it to a public subnet or to a private subnet without Internet access. Instead, attach it only to private subnets with Internet access through a NAT instance or an Amazon VPC NAT gateway.
http://docs.aws.amazon.com/lambda/latest/dg/vpc.html
The functions will otherwise time out the first time and any time your code attempts to access any resource outside the VPC.
Related
I am trying to trigger code pipeline from lambda using below link got the lambda python code as well.
https://aws.amazon.com/blogs/devops/adding-custom-logic-to-aws-codepipeline-with-aws-lambda-and-amazon-cloudwatch-events/
But somehow while running i am getting exception as
Connect timeout on endpoint URL "https://codepipeline.ap-southeast-2.amazonaws.com/"
I have opened all traffic using security group attached to lambda.
Please suggest what else to check here.
Thanks
Sharad
You are running your Lambda function in a VPC (as evidenced by the fact that you said it has a security group attached). A Lambda function in a VPC cannot access anything outside the VPC without a route to a NAT Gateway. A Lambda function in a VPC never gets a public IP assigned to it, so it can never use a VPC Internet Gateway directly. Thus to access anything outside your VPC, such as the AWS API to trigger a CodePipeline run, the Lambda function needs to be deployed only in subnets of your VPC that have a route to a NAT Gateway.
The alternative would be to add a VPC Endpoint for the specific AWS Service you want to access.
Even after going through the AWS documentation and various blogs, I still don't understand how AWS lambda would behave when it is configured with VPC.
When AWS lambda configured with VPC, does that mean all instances of lambda would get the IP address from the specified subnet of that VPC?
How the ENI plays the role in AWS Lambda-VPC configuration?
The formula for ENI capacity from AWS doc -
Projected peak concurrent executions * (Memory in GB / 3GB)
Does it means AWS lambda’s running instance used to have 3 GB memory? And when that exceeds another ENI would get attached?
Most the AWS Lambda-VPC configuration related architecture diagrams shows Lambda inside VPC. Does that means Lambda would run inside VPC?
Here, I’m sure I’m missing a few pieces of information. Any pointers would be helpful.
When you configure a Lambda function to run in the VPC it uses an ENI that is created with and IP address in one of the subnets you select. Based on the formula of expected ENIs needed it seems that ENIs can be shared between lambdas.
There are only two reasons that I know of for running your lambda in a VPC.
It needs to access resources inside your VPC that do not have a public endpoint, e.g. Redis/Memcached caching clusters (Elasticache) or an RDS/Redshift cluster that doesn't have a public ip (good idea to not have public ip's on databases). When you lambda runs inside the VPC it uses a private ip and can connect to the private resources in your VPC
If you need to have your lambda's have a consistent IP address (perhaps a service that only allows whitelisting of IPs for authentication). This is achieved by using a NAT gateway.
Lambda functions cannot received inbound connections in any case.
Disadvantages of putting your lambda in a VPC are
Slower cold start times since a ENI might need to be provisioned.
You need a NAT gateway (or VPC endpoint) to access external resources
Needing to manage concurrency and available ip addresses more closely.
The EC2 instance in question is in private subnet, and does not have Internet access enabled through NAT by design.
Currently, my Lambda is in a public subnet connected to the same VPC.
When I try to invoke Lambda, I receive Connection to lambda.us-east-1.amazonaws.com timed out.
Amazon EC2 instances cannot "access a Lambda function", but they can certainly call the AWS Lambda API to invoke a Lambda function.
However, the API endpoints for AWS services all reside on the Internet. Therefore, calling an API requires access to the Internet. (Two exceptions to this are currently Amazon S3 and Amazon DynamoDB, which can use a VPC Endpoint to access the API endpoints.)
Therefore, to invoke the Lambda function from an Amazon EC2 instance in a private subnet, you will need a NAT Gateway and a private route table configured to send Internet-bound traffic to the NAT Gateway.
The fact that the Lambda function is connected to your public subnet is irrelevant for invoking a Lambda function. It simply means that the Lambda function, when running, can access resources within the VPC. (In fact, Lambda functions are normally connected to private subnets rather than public subnets, since they can only access the Internet if their traffic is routed through a NAT Gateway.)
When you try to invoke the Lambda function it doesn't matter where the Lambda function is at all. That's because you aren't connecting directly to the Lambda function when you are invoking it. You are connecting to the AWS API to invoke the Lambda function. The only way you would be able to do this from an EC2 instance in a private subnet would be through a NAT Gateway.
I am running AWS Lambda functions in a VPC.
And during the course of the project I have hit problems because:
no access to my database - had to solve this somehow
no access to AWS SES - had to find workaround
no access to AWS SQS -removed all queuing functionality from Lambda functions
no access to external Internet - still don't know how to implement ReCapthca
without Internet access
no access to AWS Cognito - cannot get
information about logged in users
I COULD implement a NAT gateway in the VPC but what is the point of serverless if I have to run a NAT server instance? That's not serverless.
So finally AWS has worn me down and I have decided to give up on running my AWS Lambda functions in a VPC - without endpoints for Internet proxying and the various AWS services its just too hard.
SO my question is - what is the downside/disadvantage of running my AWS Lambda functions with no VPC?
If you need access to resources within a VPC, then run your AWS Lambda function within a VPC. If you do not require this access, then do not run it within a VPC.
If you require Internet access, then you should connect your Lambda functions to a Private Subnet and use a NAT Gateway, which is a fully-managed NAT so you can remain serverless. It will solve the problems you listed.
AWS has provided a reference document for Lambda deployments: Serverless Application Lens, AWS Well-Architected Framework. In it they provide the following decision tree:
The only major downside noted is that a Lambda outside of a VPC cannot directly access private resources within a VPC.
One reason to create a Lambda in a VPC would be that you have a specific IP or IP range for it. This could be the case if a system just accepts calls from a specific IP which would need to be whitlistet for it.
Fix IP for Lambda function is discussed here: Is there a way to assign a Static IP to a AWS Lambda without VPC?
Downside of not having Lambda in VPC: Not having specific IP / IP-range for your Lambda function.
In the end I stayed with the VPC but I added an EC2 instance into the VPC and ran TinyProxy on it. I then configured my AWS Lambda functions with the environment variable:
HTTPS_PROXY https://ip-10-0-1-53.eu-west-1.compute.internal:8888
boto3 picked up the environment variable and sent all requests to the proxy. This seems to work fine without the complexity of a NAT gateway.
I have a Lambda function that needs to read messages from an SQS queue using it's URL. Then it needs to insert that data to Cassandra running on a server inside a VPC.
I am able to access the Cassandra server from my Lambda function, using it's private IP and configuring the security groups correctly.
However, I am not able to read messages from the SQS Queue. When I change the configuration of Lambda function to No VPC, then I am able to read the messages from the SQS Queue. However, with VPC settings, it just times out.
How can I overcome this ? I have checked the security group of my Lambda function has full outbound access to all IP addresses.
At the end of 2018, AWS announced support for SQS endpoints which provide
connectivity to Amazon SQS without requiring an internet gateway, network address translation (NAT) instance, or VPN connection.
There is a tutorial for Sending a Message to an Amazon SQS Queue from Amazon Virtual Private Cloud
See also the SQS VPC Endpoints Documentation for more information.
Its important to note that if you want to access SQS within the Lambda VPC there are a couple other things you need to do:
Make sure to specify the SQS region in your code. For example, I had to set my endpoint_url to "https://sqs.us-west-2.amazonaws.com"
Make sure that you have attached a "wide open" security group to the SQS VPC Interface, otherwise SQS will not work.
Make sure that your subnets in your Lambda VPC match what you have set up for your SQS VPC Interface.
Some services (e.g. S3) are offering VPC endpoints to solve this particular problem but SQS is not one of them. I think the only real solution to this problem is to run a NAT inside your VPC so the network traffic from the Lambda function can be routed to the outside world.
I ran into the same kind of problem when I was running lambda function with access to elasticache on the VPC. While the function was configured to run in the VPC, I wasnt able to talk to any other service (specifically codedeploy for me).
As #garnaat pointed out NAT seems to be the only way to go about solving this problem for services without VPC endpoints.
And like you pointed out, I also ran into the same trouble where I could'nt SSH into the machine(s) once I replaced the entry with the IGW in the route table. Seems like detaching the IGW starves the VPC of either the incoming traffic (mostly) or the outgoing traffic from or to the internet respectively. So here's what I did and it worked for me:
Create a new Subnet within the VPC
Now, when lambda runs, make sure lambda operates from this subnet.
You can do this by using aws-cli like so:
aws lambda update-function-configuration --function-name your-function-name --vpc-config SubnetIds="subnet-id-of-created-subnet",SecurityGroupIds="sg-1","sg-2"
Make sure you add all the security groups whose inbound and outbound traffic rules apply for your lambda function.
Next, go to Route Tables in the VPC console and create a new route table.
Here is where you add the NAT gateway to the target.
finally go to the Subnet Associations tab in the new route table and add the newly created subnet there.
Thats all this should get it working . Mind you, please treat this as only a workaround. I haven't done much digging and I have a very limited idea on how things get resolved internally while doing this. This might not be an ideal solution.
The ideal solution seems to be to design the VPC before hand. Use subnets to isolate resources/instances that need internet access and that dont(private and public subnets) and place appropriate gateways where needed.( so that you may not have to create a seperate subnet for this purpose later). Thanks
I was unable to get either of the other two answers to this question to work. Perhaps this is due to one or more mistakes on my part. Regardless, I did find a workaround that I wanted to share, in case I'm not alone with this problem.
Solution: I created two Lambda functions. The first Lambda function runs inside my VPC and performs the desired work (in mandeep_m91's case, that's a data insertion to Cassandra; in my case is was accessing an RDS instance). The second Lambda function lives outside the VPC, so I could hook it up to the SQS queue. I then had the second Lambda function call the first, using the information found this this StackOverflow Q&A answer. Note, the linked question has both node.js and Python examples in the answers.
This will effectively double the cost of making a function call, since each call results in two function executions. However, for my situation, the volume is so low it won't make a real difference.
To clarify a point above about a "wide open" security group, the group set on the endpoint needs to allow inbound access to SQS from your lambda function.
I created a security group for my endpoint that only opened 443 to my lambda's security group.