I am currently using AWS CloudFormation for my application. Right now I am able to auto scale the instances. Now, I want to put every thing on an Amazon VPC. Can we create a VPC using CloudFormation? And how can we manage the Elastic IP address via CloudFormation, when we have an ELB in the template? I have found a VPC related example on AWS CloudFormation Sample Templates, but it only provisions resources into an existing VPC and doesn't create a new one in the template.
Update
As pointed out by Jeff already (+1), AWS has just announced AWS CloudFormation Support for Creating VPC Resources as of April 25, 2012, covering the missing piece of their initial VPC support:
We are excited to announce that AWS CloudFormation now supports the
creation of Amazon Virtual Private Cloud (VPC) resources. [...]
Now, you can create new Virtual Private Clouds (VPC), subnets,
gateways, network ACLs, routes and route tables using CloudFormation
templates. [...]
[...] A CloudFormation can now fully represent your VPC configuration
along with all the resources needed to run your application in the
VPC.
See Jeff Barr's introductory post AWS CloudFormation Can Now Create Virtual Private Clouds for more details and examples. In particular, the AWS CloudFormation Sample Templates feature two new sample templates [...] to get you started as well:
VPC with a single EC2 Instance - Sample template showing how to create a VPC and add an EC2 instance with an Elastic IP address and a security group.
VPC with public and private subnets, an Elastic load Balancer, and an EC2 instance - Sample template showing how to create a VPC with multiple subnets. The first subnet is public and contains the load balancer, the second subnet is private and contains an EC2 instance behind the load balancer.
Initial Answer
I don't think creating an Amazon VPC with AWS CloudFormation is already supported.
While AWS has just announced AWS CloudFormation Support For VPC as of February 12, 2012 indeed, this covers existing resource types only:
All resource types such as Amazon EC2 instances, security groups and
Elastic IP addresses, Elastic Load Balancers, Auto Scaling Groups and
Amazon RDS Database instances can now be deployed into any existing
Amazon VPC using CloudFormation templates. The templates allow you to
run multi-tier web applications and corporate applications in a
private network. With Amazon VPC and CloudFormation, you can easily
control which resources you want to expose publicly and which ones
should be private.
Amazon VPC is notably absent from this list, which matches the fact that it isn't listed in the supported AWS Resource Types Reference either.
It's supported now: see AWS CloudFormation Support for Creating VPC Resources for details.
Related
Even after going through the AWS documentation and various blogs, I still don't understand how AWS lambda would behave when it is configured with VPC.
When AWS lambda configured with VPC, does that mean all instances of lambda would get the IP address from the specified subnet of that VPC?
How the ENI plays the role in AWS Lambda-VPC configuration?
The formula for ENI capacity from AWS doc -
Projected peak concurrent executions * (Memory in GB / 3GB)
Does it means AWS lambda’s running instance used to have 3 GB memory? And when that exceeds another ENI would get attached?
Most the AWS Lambda-VPC configuration related architecture diagrams shows Lambda inside VPC. Does that means Lambda would run inside VPC?
Here, I’m sure I’m missing a few pieces of information. Any pointers would be helpful.
When you configure a Lambda function to run in the VPC it uses an ENI that is created with and IP address in one of the subnets you select. Based on the formula of expected ENIs needed it seems that ENIs can be shared between lambdas.
There are only two reasons that I know of for running your lambda in a VPC.
It needs to access resources inside your VPC that do not have a public endpoint, e.g. Redis/Memcached caching clusters (Elasticache) or an RDS/Redshift cluster that doesn't have a public ip (good idea to not have public ip's on databases). When you lambda runs inside the VPC it uses a private ip and can connect to the private resources in your VPC
If you need to have your lambda's have a consistent IP address (perhaps a service that only allows whitelisting of IPs for authentication). This is achieved by using a NAT gateway.
Lambda functions cannot received inbound connections in any case.
Disadvantages of putting your lambda in a VPC are
Slower cold start times since a ENI might need to be provisioned.
You need a NAT gateway (or VPC endpoint) to access external resources
Needing to manage concurrency and available ip addresses more closely.
I am running AWS Lambda functions in a VPC.
And during the course of the project I have hit problems because:
no access to my database - had to solve this somehow
no access to AWS SES - had to find workaround
no access to AWS SQS -removed all queuing functionality from Lambda functions
no access to external Internet - still don't know how to implement ReCapthca
without Internet access
no access to AWS Cognito - cannot get
information about logged in users
I COULD implement a NAT gateway in the VPC but what is the point of serverless if I have to run a NAT server instance? That's not serverless.
So finally AWS has worn me down and I have decided to give up on running my AWS Lambda functions in a VPC - without endpoints for Internet proxying and the various AWS services its just too hard.
SO my question is - what is the downside/disadvantage of running my AWS Lambda functions with no VPC?
If you need access to resources within a VPC, then run your AWS Lambda function within a VPC. If you do not require this access, then do not run it within a VPC.
If you require Internet access, then you should connect your Lambda functions to a Private Subnet and use a NAT Gateway, which is a fully-managed NAT so you can remain serverless. It will solve the problems you listed.
AWS has provided a reference document for Lambda deployments: Serverless Application Lens, AWS Well-Architected Framework. In it they provide the following decision tree:
The only major downside noted is that a Lambda outside of a VPC cannot directly access private resources within a VPC.
One reason to create a Lambda in a VPC would be that you have a specific IP or IP range for it. This could be the case if a system just accepts calls from a specific IP which would need to be whitlistet for it.
Fix IP for Lambda function is discussed here: Is there a way to assign a Static IP to a AWS Lambda without VPC?
Downside of not having Lambda in VPC: Not having specific IP / IP-range for your Lambda function.
In the end I stayed with the VPC but I added an EC2 instance into the VPC and ran TinyProxy on it. I then configured my AWS Lambda functions with the environment variable:
HTTPS_PROXY https://ip-10-0-1-53.eu-west-1.compute.internal:8888
boto3 picked up the environment variable and sent all requests to the proxy. This seems to work fine without the complexity of a NAT gateway.
I want to run a simple Ruby Rack app (a REST API) that takes Internet requests (from iPhone clients), talks to a PostgreSQL database, and responds with JSON.
How exactly should I set this up on Amazon Web Services (AWS)?
I want the app to be able to scale to serve a growing number of clients, so I think I should use Auto Scaling with Elastic Load Balancing.
Should I use Elastic Beanstalk or manually set everything up myself?
How does the question Manual deployment vs. Amazon Elastic Beanstalk apply when setting up a Ruby Rack server with PostgreSQL?
Default vs Custom VPC
Should I just use the default VPC and use security groups to prevent direct Internet access to the EC2 & DB instances? Or, should I create a custom VPC and use private subnets, as described in Example: Launching an Elastic Beanstalk in a VPC with Amazon RDS?
Using the concept of public and private subnets adds a fantastic layer of security to your AWS application. By placing your database and application server instances in private subnets you can by design protect them from external penetration and accidental exposure.
I would recommend that you start by provisioning a VPC in 2 AZs with 1 public and 1 private subnet in each Availability Zone (4 subnets in all).
Place a NAT instance in each public subnet and update the main route table for your private subnets to send all non-vpc traffic to the NAT. This will allow instances launched into your private subnets to communicate with the WAN Internet even though they are not publicly addressable themselves.
I would recommend that you use a Multi-AZ RDS deployment for your Postgres deployment with the RDS instances in your private subnets within each AZ. This will maximize security (Postgres is not publicly accessible) and will provide you with fault tolerance (an AZ failure will not take down your app).
I would setup your Ruby app on Elastic Beanstalk. This will provide you with fault tolerance and auto-scaling. Your Elastic Beanstalk load balancers will reside in the public subnet of each AZ and your Elastic Beanstalk EC2 instances will reside in the private subnets.
My specific need is to get the list of EC2 instances in the deployment from within one of the instances.
I've tried using AWS command line for example aws elb describe-load-balancers however it would just give details of all my AWS services. I know you can specify an instances name with --load-balancer-name but I just don't have access to that from within the instance automatically.
Perhaps a file can be created on instance creation by placing something in .ebextensions?
You can do it in a two step process using the AWS CLI.
First you get the endpoint for your Elastic Beanstalk application:
aws elasticbeanstalk describe-environments --query='Environments[?ApplicationName==`Your-application-name`].EndpointURL'
Then you use the endpoint to get the instances:
aws elb describe-load-balancers --query='LoadBalancerDescriptions[?DNSName==`load-balancer-end-point-from-previous-step`].Instances[0]'
I am planning to have a multi server architecture in amazon EC2 where the servers need to talk to each other. These servers need to be located in different amazon regions (different datacenters). Can I just use the internal network of the amazon ec2? What are the security issues? Should I mandatorily use Amazon VPC in this setup.
Jam ,
If you are planning to create instances on different regions then go for VPC ,because VPC gives you more security .You can restrict these machines for limited external access also .
As security part , VPC is better than classic EC2 instances ,as you can even only allow VPC to VPC connections also .