I'm a bit confused about the use of the Elastic IP service offered by Amazazon Web Services. I guess the main idea is that I can switch to a new version of the web application with no downtime following this simple procedure:
Deploy the new version on a new EC2 instance
Configure the new version properly and test it using a staging DB
Once properly tested, make this new version use the live DB
Associate the Elastic IP to this instance
Terminate all the useless services (staging DB and old EC2 instance)
Is this the common way to deploy a new version of a web application?
Now, what if the application is scaled on more instances? I configured the auto scaling in the Elastic Beanstalk settings and this created a load balancer (I can it see in the EC2 section of the AWS Management Console). The problem is that I apparently cannot associate the Elastic IP with the load balancer, I have to associate it with an existing instance. To which instance should I associate it to? I'm confused...
Sorry if some questions may sound stupid but I'm only a programmer and this is the first time I set up a cloud system.
Thank you!
Elastic Load Balancing (ELB) does not work with Amazon EC2 Elastic IP addresses, in fact the two concepts do not go together at all.
Elasticity via Elastic Load Balancing
Rather, ELB is usually used via CNAME records (but see below), and this provides the first level of elasticity/availability by allowing the aliased DNS address to change the IP of the ELB(s) in use, if need be. The second level of elasticity/availability is performed by the load balancer when distributing the traffic between the EC2 instances you have registered.
Think of it this way: The CNAME never changes (just like the Elastic IP address) and the replacement of EC2 instances is handled via the load balancer, Auto Scaling, or yourself (by registering/unregistering instances).
This is explained in more detail within Shlomo Swidler's excellent analysis The “Elastic” in “Elastic Load Balancing”: ELB Elasticity and How to Test it, which in turn refers to the recently provided Best Practices in Evaluating Elastic Load Balancing by AWS, which confirm his analysis and provide a good overall read regarding the Architecture of the Elastic Load Balancing Service and How It Works in itself (but lacks the illustrative step by step samples Shlomo provides).
Domain Names
Please note that the former limitation requiring a CNAME has meanwhile been addressed by respective additions to Amazon Route 53 to allow the root domain (or Zone Apex) being used as well, see section Aliases and the Zone Apex within Moving Ahead With Amazon Route 53 for a quick overview and Using Domain Names with Elastic Load Balancing for details.
Elasticity via Elastic Beanstalk
First and foremost, AWS Elastic Beanstalk uses Elastic Load Balancing in turn as described above. On top if that, it adds application lifecycle management:
AWS Elastic Beanstalk is an even easier way for you to quickly deploy
and manage applications in the AWS cloud. You simply upload your
application, and Elastic Beanstalk automatically handles the
deployment details of capacity provisioning, load balancing,
auto-scaling, and application health monitoring. [...] [emphasis mine]
This is achieved by adding the concept of an Environment into the mix, which is explained in the Architectural Overview:
The environment is the heart of the application. [...] When you create
an environment, AWS Elastic Beanstalk provisions the resources
required to run your application. AWS resources created for an
environment include one elastic load balancer (ELB in the diagram), an
Auto Scaling group, and one or more Amazon EC2 instances.
Please note that Every environment has a CNAME (URL) that points to a load balancer, i.e. just like using an ELB on its own.
All this comes together in Managing and Configuring Applications and Environments, which discusses some of the most important features of AWS Elastic Beanstalk in detail, including usage examples using the AWS Management Console, CLI, and the APIs.
Zero Downtime
Its hard to identify the most relevant part for illustration purposes, but Deploying Versions With Zero Downtime precisely addresses your use case and implies all required preceding steps (e.g. Creating New Application Versions and Launching New Environments), so reading section AWS Management Console might give you the best overall picture how this platform works.
Good luck!
In addition to the options described in Steffen's awesome answer, Elastic Beanstalk seems to have very recently enabled Elastic IP as an option if you don't need the full features of an Elastic Load Balancer (like auto-scaling beyond one instance).
I describe the option in my answer to a similar question. Elastic Beanstalk now allows you to choose between two Environment Types, and the Single-instance option creates an Elastic IP.
I think using an ELB will be the preferable option in most cases, but e.g. for a staging server it is nice to have an alternative that is less complex (and cheaper).
Apologies for answering a post a few years later, however for those that do actually need a set of static IP addresses on an ELB, it is possible to ask AWS nicely to add what they call 'Stable IP' addresses to an ELB, and thereby give it that static IP address feature.
They don't like doing this at all of course - but will if you can justify it (the main justification is when you have clients that have IP whitelist restrictions on outbound connections via their firewalls and are completely unwilling to budge on that stance).
Just be aware that the 'auto scaling' based on traffic option isn't straight forward any more - AWS would be unable to dynamically add more ELB endpoints to your ELB as they do with the out of the box solution and you have to go through the pain of opening up new IP addresses with your customers over time.
For the original question though, EB using an ELB to front EC2 instances where static IP addresses are not actually required (no client outbound firewall issues) is the best way as per the accepted answer.
In the case that none of the above solutions works, one alternative is to attach a NAT gateway to a private subnet and associate an EIP with the NAT gateway. In this case you’re able to use the ELB, use auto-scaling, and have a reserved EIP.
This is a bit more expensive though, especially for large throughput use cases. Also, SSHing into the instance to debug becomes a bit more complex.
I wrote a post describing how to accomplish this using a Cloudwatch rule when a new instance is launched, and a lambda function. Here's the lambda function code:
const AWS = require('aws-sdk');
const ec2 = new AWS.EC2();
const PROD_ENV_NAME = 'my-prod-env-name';
// Example Event
// {
// "version": "0",
// "id": "ee376907-2647-4179-9203-343cfb3017a4",
// "detail-type": "EC2 Instance State-change Notification",
// "source": "aws.ec2",
// "account": "123456789012",
// "time": "2015-11-11T21:30:34Z",
// "region": "us-east-1",
// "resources": [
// "arn:aws:ec2:us-east-1:123456789012:instance/i-abcd1111"
// ],
// "detail": {
// "instance-id": "i-abcd1111",
// "state": "running"
// }
// }
exports.handler = async (event) => {
console.log("EVENT:", event);
// The newly launched instance ID.
const instanceId = event.detail['instance-id'];
// Fetch info about the newly launched instance
const result = await ec2.describeInstances({
Filters: [ { Name: "instance-id", Values: [instanceId] } ]
}).promise()
// The instance details are buried in this object
const instance = result.Reservations[0].Instances[0];
const isAttached = instance.NetworkInterfaces.find(int => int.Association.IpOwnerId !== 'amazon');
// Bail if the instance is already attached to another EIP
if (isAttached) {
console.log("This instance is already assigned to an elastic IP")
return { statusCode: 200, body: '' }
}
// In elastic beanstalk, the instance name gets assigned to the enviroment name.
// There is also an environment name tag, which could be used here.
const name = instance.Tags.find(t => t.Key === 'Name').Value;
// Only assign EIPs to production instances
if (name !== PROD_ENV_NAME) {
console.log('Not a production instance. Not assigning. Instance name:', name)
return { statusCode: 200, body: ''}
}
// Get a list of elastic IP addresses
const addresses = await ec2.describeAddresses().promise();
// Filter out addresses already assigned to instances
const availableAddresses = addresses.Addresses.filter(a => !a.NetworkInterfaceId);
// Raise an error if we have no more available IP addresses
if (availableAddresses.length === 0) {
console.log("ERROR: no available ip addresses");
return { statusCode: 400, body: JSON.stringify("ERROR: no available ip addresses") }
}
const firstAvail = availableAddresses[0]
try {
// Associate the instance to the address
const result = await ec2.associateAddress({
AllocationId: firstAvail.AllocationId,
InstanceId: instanceId
}).promise();
console.log('allocation result', result)
return { statusCode: 200, body: JSON.stringify('Associated IP address.') };
} catch (err) {
console.log("ERROR: ", err);
}
};
You can set the environment as a Single Instance as stated in the already accepted answer, or if you want to use an Elastic IP that you have already created, you can do the following.
Inside of the .ebextensions folder at the root of your project, make a file called setup.config and paste in the following:
container_commands:
00_setup_elastic_ip:
command: |
export AWS_ACCESS_KEY_ID={YOUR-ACCESS-KEY-ID}
export AWS_SECRET_ACCESS_KEY={YOUR-SECRET-ACCESS-KEY}
export AWS_DEFAULT_REGION=us-east-1
INSTANCE_ID=$(ec2-metadata -i)
words=( $INSTANCE_ID )
EC2_ID="${words[1]}"
aws ec2 associate-address --instance-id $EC2_ID --allocation-id {eipalloc-ID-TO-THE-IP}
All you have to do is replace the 3 parts contained inside of the {} and you are good to go. This will replace the IP of your Elastic Beanstalk instance with the Elastic IP of your choosing.
The parts contained in {} are (get rid of the {} though, that is just there to show you which parts to replace with your info):
Your AWS Access Key ID
Your AWS Secret Access Key
the allocation ID of the Elastic IP you want to assign to your Elastic Beanstalk environment's instance.
Related
While going through IAM policy for AWS elasticsearch with terraform, i can see the options to add principals as role arns :
data "aws_iam_policy_document" "default" {
count = ( length(var.iam_role_arns) > 0) ? 1 : 0
//for resources within
statement {
effect = "Allow"
actions = distinct(compact(var.iam_actions))
resources = [
join("", aws_elasticsearch_domain.default.*.arn),
"${join("", aws_elasticsearch_domain.default.*.arn)}/*"
]
principals {
type = "AWS"
identifiers = distinct(compact(concat(var.iam_role_arns)))
}
}
}
while trying to access aws es from kibana dashboard i have to specify the condtional IPs to get access to the es domain. what should be the use case where these var.iam_role_arns are getting used?
Thanks
You would use iam_role_arns to access your ES using signed https queries by your IAM user/role credentials. By default, ES nor AWS provide an option to create such queries, but they are third party tools for that, such as popular aws-requests-auth library for python.
The library also has an example specific to ES.
This can be very useful if you want to interact with your ES through a lambda function or an EC2 instance. In these cases, your iam_role_arns would refer to lambda execution and instance roles respectively. In this scenario, you could use aws-requests-auth to construct signed queries to your ES domain.
Just check it out Signing HTTP Requests to Amazon Elasticsearch Service
I have a lambda function that is running:
ssm = boto3.client('ssm')
print(ssm.describe_instance_information())
It returns 6 instances. 5 are old instances that have been terminated and no longer show up in my console anymore. One instance is correct. I created an AMI image of that instance and tried launching several instances under the same security group and subnet. None of those instances are returned from describe_instance_information. Is it reporting old data?9
My end goal is to have the lambda function launch an instance using the AMI and send a command to it. Everything works if I use the existing instance. I am trying to get it to work with one created from the AMI.
EDIT:
After a while, the instances did show up, I guess it takes a while. I dont understand why terminated instances still show up. I can poll describe_instance_information until the instance_id I want shows up but is there a cleaner built-in function, like wait_for_xxxxx()?
You can use filter parameter with PingStatus which determine Connection status of SSM Agent.
response = client.describe_instance_information(
Filters=[
{
'Key': 'PingStatus',
'Values': [
'Online'
]
},
]
)
I am programatically spinning up an EC2 instance from the AWS Java SDK v2. I then associate an elastic IP address to this instance. This takes a couple minutes. AWS then initializes the instance. In the AWS console it looks like this...
I cannot ssh into this instance until the initialization completes. Is there a way to programatically check that the instance has completed initialization? I have seen the method describeInstances(), but to the best of my knowledge, the most relevant information this method's response contains is the instance's state. The state is "running" and is not therefore probably not the variable that indicates initializing status. Maybe there is a different variable or a different method that is helpful for monitoring this initialization process? Cheers.
Not really.
Amazon EC2 is responsible for creating the virtual machine, attaching a disk with the operating system and then booting the instance. Everything after that is up to the operating system and startup scripts. The Amazon EC2 service can't 'see' what is happening on the instance.
If you particularly need to know this information, you could use a script on the instance to 'publish' this fact somewhere. For example, it could add a Tag to the instance saying State: Ready. Or, if you need to programmatically respond somehow, the script could trigger an AWS Lambda function or put a message into an Amazon SQS queue.
I check EC2 instance status after starting an instance with this Java code and the AWS SDK. An EC2 cd5.large instance takes about 90 seconds to stop and start back to the "running" state and then takes about another 3.5 minutes to complete initializing. This code is after the part which you probably have that spins the instance up to "running"...
boolean isInstanceInitialized = false;
while (!isInstanceInitialized) {
final DescribeInstanceStatusResult result = ec2
.describeInstanceStatus(
new DescribeInstanceStatusRequest()
.withInstanceIds(instanceId));
final List<InstanceStatus> instanceStatuses = result.getInstanceStatuses();
boolean isInstanceStatusOK;
boolean isSystemStatusOK;
for (final InstanceStatus instanceStatus : instanceStatuses) {
LOGGER.info(" InstanceStatus.instanceStatus: " + instanceStatus.getInstanceStatus().getStatus());
LOGGER.info(" SystemStatus: " + instanceStatus.getSystemStatus().getStatus());
isInstanceStatusOK = ("ok".equals(instanceStatus.getInstanceStatus().getStatus()));
isSystemStatusOK = ("ok".equals(instanceStatus.getSystemStatus().getStatus()));
isInstanceInitialized = isInstanceStatusOK && isSystemStatusOK;
}
if (isInstanceInitialized) {
break;
} else {
try {
// sleep 10 seconds between polling the instance state
Thread.sleep(10_000);
} catch (InterruptedException ex) {
// ignore
}
}
}
You can potentially use the waiter feature in AWS SDK for Java. EC2 supports waiters for a number of operations, one of which is waitUntilInstanceExists.
When running Kubernetes on AWS, exposing a Service with "type=LoadBalancer" works well. However, the name given to the Elastic Load Balancer is a rather long hash and as such, it is hard to keep track through the AWS console as to which load balancer belongs to which Service.
Is it possible to specify he name of the ELB object at service creation time?
If not, I might create an issue that the service name be used when creating the ELB.
On a related note, is it possible to modify the security group (firewall) that the load balancer uses?
The tags of the ELB contain the information you're looking for.
$ aws elb describe-tags --load-balancer-names xxxxx
{
"TagDescriptions": [
{
"LoadBalancerName": "xxxxx",
"Tags": [
{
"Value": "default/nginx",
"Key": "kubernetes.io/service-name"
},
{
"Value": "my-cluster",
"Key": "KubernetesCluster"
}
]
}
]
}
If you want to give the ELB a proper domain name, you can assign one using Route53. It can be automated with something like route53-kubernetes.
I've created new EC2 spot requests over the last weeks. A new security group was created for every request. When the spot requests were deleted the security groups were not deleted. I've hit the 100 groups limit and want to delete them. The EC2 interface apparently allows only one deletion at a time, which means I would have to make 300 clicks to delete these groups. Or is there a better way to delete multiple security groups with few clicks or lines of code?
THis would need some basic scripting and AWS SDK. you can do this with pretty much all the SDK provided by AWS.
I would prefer AWS-CLI as i already have it installed and configured. This is what I would do:
list all the SGs with describe-security-groups
Install jq (the Json parser for BASH)
Pull the SG IDs (check this for jq syntax)
Once you have the SG IDs, run delete-security-group by usig a for loop.
This is fairly simple and straight forward way of doing wat you want to do. THis can be done by any of the AWS SDKs.
These are just a couple of commands which can be constructed into a Bash script, provided:
You have aws-cli installed and configured
you have jq installed on your system.
If you already have some other AWS SDK installed, then you are better off with that as java/python/ruby...etc all have their own inbuilt way of parsing JSON/HASH/DataStructure.
Hope this helps.
I think you can do this by combining a command that lists all security groups and one other that deletes them.
If you are using the python boto API (for example) that would be:
import boto
conn = boto.connect_ec2(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
groups = conn.get_all_security_groups()
Which returns (as an example): [SecurityGroup:appserver, SecurityGroup:default, SecurityGroup:vnc, SecurityGroup:webserver]
And then you delete them all:
for group in groups:
conn.delete_security_group(group.split(":")[1])
Edit
You should run the commands on your shell.
These solutions only work if you don't have rules in other security groups that reference the security groups in question. I have a couple scripts that will delete a single security group, including the ingress rules in other security groups. I also handle the special case of ingress rules referencing the AWS ELB default security group. If you have this more complex situation, the solutions above won't delete your security group because of these other rules. My scripts are here (one for ec2-classic and one for VPC based security groups): https://gist.github.com/arpcefxl/2acd7d873b95dbebcd42
private static void delete(List<String> sgs) {
AmazonEC2Client ec2 = new AmazonEC2Client(Credentials.getCredentialsProvider());
ec2.setEndpoint("ec2.us-west-2.amazonaws.com"); // default
for(String sg:sgs) {
System.out.println("DELETING SECURITY GROUP " + sg);
DeleteSecurityGroupRequest delReq = new DeleteSecurityGroupRequest().withGroupName(sg);
try {
ec2.deleteSecurityGroup(delReq);
} catch (Exception e) {
// e.printStackTrace();
}
}
}