I am programatically spinning up an EC2 instance from the AWS Java SDK v2. I then associate an elastic IP address to this instance. This takes a couple minutes. AWS then initializes the instance. In the AWS console it looks like this...
I cannot ssh into this instance until the initialization completes. Is there a way to programatically check that the instance has completed initialization? I have seen the method describeInstances(), but to the best of my knowledge, the most relevant information this method's response contains is the instance's state. The state is "running" and is not therefore probably not the variable that indicates initializing status. Maybe there is a different variable or a different method that is helpful for monitoring this initialization process? Cheers.
Not really.
Amazon EC2 is responsible for creating the virtual machine, attaching a disk with the operating system and then booting the instance. Everything after that is up to the operating system and startup scripts. The Amazon EC2 service can't 'see' what is happening on the instance.
If you particularly need to know this information, you could use a script on the instance to 'publish' this fact somewhere. For example, it could add a Tag to the instance saying State: Ready. Or, if you need to programmatically respond somehow, the script could trigger an AWS Lambda function or put a message into an Amazon SQS queue.
I check EC2 instance status after starting an instance with this Java code and the AWS SDK. An EC2 cd5.large instance takes about 90 seconds to stop and start back to the "running" state and then takes about another 3.5 minutes to complete initializing. This code is after the part which you probably have that spins the instance up to "running"...
boolean isInstanceInitialized = false;
while (!isInstanceInitialized) {
final DescribeInstanceStatusResult result = ec2
.describeInstanceStatus(
new DescribeInstanceStatusRequest()
.withInstanceIds(instanceId));
final List<InstanceStatus> instanceStatuses = result.getInstanceStatuses();
boolean isInstanceStatusOK;
boolean isSystemStatusOK;
for (final InstanceStatus instanceStatus : instanceStatuses) {
LOGGER.info(" InstanceStatus.instanceStatus: " + instanceStatus.getInstanceStatus().getStatus());
LOGGER.info(" SystemStatus: " + instanceStatus.getSystemStatus().getStatus());
isInstanceStatusOK = ("ok".equals(instanceStatus.getInstanceStatus().getStatus()));
isSystemStatusOK = ("ok".equals(instanceStatus.getSystemStatus().getStatus()));
isInstanceInitialized = isInstanceStatusOK && isSystemStatusOK;
}
if (isInstanceInitialized) {
break;
} else {
try {
// sleep 10 seconds between polling the instance state
Thread.sleep(10_000);
} catch (InterruptedException ex) {
// ignore
}
}
}
You can potentially use the waiter feature in AWS SDK for Java. EC2 supports waiters for a number of operations, one of which is waitUntilInstanceExists.
Related
I have a site I'm working on, and one of the pages retrieves data from another server, let's call it server B.
Occasionally server B fails to return data, and the main site will give a 500 error.
I want to restart server B when that happens, and I was thinking I could use CW synthetics to do that. I've created a CW alarm to trigger, but I don't have a direct way to restart an ec2 server, since it's not associated directly with one.
I've thought of calling a lambda that will restart the server, but I'm wondering if there's a simpler configuration/solution I can use.
Thanks
You can creat an Event Bridge rule for failed canary run by selecting Synthetics Canary TestRun Failure from AWS events then in Event pattern -> AWS service select Cloudwatch Syntheticsand in Event type select Synthetics Canary TestRun Failure. From the Target select AWS service -> Select a target select EC2 Rebootinstances API call and give the instance id.
UPDATED:
You can use custom patterns and pass your json which can match the failure pattern.
In your case I would use something like,
{
"source": ["aws.synthetics"],
"detail-type": ["Synthetics Canary TestRun Failure"],
"region": ["us-east-1"],
"detail": {
"account-id": ["123456789012"],
"canary-id": ["EXAMPLE-dc5a-4f5f-96d1-989b75a94226"],
"canary-name": ["events-bb-1"]
}
}
Create an Event Bridge rule for a failed canary run, that triggers a Lambda function. Have the Lambda function restart the EC2 server via the AWS API/SDK.
I have a lambda function that is running:
ssm = boto3.client('ssm')
print(ssm.describe_instance_information())
It returns 6 instances. 5 are old instances that have been terminated and no longer show up in my console anymore. One instance is correct. I created an AMI image of that instance and tried launching several instances under the same security group and subnet. None of those instances are returned from describe_instance_information. Is it reporting old data?9
My end goal is to have the lambda function launch an instance using the AMI and send a command to it. Everything works if I use the existing instance. I am trying to get it to work with one created from the AMI.
EDIT:
After a while, the instances did show up, I guess it takes a while. I dont understand why terminated instances still show up. I can poll describe_instance_information until the instance_id I want shows up but is there a cleaner built-in function, like wait_for_xxxxx()?
You can use filter parameter with PingStatus which determine Connection status of SSM Agent.
response = client.describe_instance_information(
Filters=[
{
'Key': 'PingStatus',
'Values': [
'Online'
]
},
]
)
I'm using Terraform to create my AWS infrastructure.
I've a module that creates an "aws_iam_role", an "aws_iam_role_policy", and an "aws_iam_instance_profile" and then launches an EC2 Instance with that aws_iam_instance_profile.
"terraform plan" works as expected, but with "terraform apply" I consistently get this error:
* aws_instance.this: Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::<deleted>:instance-profile/<deleted>" has no associated IAM Roles
If I immediately rerun "terraform apply", it launches the EC2 instance with no problem. If I run a "terraform graph", it does show that the instance is dependent on the profile.
Since the second "apply" is successful, that implies that the instance_policy and all that it entails is getting created correctly, doesn't it?
I've tried adding a "depends_on" and it doesn't help, but since the graph already shows the dependency, I'm not sure that is the way to go anyway.
Anyone have this issue?
Race conditions are quite common between services - where state is only eventually consistent due to scale. This is particularly true with IAM where you will often create a role and give a service such as EC2 a trust relationship to use the role for an EC2 instance, but due to however IAM is propogated throughout AWS, the role will not be available to EC2 services for a few seconds after creation.
The solution I have used, which is not a great one but gets the job done, is to put the following provisioner on every single IAM role or policy attachment to give the change time to propagate:
resource "aws_iam_role" "some_role" {
...
provisioner "local-exec" {
command = "sleep 10"
}
In this case you may use operation timeouts. Timeouts are handled entirely by the resource type implementation in the provider, but resource types offering these features follow the convention of defining a child block called timeouts that has a nested argument named after each operation that has a configurable timeout value. Each of these arguments takes a string representation of duration, such as "60m" for 60 minutes, "10s" for ten seconds, or "2h" for two hours.
resource "aws_db_instance" "example" {
# ...
timeouts {
create = "60m"
delete = "2h"
}
}
Ref: https://www.terraform.io/docs/configuration/resources.html
I would like to create a new instance based on my stored AMI.
I achieve this by the following code:
RunInstancesRequest rir = new RunInstancesRequest(imageId,1, 1);
// Code for configuring the settings of the new instance
...
RunInstancesResult runResult = ec2.runInstances(rir);
However, I cannot find a wait to "block"/wait until the instance is up and running apart from Thread.currentThread().sleep(xxxx) command.
On the other hand, StartInstancesResult and TerminateInstancesResult gives you a way to have access on the state of the instances and be able to monitor any changes. But, what about the state of a completely new instance?
boto3 has:
instance.wait_until_running()
From the boto3 docs:
Waits until this Instance is running. This method calls EC2.Waiter.instance_running.wait() which polls EC2.Client.describe_instances() every 15 seconds until a successful state is reached. An error is returned after 40 failed checks.
From the AWS CLI changelog for v1.6.0:
Add a wait subcommand that allows for a command to block until an AWS
resource reaches a given state (issue 992, issue 985)
I don't see this mentioned in the documentation, but the following worked for me:
aws ec2 start-instances --instance-ids "i-XXXXXXXX"
aws ec2 wait instance-running --instance-ids "i-XXXXXXXX"
The wait instance-running line did not finish until the EC2 instance was running.
I don't use Python/boto/botocore but assume it has something similar. Check out waiter.py on Github.
Waiting for the EC2 instance to get ready is a common pattern. In the Python library boto you also solve this with sleep calls:
reservation = conn.run_instances([Instance configuration here])
instance = reservation.instances[0]
while instance.state != 'running':
print '...instance is %s' % instance.state
time.sleep(10)
instance.update()
With this mechanism you will be able to poll when your new instance will come up.
Depending on what you are trying to do (and how many servers you plan on starting), instead of polling for the instance start events, you could install on the AMI a simple program/script that runs once when the instance starts and sends out a notification to that effect, i.e. to an AWS SNS Topic.
The process that needs to know about new servers starting could then subscribe to this SNS topic, and would receive a push notifications each time a server starts.
Solves the same problem from a different angle; your mileage may vary.
Go use Boto3's wait_until_running method:
http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Instance.wait_until_running
You can use boto3 waiters,
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#waiters
for this ex: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Waiter.InstanceRunning
Or in Java https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/
I am sure there are waiters implemented in all the AWS sdks.
I'm a bit confused about the use of the Elastic IP service offered by Amazazon Web Services. I guess the main idea is that I can switch to a new version of the web application with no downtime following this simple procedure:
Deploy the new version on a new EC2 instance
Configure the new version properly and test it using a staging DB
Once properly tested, make this new version use the live DB
Associate the Elastic IP to this instance
Terminate all the useless services (staging DB and old EC2 instance)
Is this the common way to deploy a new version of a web application?
Now, what if the application is scaled on more instances? I configured the auto scaling in the Elastic Beanstalk settings and this created a load balancer (I can it see in the EC2 section of the AWS Management Console). The problem is that I apparently cannot associate the Elastic IP with the load balancer, I have to associate it with an existing instance. To which instance should I associate it to? I'm confused...
Sorry if some questions may sound stupid but I'm only a programmer and this is the first time I set up a cloud system.
Thank you!
Elastic Load Balancing (ELB) does not work with Amazon EC2 Elastic IP addresses, in fact the two concepts do not go together at all.
Elasticity via Elastic Load Balancing
Rather, ELB is usually used via CNAME records (but see below), and this provides the first level of elasticity/availability by allowing the aliased DNS address to change the IP of the ELB(s) in use, if need be. The second level of elasticity/availability is performed by the load balancer when distributing the traffic between the EC2 instances you have registered.
Think of it this way: The CNAME never changes (just like the Elastic IP address) and the replacement of EC2 instances is handled via the load balancer, Auto Scaling, or yourself (by registering/unregistering instances).
This is explained in more detail within Shlomo Swidler's excellent analysis The “Elastic” in “Elastic Load Balancing”: ELB Elasticity and How to Test it, which in turn refers to the recently provided Best Practices in Evaluating Elastic Load Balancing by AWS, which confirm his analysis and provide a good overall read regarding the Architecture of the Elastic Load Balancing Service and How It Works in itself (but lacks the illustrative step by step samples Shlomo provides).
Domain Names
Please note that the former limitation requiring a CNAME has meanwhile been addressed by respective additions to Amazon Route 53 to allow the root domain (or Zone Apex) being used as well, see section Aliases and the Zone Apex within Moving Ahead With Amazon Route 53 for a quick overview and Using Domain Names with Elastic Load Balancing for details.
Elasticity via Elastic Beanstalk
First and foremost, AWS Elastic Beanstalk uses Elastic Load Balancing in turn as described above. On top if that, it adds application lifecycle management:
AWS Elastic Beanstalk is an even easier way for you to quickly deploy
and manage applications in the AWS cloud. You simply upload your
application, and Elastic Beanstalk automatically handles the
deployment details of capacity provisioning, load balancing,
auto-scaling, and application health monitoring. [...] [emphasis mine]
This is achieved by adding the concept of an Environment into the mix, which is explained in the Architectural Overview:
The environment is the heart of the application. [...] When you create
an environment, AWS Elastic Beanstalk provisions the resources
required to run your application. AWS resources created for an
environment include one elastic load balancer (ELB in the diagram), an
Auto Scaling group, and one or more Amazon EC2 instances.
Please note that Every environment has a CNAME (URL) that points to a load balancer, i.e. just like using an ELB on its own.
All this comes together in Managing and Configuring Applications and Environments, which discusses some of the most important features of AWS Elastic Beanstalk in detail, including usage examples using the AWS Management Console, CLI, and the APIs.
Zero Downtime
Its hard to identify the most relevant part for illustration purposes, but Deploying Versions With Zero Downtime precisely addresses your use case and implies all required preceding steps (e.g. Creating New Application Versions and Launching New Environments), so reading section AWS Management Console might give you the best overall picture how this platform works.
Good luck!
In addition to the options described in Steffen's awesome answer, Elastic Beanstalk seems to have very recently enabled Elastic IP as an option if you don't need the full features of an Elastic Load Balancer (like auto-scaling beyond one instance).
I describe the option in my answer to a similar question. Elastic Beanstalk now allows you to choose between two Environment Types, and the Single-instance option creates an Elastic IP.
I think using an ELB will be the preferable option in most cases, but e.g. for a staging server it is nice to have an alternative that is less complex (and cheaper).
Apologies for answering a post a few years later, however for those that do actually need a set of static IP addresses on an ELB, it is possible to ask AWS nicely to add what they call 'Stable IP' addresses to an ELB, and thereby give it that static IP address feature.
They don't like doing this at all of course - but will if you can justify it (the main justification is when you have clients that have IP whitelist restrictions on outbound connections via their firewalls and are completely unwilling to budge on that stance).
Just be aware that the 'auto scaling' based on traffic option isn't straight forward any more - AWS would be unable to dynamically add more ELB endpoints to your ELB as they do with the out of the box solution and you have to go through the pain of opening up new IP addresses with your customers over time.
For the original question though, EB using an ELB to front EC2 instances where static IP addresses are not actually required (no client outbound firewall issues) is the best way as per the accepted answer.
In the case that none of the above solutions works, one alternative is to attach a NAT gateway to a private subnet and associate an EIP with the NAT gateway. In this case you’re able to use the ELB, use auto-scaling, and have a reserved EIP.
This is a bit more expensive though, especially for large throughput use cases. Also, SSHing into the instance to debug becomes a bit more complex.
I wrote a post describing how to accomplish this using a Cloudwatch rule when a new instance is launched, and a lambda function. Here's the lambda function code:
const AWS = require('aws-sdk');
const ec2 = new AWS.EC2();
const PROD_ENV_NAME = 'my-prod-env-name';
// Example Event
// {
// "version": "0",
// "id": "ee376907-2647-4179-9203-343cfb3017a4",
// "detail-type": "EC2 Instance State-change Notification",
// "source": "aws.ec2",
// "account": "123456789012",
// "time": "2015-11-11T21:30:34Z",
// "region": "us-east-1",
// "resources": [
// "arn:aws:ec2:us-east-1:123456789012:instance/i-abcd1111"
// ],
// "detail": {
// "instance-id": "i-abcd1111",
// "state": "running"
// }
// }
exports.handler = async (event) => {
console.log("EVENT:", event);
// The newly launched instance ID.
const instanceId = event.detail['instance-id'];
// Fetch info about the newly launched instance
const result = await ec2.describeInstances({
Filters: [ { Name: "instance-id", Values: [instanceId] } ]
}).promise()
// The instance details are buried in this object
const instance = result.Reservations[0].Instances[0];
const isAttached = instance.NetworkInterfaces.find(int => int.Association.IpOwnerId !== 'amazon');
// Bail if the instance is already attached to another EIP
if (isAttached) {
console.log("This instance is already assigned to an elastic IP")
return { statusCode: 200, body: '' }
}
// In elastic beanstalk, the instance name gets assigned to the enviroment name.
// There is also an environment name tag, which could be used here.
const name = instance.Tags.find(t => t.Key === 'Name').Value;
// Only assign EIPs to production instances
if (name !== PROD_ENV_NAME) {
console.log('Not a production instance. Not assigning. Instance name:', name)
return { statusCode: 200, body: ''}
}
// Get a list of elastic IP addresses
const addresses = await ec2.describeAddresses().promise();
// Filter out addresses already assigned to instances
const availableAddresses = addresses.Addresses.filter(a => !a.NetworkInterfaceId);
// Raise an error if we have no more available IP addresses
if (availableAddresses.length === 0) {
console.log("ERROR: no available ip addresses");
return { statusCode: 400, body: JSON.stringify("ERROR: no available ip addresses") }
}
const firstAvail = availableAddresses[0]
try {
// Associate the instance to the address
const result = await ec2.associateAddress({
AllocationId: firstAvail.AllocationId,
InstanceId: instanceId
}).promise();
console.log('allocation result', result)
return { statusCode: 200, body: JSON.stringify('Associated IP address.') };
} catch (err) {
console.log("ERROR: ", err);
}
};
You can set the environment as a Single Instance as stated in the already accepted answer, or if you want to use an Elastic IP that you have already created, you can do the following.
Inside of the .ebextensions folder at the root of your project, make a file called setup.config and paste in the following:
container_commands:
00_setup_elastic_ip:
command: |
export AWS_ACCESS_KEY_ID={YOUR-ACCESS-KEY-ID}
export AWS_SECRET_ACCESS_KEY={YOUR-SECRET-ACCESS-KEY}
export AWS_DEFAULT_REGION=us-east-1
INSTANCE_ID=$(ec2-metadata -i)
words=( $INSTANCE_ID )
EC2_ID="${words[1]}"
aws ec2 associate-address --instance-id $EC2_ID --allocation-id {eipalloc-ID-TO-THE-IP}
All you have to do is replace the 3 parts contained inside of the {} and you are good to go. This will replace the IP of your Elastic Beanstalk instance with the Elastic IP of your choosing.
The parts contained in {} are (get rid of the {} though, that is just there to show you which parts to replace with your info):
Your AWS Access Key ID
Your AWS Secret Access Key
the allocation ID of the Elastic IP you want to assign to your Elastic Beanstalk environment's instance.