I have been struggling several days with Auto scaling Ec2 instances. Please help....
Here are the steps:
Create a Launch Template (nothing special here)
Create an Auto Scaling Group by using the lauch template created in #1.
Create an alarm in CloudWatch (When CPU Utilization > 30 in consecutive 60s, trigger action)
Create a policy(add one unit) in autoScalingGroup created in #2. Attach the policy to the alarm.
Lauch ec2 instance and use script to make cpu utilization > 60
Result:
No new Ec2 instance is created when CPU utilization of existing Ec2 instance is higher than 30.
Alarm status is always "Insufficient Data" when running the existing Ec2 instance more than 30 mins with CPU utilization higher than 30.
Even changing the option of alerm to "Treat missing data as good (not breaching threshold)", still no new Ec2 instance is created.
Made some progress, Even created a new instance manually without AutoScaling, Alarm in cloudWatch not working. Not sure why. I strictly followed the instructions on an Yutube success example.
Update at 18th Feb.
It's ... that Cloudwatch delayed 4 mins to send out alarm email.
Run "stress xxx &" script at 10:30:32 (this is shown in console when running date command)
Alarm triggered an email. Email was received with content "Friday 18 February, 2022 10:36:03 UTC"
Related
I have created a windows service exe from the python code, it starts when I start it manually in AWS Ec2 instances. Also Starts automatically some time when the ec2 boots up. But sometimes the service will not be stared in the instance, why is it happening some times. For your info I also increased the timeout to service start till 700000 in regedit key. still the service will not start automatically. Why is that happening? can I get some solution for this?
If the service is set to start automatically at boot but it isn't, there should be a record describing the failure in the "System" area in the Event Viewer. Check those logs.
Also, try setting the service's "Startup type" to "Automatic (Delayed Start)". Doing so will delay service startup by a couple of minutes, which may be enough to fix the problem if it is a "race condition" as the system starts.
How to automate bringing my RDS database offline and online via scripts/ANY PROCEDURE to save money? Note before to OFFLINE, I need to take a final SNAP SHOT,want to AUTOMATE this also.
The steps as follows.. 1)Take a SNAPSHOT 2)Shutdown Instance 3)STARTUP Instance in morning with same DATA available for Business.
When I put a RDS offline to stop the "work hours" counter running and billing me, when I bring it back online will it still have the same content (i.e will all my data stay there, or will it have to be a blank DB?).
I'm using Terraform to create my AWS infrastructure.
I've a module that creates an "aws_iam_role", an "aws_iam_role_policy", and an "aws_iam_instance_profile" and then launches an EC2 Instance with that aws_iam_instance_profile.
"terraform plan" works as expected, but with "terraform apply" I consistently get this error:
* aws_instance.this: Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::<deleted>:instance-profile/<deleted>" has no associated IAM Roles
If I immediately rerun "terraform apply", it launches the EC2 instance with no problem. If I run a "terraform graph", it does show that the instance is dependent on the profile.
Since the second "apply" is successful, that implies that the instance_policy and all that it entails is getting created correctly, doesn't it?
I've tried adding a "depends_on" and it doesn't help, but since the graph already shows the dependency, I'm not sure that is the way to go anyway.
Anyone have this issue?
Race conditions are quite common between services - where state is only eventually consistent due to scale. This is particularly true with IAM where you will often create a role and give a service such as EC2 a trust relationship to use the role for an EC2 instance, but due to however IAM is propogated throughout AWS, the role will not be available to EC2 services for a few seconds after creation.
The solution I have used, which is not a great one but gets the job done, is to put the following provisioner on every single IAM role or policy attachment to give the change time to propagate:
resource "aws_iam_role" "some_role" {
...
provisioner "local-exec" {
command = "sleep 10"
}
In this case you may use operation timeouts. Timeouts are handled entirely by the resource type implementation in the provider, but resource types offering these features follow the convention of defining a child block called timeouts that has a nested argument named after each operation that has a configurable timeout value. Each of these arguments takes a string representation of duration, such as "60m" for 60 minutes, "10s" for ten seconds, or "2h" for two hours.
resource "aws_db_instance" "example" {
# ...
timeouts {
create = "60m"
delete = "2h"
}
}
Ref: https://www.terraform.io/docs/configuration/resources.html
I would like to create a new instance based on my stored AMI.
I achieve this by the following code:
RunInstancesRequest rir = new RunInstancesRequest(imageId,1, 1);
// Code for configuring the settings of the new instance
...
RunInstancesResult runResult = ec2.runInstances(rir);
However, I cannot find a wait to "block"/wait until the instance is up and running apart from Thread.currentThread().sleep(xxxx) command.
On the other hand, StartInstancesResult and TerminateInstancesResult gives you a way to have access on the state of the instances and be able to monitor any changes. But, what about the state of a completely new instance?
boto3 has:
instance.wait_until_running()
From the boto3 docs:
Waits until this Instance is running. This method calls EC2.Waiter.instance_running.wait() which polls EC2.Client.describe_instances() every 15 seconds until a successful state is reached. An error is returned after 40 failed checks.
From the AWS CLI changelog for v1.6.0:
Add a wait subcommand that allows for a command to block until an AWS
resource reaches a given state (issue 992, issue 985)
I don't see this mentioned in the documentation, but the following worked for me:
aws ec2 start-instances --instance-ids "i-XXXXXXXX"
aws ec2 wait instance-running --instance-ids "i-XXXXXXXX"
The wait instance-running line did not finish until the EC2 instance was running.
I don't use Python/boto/botocore but assume it has something similar. Check out waiter.py on Github.
Waiting for the EC2 instance to get ready is a common pattern. In the Python library boto you also solve this with sleep calls:
reservation = conn.run_instances([Instance configuration here])
instance = reservation.instances[0]
while instance.state != 'running':
print '...instance is %s' % instance.state
time.sleep(10)
instance.update()
With this mechanism you will be able to poll when your new instance will come up.
Depending on what you are trying to do (and how many servers you plan on starting), instead of polling for the instance start events, you could install on the AMI a simple program/script that runs once when the instance starts and sends out a notification to that effect, i.e. to an AWS SNS Topic.
The process that needs to know about new servers starting could then subscribe to this SNS topic, and would receive a push notifications each time a server starts.
Solves the same problem from a different angle; your mileage may vary.
Go use Boto3's wait_until_running method:
http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Instance.wait_until_running
You can use boto3 waiters,
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#waiters
for this ex: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Waiter.InstanceRunning
Or in Java https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/
I am sure there are waiters implemented in all the AWS sdks.
I have few servers running on Amazon's EC2 and I would like to backup the image (create AMI) every week (replacing the old image).
Is there any way to automate this?
You should be able to use the command line tools to create an ami. Something like ec2-create-image -n "<image name here>" <your instancId here>. Put that in a cron entry that is scheduled weekly and you are done. You should be able to use the answer to this question to figure out what your instance id is programatically.
You can now use AWS Lambda to create AMIs automatically. The whole setup should be completed in around 10 minutes along with the schedule that you like.