I've been evaluating several cloud compute providers, Amazon EC2 among them. I started an instance with a Windows image, and ever since then it's been "pending", for more than 30 minutes now.
Is this a typical amount of wait for an instance to start? This would be highly undesirable for my purpose. Perhaps I started it incorrectly? I couldn't find any info on what "pending" means on Amazon - does anyone here know?
Pending means the instance is being created, if it has been like that for 30 minutes something went wrong, typicaly I wait for 3 minutes.
I would just create another instance and when the pending one is over, terminate it.
you would probably waste 12.5 cents tho...
I recently faced the same issue. After reaching out to AWS support, they provided a workaround which worked well.
You can use the AWS CLI to stop the instance, instead of AWS console. Although an instance stack in Pending state cannot be managed through AWS console, using the AWS CLI allows you to stop it and start it again. The following command should allow you to stop the instance and force it move to the "Stopped" state:
aws ec2 stop-instances --instance-ids <You instance Id>
You can find more information about how to install and use AWS CLI here:
https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
Related
Followed TeamCity's description of running a TeamCity build server on AWS with a cloudformation template. Launched it, it gets stuck at AgentService (Resource creation initiated). Waited for half an hour, no progress.
Resources tab shows the following:
What am I doing wrong here?
(For me) this typically happens if the service cannot be started for some reasons. For instance if the cluster does not have enough suitable instances to start your service or for some other reason.
For diagnostic, check your service in the ECS cluster and there check events and in tasks of your service, check stopped tasks (and reasons they were stopped).
Got a tip from a colleague that if you are creating a CF template based service, it may take up to 3(!) hours. Tried again today, after 3 hours it was up and running.
The reason for this is the setup of the ECS, which involves DNS setup for an internet facing service.
I have an Amazon EC2 instance which is registered to a cluster of Amazon ECS.
And I want to change this instance's type from c4.large to c4.8xlarge.
I'm able to change its type from c4.large to c4.8xlarge in AWS console. But after the change, I found
[ERROR] Could not register module="api client" err="ClientException: Container instance type changes are not supported. Container instance XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX was previously registered as c4.large.
being printed in /var/log/ecs/ecs-agent.log.20XX-XX-XX-XX file.
Is it possible to change ec2 instance type and re-register it to a cluster?
I think maybe deregister it first, then register it again should work. But I'm afraid this may cause something irreversible in my AWS working environment. So I haven't tried this method yet.
To solve this connection problem between the agent and cluster, just delete the file /var/lib/ecs/data/ecs_agent_data.json and restart docker and ECS.
After that, a new container instance will be created in your cluster with the new size.
sudo rm /var/lib/ecs/data/ecs_agent_data.json
sudo service docker restart
sudo start ecs
Then you can go to the ECS cluster console and deregister the old container instance
UPDATE:
According to #florins and #MBear commented below, AWS updated the data file on ECS instances.
sudo rm /var/lib/ecs/data/agent.db
sudo service docker restart
sudo start ecs
As of March 2021 / AMI image ami-0db98e57137013b2d, /var/lib/ecs/data/ecs_agent_data.json mentioned in the last useful answer does not exist. For me, the commands to execute on the changed instance were:
sudo rm /var/lib/ecs/data/agent.db
sudo service docker restart
After that, it was possible to deploy containers to the instance, without fresh registration (AWS automatically registered a second ECS container instance of the new type). I did have a leftover container instance with the resources of the old instance type to remove.
You can't do this. Per their docs:
The type of EC2 instance that you choose for your container instances determines the resources available in your cluster. Amazon EC2 provides different instance types, each with different CPU, memory, storage, and networking capacity that you can use to run your tasks. For more information, see Amazon EC2 Instances.
This means that when you launch a container on an instance, the agent gathers a bunch of metadata about the instance to run it. If you change it, all of that metadata (or a lot) has changed in a bad way. CPU units, memory, etc. The agent is aware of this and will report it as an error.
You should spin up a new instance of the new type and register it to the cluster and let the task run on it. If it's a service, just terminate the old instance and let it run it against the new one.
I can't think of any real reason why terminating your old instance would cause something irreversible unless it is misconfigured or fragile via user specific settings, by default this would not cause anything destructive.
As alternative approach if the EC2 instance does not store any valuable a new instance using the old instance as template could be started. This takes all existing values and can be achieved just with a few clicks in minutes.
Select the EC2 instance and then "Actions -> Images and templates -> Start more like this". Just change the instance type.
When the instance is running got the the ECS cluster to the tab "ECS instances" and activate the new created instance.
Shutdown the old instance
Update your task maybe taking more cpu and memory and update the service to take the new task revision
Sometimes my application dies without any reason and I can detect that using CloudWatch and CPU usage metric going down.
At this moment I want to restart the java application or the whole EC2 instance.
Any suggestions how can I achive that?
Thanks.
AWS CloudWatch now provides a reboot EC2 instance action.
You can let CloudWatch terminate your EC2 instance and let AutoScaling bring up another "fresh" instance with your application configured.
If only your application halt but EC2 instance works.
You could write a shell monitor the app using CloudWatch API and shoot the app when necessary,then make it a task in cron. Or you can using a shell right in your EC2 instance.
To restart your EC2 instance automatically is way too dangerous ,you can try Autoscaling, which automatically starts a new instance when your instance frozen and keep your SLA.
You can configure in Cloudwatch Alarm EC2 Action for reboot the instance.
Screenshot:
References:
https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-action-reboot-ec2-instance/
In my application I need the tomcat to be powered by SunJDK. But the default AWS EBS AMI comes with OpenJDK. So I wanted to change this to Sun JDK. But this simplistic task is turning out to be not-so-simple. Here is what I did -
On the EC2 instance that is powering my EBS Environment I installed Sun JDK by downloading the rom manually and then running rpm -i <jdk-rpm-file.rpm> .
Then I updated the java alternatives as listed here.
Next I restarted the app server to make sure that the feature that requires Sun JDK is working. It works.
Next I create an image by right clicking on the EC2 instance and selecting "Create Image (EBS AMI)"
I wait for the AMI to be created. Then I get the AMI ID.
Set the "Custom AMI ID" in configuration of the test environment to be the newly created AMI.
Apply change. This triggers update of environment.
Now comes the problem. As soon as it updates the environment it creates a new EC2 instance to connect to this environment.
Then after "adding" the instance it starts throwing this warning messages - "Failed to retrieve status of instance 'i-eb800c88' 2 consecutive time(s). Elastic Beanstalk will attempt to retrieve status up to 10 consecutive times before terminating the instance."
This continues for 10 tries and then it kills the instance and adds another instance and this continues for a long time.
I am not sure where am I going wrong. Any pointers appreciated.
Recently I got a response from Saad working at AWS Team & it solved my problem. Here is his answer -
You will need to launch the AMI outside of Elastic Beanstalk (directly from the EC2 console), log into it and do your customizations then burn the AMI. Otherwise, the Host Manager might get corrupted and your instance will fail to come up.
The following documentation highlights the steps needed to create an AMI compatible with Elastic Beanstalk: http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/index.html?using-features.customami.html.
If I stop an EC2 instance, rather than terminate, will this image be saved in my account and be available to be used at a later stage? ..as I noticed terminated instances eventually dissappear, would a stopped instance be available to boot up and start when ever?
Also regarding charging, I assume the cost for having a stopped instance would be charged per GB similar to a custom AMI?
Also another possibly simple question, if I shutdown a machine over SSH or via a script, does this initiate a termination or just a stop an instance (I assume it terminates the instance).
Thanks
If you terminate the EBS backed instance, it will remove it from the list of running instance, including it's allocated EBS volume. Unless you set the instance attribute not to delete the volume.
If you only stop, it will changed to stopped status and you can start it again later.
If you shutdown a machine, it default's to stop.
A good read to protect your instance see: http://alestic.com/2010/01/ec2-instance-locking
If you stop an instance based on EBS, then the instance will terminate automatically but you'll be charged for EBS storage until you delete the EBS.