I usually restart node instance, but since client was complaining speed issue, i read somewhere online and stopped and started the instance.
It changed the IP of my instance and the code is all gone.
Now when i ssh, i can see /srv/www but i am not able to get into www folder. Changed the permissions, owner but still www is behaving differently.
Its a ROR application, deployed through aws-opswork
#Undo has the right idea. I haven't checked this in the last version of OpsWorks, but by default OpsWorks managed instances have an instance based store: meaning the storage is destroyed when the instance is stopped.
In OpsWorks, in the settings for a Layer, you can specify EBS volumes. EBS means Elastic Block Store, which will persist even if the instance is stopped and a new err instance is started.
Related
I have created an environment with Elastic Beanstalk with a EC2 instance with PHP installed: my files are in /var/www/html.
First I allowed Auto-Scaling/Load balancer but when the auto-scaling triggered, it created another instance and terminated the old one. And then I realized, the new one was not a clone of the old one : I lost all my config and my files, while I did attached a SSD root volume in my EB config.
I tried again and I created an AMI image which I included in my EB config (in Custom AMI ID). This time my config stays but my folder /var/www/html is emptied and replaced by default index.html files.
1. Is it supposed to happen ? I thought the auto scaling created a cloned of the instance ?
So I decided to disable auto scaling / load balancer and to work on a single instance mode. But then even when I reboot my EC2 instance, the config is preserved but my whole folder /var/www/html is emptied again and only the default files are inside.
2. Why ? There is an EBS volume attached to my instance (EB did that automatically), so it should not happen, if I understand correctly how it works.
Maybe it is the same issue for both but I really don't get why my files are deleted.
Thanks a lot for your help !
Romain
Autoscaling uses an AMI to launch new instances, and AMIs are no more than snapshots of EC2 instances at a certain point in time. Because of this, every time Autoscaling launches a new instance, any differences between the AMI and the current desired state must be applied in boot time prior to receive new traffic.
ElasticBeanstalk provides tools to manage application deployments integrated into the Autoscaling dynamic and also manage instance configurations. Sometimes these configurations become too complex to achieve during bootstrap using the EB tools and there is when custom AMIs come handy.
If you SSH into an autoscaling instance and start manually performing actions out of the ElasticBeanstalk toolstack's scope, all of those changes will be lost in the next Autoscaling event unless you save an updated AMI from your instance and apply it to your Autoscaling Group.
I have an Amazon EC2 instance which is registered to a cluster of Amazon ECS.
And I want to change this instance's type from c4.large to c4.8xlarge.
I'm able to change its type from c4.large to c4.8xlarge in AWS console. But after the change, I found
[ERROR] Could not register module="api client" err="ClientException: Container instance type changes are not supported. Container instance XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX was previously registered as c4.large.
being printed in /var/log/ecs/ecs-agent.log.20XX-XX-XX-XX file.
Is it possible to change ec2 instance type and re-register it to a cluster?
I think maybe deregister it first, then register it again should work. But I'm afraid this may cause something irreversible in my AWS working environment. So I haven't tried this method yet.
To solve this connection problem between the agent and cluster, just delete the file /var/lib/ecs/data/ecs_agent_data.json and restart docker and ECS.
After that, a new container instance will be created in your cluster with the new size.
sudo rm /var/lib/ecs/data/ecs_agent_data.json
sudo service docker restart
sudo start ecs
Then you can go to the ECS cluster console and deregister the old container instance
UPDATE:
According to #florins and #MBear commented below, AWS updated the data file on ECS instances.
sudo rm /var/lib/ecs/data/agent.db
sudo service docker restart
sudo start ecs
As of March 2021 / AMI image ami-0db98e57137013b2d, /var/lib/ecs/data/ecs_agent_data.json mentioned in the last useful answer does not exist. For me, the commands to execute on the changed instance were:
sudo rm /var/lib/ecs/data/agent.db
sudo service docker restart
After that, it was possible to deploy containers to the instance, without fresh registration (AWS automatically registered a second ECS container instance of the new type). I did have a leftover container instance with the resources of the old instance type to remove.
You can't do this. Per their docs:
The type of EC2 instance that you choose for your container instances determines the resources available in your cluster. Amazon EC2 provides different instance types, each with different CPU, memory, storage, and networking capacity that you can use to run your tasks. For more information, see Amazon EC2 Instances.
This means that when you launch a container on an instance, the agent gathers a bunch of metadata about the instance to run it. If you change it, all of that metadata (or a lot) has changed in a bad way. CPU units, memory, etc. The agent is aware of this and will report it as an error.
You should spin up a new instance of the new type and register it to the cluster and let the task run on it. If it's a service, just terminate the old instance and let it run it against the new one.
I can't think of any real reason why terminating your old instance would cause something irreversible unless it is misconfigured or fragile via user specific settings, by default this would not cause anything destructive.
As alternative approach if the EC2 instance does not store any valuable a new instance using the old instance as template could be started. This takes all existing values and can be achieved just with a few clicks in minutes.
Select the EC2 instance and then "Actions -> Images and templates -> Start more like this". Just change the instance type.
When the instance is running got the the ECS cluster to the tab "ECS instances" and activate the new created instance.
Shutdown the old instance
Update your task maybe taking more cpu and memory and update the service to take the new task revision
In my application I need the tomcat to be powered by SunJDK. But the default AWS EBS AMI comes with OpenJDK. So I wanted to change this to Sun JDK. But this simplistic task is turning out to be not-so-simple. Here is what I did -
On the EC2 instance that is powering my EBS Environment I installed Sun JDK by downloading the rom manually and then running rpm -i <jdk-rpm-file.rpm> .
Then I updated the java alternatives as listed here.
Next I restarted the app server to make sure that the feature that requires Sun JDK is working. It works.
Next I create an image by right clicking on the EC2 instance and selecting "Create Image (EBS AMI)"
I wait for the AMI to be created. Then I get the AMI ID.
Set the "Custom AMI ID" in configuration of the test environment to be the newly created AMI.
Apply change. This triggers update of environment.
Now comes the problem. As soon as it updates the environment it creates a new EC2 instance to connect to this environment.
Then after "adding" the instance it starts throwing this warning messages - "Failed to retrieve status of instance 'i-eb800c88' 2 consecutive time(s). Elastic Beanstalk will attempt to retrieve status up to 10 consecutive times before terminating the instance."
This continues for 10 tries and then it kills the instance and adds another instance and this continues for a long time.
I am not sure where am I going wrong. Any pointers appreciated.
Recently I got a response from Saad working at AWS Team & it solved my problem. Here is his answer -
You will need to launch the AMI outside of Elastic Beanstalk (directly from the EC2 console), log into it and do your customizations then burn the AMI. Otherwise, the Host Manager might get corrupted and your instance will fail to come up.
The following documentation highlights the steps needed to create an AMI compatible with Elastic Beanstalk: http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/index.html?using-features.customami.html.
Amazon's new EBS-based EC2 instances have two options to shutdown: terminate or stop. Stopped instances can be later started again, automatically continuing from the same EBS root disk state they had when they were stopped.
But what happens when an Amazon datacenter has a hardware failure, and the EC2 instance is forced to shutdown. Does it terminate or stop? If the instance has been configured to stop by default on shutdown, can I rely on it being stopped also in this situation, and being able to start it again later?
An EC2 instance can be terminated at any time and one must account for this indeed, as already mentioned in David's answer (+1). You can arrange for a failed instance's Elastic Block Store (EBS) to remain available regardless though, see e.g. the respective FAQ What happens to my data when a system terminates?:
The data stored on a local instance store will persist only as long as
that instance is alive. However, data that is stored on an Amazon EBS
volume will persist independently of the life of the instance. If you
are using an Amazon EBS volume as a root partition, then you have set
the Delete On Terminate flag to "N" for your Amazon EBS volume to
persist outside the life of the instance. [emphasis mine]
This is explained in more detail in section 2. Delete on Termination within Eric Hammond's recommended article Three Ways to Protect EC2 Instances from Accidental Termination and Loss of Data:
Though EBS volumes created and attached to an instance at
instantiation are preserved through a “stop”/”start” cycle, by default
they are destroyed and lost when an EC2 instance is terminated. This
behavior can be changed with a delete-on-termination boolean value
buried in the documentation for the --block-device-mapping option of
ec2-run-instances.
He is referring to the ec2-run-instances documentation, and all this is meanwhile illustrated in more detail within Amazon EC2 Root Device Storage Concepts as well:
By default, the root device volume and the other volumes created when
an Amazon EBS-backed instance is launched are automatically deleted
when the instance terminates [...]. You can change
the default behavior by setting the DeleteOnTermination flag to the
value you want when you launch the instance. For an example of how to
change the flag at launch time, see Using Amazon EC2 Root Device Storage.
I assume you mean the CPU related hardware fails, rather than the network disk. The way I treat EC2 is to create a system that can go up and down without data loss. Anything important you should use an S3 bucket, not EBS.
I have an EC2 instance with "instance store" device as a root device (I did not know the difference between it and EBS before launching it). I would like to stop it but I can not do it with the command ec2-stop-instances, the output is:
Client.UnsupportedOperation: The instance 'i-XXXXXXXX' does not have an 'ebs' root device type and cannot be stopped.
Does anybody know how to stop it with Windows Console (I am not the owner of the Amazon account and I won't be able to contact with him for weeks)
Thanks in advance.
EC2 instance with "instance store" cannot be stopped. They can only be terminated.
If you would like the ability to stop an instance, I suggest recreating the instance with an EBS root device. These types of instances support the ec2-stop-instances command. You can consider this command as a "pause", since it can be resumed at any time, and you will not be charged for the time the instance is stopped.
Amazon announced the ability to boot instances from an EBS root volume only in December 2009, so you will find that older documentation and tutorials ignore the extended command-set that came with the EBS root volumes.
Further reading: Amazon EC2 Instances Now Can Boot from Amazon EBS
I am not the owner of the Amazon
account and I won't be able to contact
with him for weeks.
If you really want to "pause" this instance for a few weeks, you can create a machine image (AMI) of your instance and terminate it.
You will then be able to launch a new instance with your private AMI when you want to "resume" this instance. It will not be the same instance, as in it will have a different instance ID and a different IP, but you will be cloning the setup of your instance.
The methods to build an AMI differ if your instance is running Windows or Linux, but you should be able to find adequate information on the web about both scenarios.