So, I manage a large fleet of EC2 instances, and some of them have instance store data volumes (i.e. not the root volume). I want to find all the environments which use this type of storage, but neither describe-instances nor describe-instance-attribute (with the blockDeviceMapping attribute) seem to tell me. In other words, instances which I know use instance storage (by checking http://169.254.169.254/latest/meta-data/block-device-mapping/ from the instance itself as per the documentation), don't have those volumes show up in the volume list for the regular AWS API call.
Are there any API calls I can use external to the instance to determine whether the instance was launched with instance store volumes, or will I have to run commands on the instances every time?
Really hope I'm just missing something obvious. :P
It doesn't look like you are missing anything obvious. The documentation only mentions using instance metadata.
When you view the block device mapping for your instance, you can see only the EBS volumes, not the instance store volumes. You can use instance metadata to query the complete block device mapping.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html
Related
I want to change the instance type of an aws ec2 instance based on the cpu utilization of the instance. Basically i want the instance type to be t3.large if the cpu utilization is > 80% and t3.medium if it is below that. Is there any way to achieve this without any downtime? Can i achieve this with aws cloudformation? If so can you provide me a guide or link which i can refer to?
Unfortunately, you can't do this. Changing instance type requires stopping your instance:
You must stop your Amazon EBS–backed instance before you can change its instance type.
However, you could setup your instance(s) to operate in an autoscaling group. This way you could use UpdatePolicy to perform rolling or replacing update. This could be use to ensure that part or full fleet of your instances is operating while the new instance types are rolled out.
I want to use power of cloud, where master or main ec2 instance is creating multiple instances based on need and then destroying them.
need to Create multiple instance from same AMI.
I want to know best way to accomplish this.
Thanks
You can utilize EC2 APIs for this purpose.
ec2-run-instances (http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RunInstances.html) is a command that allows you to create a new instance from your own (or public) AMI. You can also specify the number of instances you wish to create.
There are also Web Service operation (RunInstances) for this purpose:
http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-RunInstances.html
Which one to use is up to you. However, I don't think starting a new instance from a master instance is a good practice in AWS. You can rely on Elastic Load Balancing (http://aws.amazon.com/elasticloadbalancing/) and Auto Scaling (http://aws.amazon.com/autoscaling/) to scale up/down your server fleet depending on incoming traffic or healthiness of your running instances.
I have a Windows 2008 EC2 instance to which I have done some customizing on the EBS boot drive.
I started the instance as m1.small (or m1.large) and the instance storage does not appear as an additional drive.
I've read that the -b switch in the ec2-run-instances command allows you to create mappings for the ephymeral instance storage. The ec2-run-instances command creates a new instance, however, in my case, the instance already exists and therefore I start it as ec2-start-instances, which does not have a -b switch for ephymeral instance storage.
Is there any way I can get to the ephymeral instance storage that comes with an m1.small instance for my existing EBS-booted instance?
UPDATE: It seems that nowadays (Feb 2015) Windows machines mount ephymeral instance storage in the Z: drive.
I'm afraid this functionality isn't available (yet) for Amazon EC2, but it's a very good question in fact - the common answer used to refer to the explicated launch time requirement, see e.g. ec2-modify-instance-attribute:
Note
If you want to add ephemeral storage to an Amazon EBS-backed instance,
you must add the ephemeral storage at the time you launch the
instance. For more information, go to Overriding the AMI's Block
Device Mapping in the Amazon Elastic Compute Cloud User Guide, or to
Adding A Default Instance Store in the Amazon Elastic Compute Cloud
User Guide. [emphasis mine]
That hasn't been that much of an issue in the past, but given the recent introduction of 64-bit ubiquity implies a significant improvement of vertical scaling versatility (see EC2 Updates: New Medium Instance, 64-bit Ubiquity, SSH Client), this is suddenly a topic indeed - your question yields even more questions in turn:
What happens for the converse case, i.e. when I start a sufficiently large instance with lots of ephemeral storage and scale it down (and possibly up again) thereafter?
In case the initial block device mapping is retained somehow, should we always start with a large instance therefore? (I actually doubt that this is the case though.)
This question can only be addressed by the AWS team I guess, so you may want to file a support request or relay the question to the Amazon Elastic Compute Cloud forum at least.
I think what you're asking (but correct me if I'm wrong) is "how do I add additional storage to an EC2 instance?".
In which case, the answer is:
Select the Volumes panel in the AWS console and create a new volume of the size you want, making sure it's in the same Availability Zone as the instance you want to attach it to. Then select that new Volume, and click 'Attach' - select the instance you want to attach it to, and click OK.
Now log-on to the instance, and in Computer Management select the Disk Management plugin, format the new unassigned partition, and give it whatever drive letter you wish. It will then show up in Explorer as a standard Windows drive.
Amazon's new EBS-based EC2 instances have two options to shutdown: terminate or stop. Stopped instances can be later started again, automatically continuing from the same EBS root disk state they had when they were stopped.
But what happens when an Amazon datacenter has a hardware failure, and the EC2 instance is forced to shutdown. Does it terminate or stop? If the instance has been configured to stop by default on shutdown, can I rely on it being stopped also in this situation, and being able to start it again later?
An EC2 instance can be terminated at any time and one must account for this indeed, as already mentioned in David's answer (+1). You can arrange for a failed instance's Elastic Block Store (EBS) to remain available regardless though, see e.g. the respective FAQ What happens to my data when a system terminates?:
The data stored on a local instance store will persist only as long as
that instance is alive. However, data that is stored on an Amazon EBS
volume will persist independently of the life of the instance. If you
are using an Amazon EBS volume as a root partition, then you have set
the Delete On Terminate flag to "N" for your Amazon EBS volume to
persist outside the life of the instance. [emphasis mine]
This is explained in more detail in section 2. Delete on Termination within Eric Hammond's recommended article Three Ways to Protect EC2 Instances from Accidental Termination and Loss of Data:
Though EBS volumes created and attached to an instance at
instantiation are preserved through a “stop”/”start” cycle, by default
they are destroyed and lost when an EC2 instance is terminated. This
behavior can be changed with a delete-on-termination boolean value
buried in the documentation for the --block-device-mapping option of
ec2-run-instances.
He is referring to the ec2-run-instances documentation, and all this is meanwhile illustrated in more detail within Amazon EC2 Root Device Storage Concepts as well:
By default, the root device volume and the other volumes created when
an Amazon EBS-backed instance is launched are automatically deleted
when the instance terminates [...]. You can change
the default behavior by setting the DeleteOnTermination flag to the
value you want when you launch the instance. For an example of how to
change the flag at launch time, see Using Amazon EC2 Root Device Storage.
I assume you mean the CPU related hardware fails, rather than the network disk. The way I treat EC2 is to create a system that can go up and down without data loss. Anything important you should use an S3 bucket, not EBS.
So, in a simple situation, if there is only one instance, then I can store the data into a EBS volume mounted on that instance. e.g. /mnt/db
However, how does it work if I scale and have multiple instance (either static or dynamic scaling)?
Because one EBS can only attach to one instance, if I have multiple instance, does it mean that I have to attach an EBS volume for each instance? If that's the case, the data on each Instance's EBS volume will be different.
It is obvious that I want all instances to access (R & W) a single volume (as data-storage). and the data in the volume will constantly grow and there is no downtime.
What is the solution? Is there a way that I don't mount the device (EBS), and just call it for accessing the data?
Here is what I can think of:
1) if each instance has its own EBS volume, then each time interval (e.g. 1 hour), all instances will unmount & detach the EBS volume,and attach a new one. Then there is one powerful instance that mount all the EBS volumes just detached, and aggregate all the data.
2) or similar to 1), instead of detach and attach, I just take a snapshot on all volumes for all instances. Then the powerful instance aggregateness the data from the snapshot. And save the result into either another EBS or S3.
These two approach seem to be working.. but require a lot of work. is there a smarter way to approach this problem? thanks.
by the way, because of performance issue, I cannot have the instance writes data to S3. :)
OH how about this
3) First, all instances have their own EBS and write data into the EBS. and then each hour, data will be sent to S3. Then another instance will aggregate them.
how about having ang NFS instance which can be mounted to the other instances?
It seems that you need to create an EBS snapshot of your most up to date EC2 instance. This will create an EBS backed AMI. You would then need to terminate all your EC2 instances that are not up to date and launch a new stack of instances from your newly created AMI. If you had a load balancer running then you would have to attach these new instances to your load balancer also.
It seems a little long-winded but it can all be done programmatically. At least this is how I think scaling in the cloud with Amazon works and far as propagating changes across multiple instances goes. Somebody else with more experience verify this. I plan to test it out myself later on.