I created a small Amazon EC2 instance like this:
ec2-run-instances --group default --block-device-mapping "/dev/sda1=:16:off" \
--instance-initiated-shutdown-behavior stop ami-cb97c68e
That created a fixed disk for the instance in Amazon's Elastic Block Store. Now I find I would like to upgrade to a large instance of EC2... but I would like to continue using the fixed disk I created for the small instance.
How do I switch it over? (For the record, that AMI is for the US West Ubuntu image.)
Coincidentally, Eric Hammond has just written an article about this exact topic. I think it should answer your question perfectly.
Related
I am planning to create cluster with three nodes and each node will be launched in three different Amazon EC2 zone.
As per Datastax Documentation, I will use Ec2MultiRegionSnitch and replication stragey is NetworkTopologyStrategy. Below is my needs to be achieved
Cluster Size : 3 (Spanning Across Amazon EC2 Region).
Replication Factor: 3
Read and Write Level : QUORUM.
Based on the above configuration, I can survive on single node loss(Meaning that down of any one of amazon region. Correct me if I am wrong).
In order to achieve the above configuration, I have two option
Option-1 : Using Datastax provided Amazon EC2 AMI image.
This option launch the instance with almost all components needed to run cassandra with some monitoring tools(opscenter..etc)
But It store all data on EC2 Instance Store hence data persists only during the life of the instance and the storage size depends upon instance type.
Option-2 : Using Customised installation
In this option, I have to launch Amazon EC2 Ubuntu AMI,installing JAVA,installing Datastax community edition.
This option enable me to store all my data on EBS. Hence I can expand EBS whenever I needed and the same time I can restore any node using EBS snapshot.
My Question:
Which one of the option is suitable for my needs?.
Note:
I read the documentation provided by Datastax and very new to cassandra. Hence, Whatever inputs you provided will be very useful to me.
Thanks
It's not true that you get Datastax AMI only with EC2 ephemeral storage. Starting from version 2.5 they claim you can choose EBS as well: Introducing the DataStax Auto-Clustering AMI 2.5. That's an relatively easy way of getting started which I've personally chosen.
Should you choose EBS or EC2 ephemeral storage?
The answer is: it depends...
The past (~2012-2013):
EC2 instances with ephemeral storage were a better choice. There were detailed performance benchmarks over the years which indicated that EBS is getting better, but still, attached physical drives were better.
The past (~2014):
EC2 choice is still better. Datastax wrote a nice post about pricing, network and failure resilience: What is the story with AWS storage?
Present (~2016):
instaclustr claims:
By running Cassandra on Amazon EBS, you can run denser, cheaper
Cassandra clusters with just as much availability as ephemeral storage
instances.
Nice presentation here: AWS re:Invent 2015 | (BDT323) Amazon EBS & Cassandra: 1 Million Writes Per Second on 60 Nodes
All in all, I suggest you doing a TCO analysis and if there isn't a big difference in price, choose EBS - because of out of the box ability to make a snapshot. What's more, chances are EBS will be improved over the time.
I would like to install DSE on m4.* instances, but it seems that the latest ComboAMI doesn't support it. Since it requires a different configuration, as it's not an instance-store backed instance, but rather an ebs-store backed instance and also a HVM instance and not PV instance.
I've started working on a new provisioning script, that basically does what the ComboAMI provisioning script does, but also adds the RAID0 creation for the EBS volumes that I attach per node, and the xfs file system creation.
My question is divided to two parts:
First of all, I was wondering if anyone in DataStax has already done something similar that he can share, since that can be a great reference for anyone that wants to do something similar.
Second of all, has anyone had any experience building a similar AMI on a m4.* instance type? I didn't manage to get something to work with the ComboAMI project.
I am trying to create an Amazon EC2 instance. I want to create a micro, 64-bit, Ubuntu 12.04 LTS instance.
In Amazon Web Services I have seen all instance have AMI numbers. Now I found two ami(s) with numbers ami-8a7f3ed8 and ami-b8a8e9ea. both looks same to me - micro, ebs-based, 64-bit Ubuntu 12.04LTS images.
If so, what is the difference and why two number for the same machine image?
When selecting an AMI, select from a trusted source.
The AMI number is just a unique identifier for a particular image that someone published. The title (e.g. Ubuntu 12.04LTS) is just a claim by the person who published the AMI about what is on it.
If you get your AMI from a source that is not known to be trustworthy, it could potentially contain built-in security holes, pre-installed spam relays, etc.
From Amazon
You launch AMIs at your own risk. Amazon cannot vouch for the integrity or security of AMIs shared by other EC2 users. Therefore, you should treat shared AMIs as you would any foreign code that you might consider deploying in your own data center and perform the appropriate due diligence.
Ideally, you should get the AMI ID from a trusted source (a web site, another EC2 user, etc). If you do not know the source of an AMI, we recommend that you search the forums for comments on the AMI before launching it. Conversely, if you have questions or observations about a shared AMI, feel free to use the AWS forums to ask or comment.
Amazon's public images have an aliased owner and display amazon in the userId field. This allows you to find Amazon's public images easily.
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AESDG-chapter-usingsharedamis.html#usingsharedamis-security
Personally I select AMIs published well-known entities like Amazon or RighScale.
An infinite number of people can create an infinite number of disk image variations that are still "64-bit micro Ubuntu 12.04 LTS". Just like there are over 500 million PCs in the world running Windows 7 64-bit, yet their hard drives all contain different data. If you wanted to be able to differentiate Joe's disk image from Sue's disk image, you'd need to give them different identifiers. That's why the AMI numbers are different.
AMI is just an image of a disk. It has nothing to do with type(micro). You can create multiple AMIs from the same instance and they will have have different IDs.
They are just different images. You could make two images from the same machine a minute after each other with no changes on the machine at all and they will have different AMI id's. The AMI ID is just applied at the time the image is created as a unique identifier, it infers nothing about the uniqueness of the image content.
I have a Windows 2008 EC2 instance to which I have done some customizing on the EBS boot drive.
I started the instance as m1.small (or m1.large) and the instance storage does not appear as an additional drive.
I've read that the -b switch in the ec2-run-instances command allows you to create mappings for the ephymeral instance storage. The ec2-run-instances command creates a new instance, however, in my case, the instance already exists and therefore I start it as ec2-start-instances, which does not have a -b switch for ephymeral instance storage.
Is there any way I can get to the ephymeral instance storage that comes with an m1.small instance for my existing EBS-booted instance?
UPDATE: It seems that nowadays (Feb 2015) Windows machines mount ephymeral instance storage in the Z: drive.
I'm afraid this functionality isn't available (yet) for Amazon EC2, but it's a very good question in fact - the common answer used to refer to the explicated launch time requirement, see e.g. ec2-modify-instance-attribute:
Note
If you want to add ephemeral storage to an Amazon EBS-backed instance,
you must add the ephemeral storage at the time you launch the
instance. For more information, go to Overriding the AMI's Block
Device Mapping in the Amazon Elastic Compute Cloud User Guide, or to
Adding A Default Instance Store in the Amazon Elastic Compute Cloud
User Guide. [emphasis mine]
That hasn't been that much of an issue in the past, but given the recent introduction of 64-bit ubiquity implies a significant improvement of vertical scaling versatility (see EC2 Updates: New Medium Instance, 64-bit Ubiquity, SSH Client), this is suddenly a topic indeed - your question yields even more questions in turn:
What happens for the converse case, i.e. when I start a sufficiently large instance with lots of ephemeral storage and scale it down (and possibly up again) thereafter?
In case the initial block device mapping is retained somehow, should we always start with a large instance therefore? (I actually doubt that this is the case though.)
This question can only be addressed by the AWS team I guess, so you may want to file a support request or relay the question to the Amazon Elastic Compute Cloud forum at least.
I think what you're asking (but correct me if I'm wrong) is "how do I add additional storage to an EC2 instance?".
In which case, the answer is:
Select the Volumes panel in the AWS console and create a new volume of the size you want, making sure it's in the same Availability Zone as the instance you want to attach it to. Then select that new Volume, and click 'Attach' - select the instance you want to attach it to, and click OK.
Now log-on to the instance, and in Computer Management select the Disk Management plugin, format the new unassigned partition, and give it whatever drive letter you wish. It will then show up in Explorer as a standard Windows drive.
What is the different between starting an AWS Image and Instances?
Example:
I do notice when I am running AWS image using boto, I can only stop the Image while running AWS instance using boto, I can only terminate.
Think of an EC2 instance as a single running server with CPU, memory, hard disk, networking, etc. Any changes you make to that instance affect only that instance.
Think of an AMI (Amazon Machine Image) as an exact copy of the root file system that gets copied to the hard disk when you start a new instance. The AMI is a hard disk sitting on a shelf. You make an exact copy of the hard disk on the shelf, install the new hard disk in a server, and turn the server on. You can do this for as many servers as you'd like to start without affecting the master copy.
The AMI defines the initial state of each instance. Each instance changes as it runs, but you can never change the original AMI once it has been created (other than to delete it).
There are more details that refine this conceptual model, but that's the basics.
Specific to the wording in your question:
Sometimes we say we're "starting an AMI" sometimes we say we're "starting an instance". We mean the same thing. We're really starting an instance using an AMI as the template.
We never say we're "stopping/terminating an image" or "stopping an ami" as, once started, it's really the instance that's running.
You can have one or more instances running that are derived from an image (AMI). Here is a good little tutorial, that's a bit old mind you, talking about how you can convert an Instance to an AMI ... which you can then redeploy one or many times:
http://webkist.wordpress.com/2010/03/16/creating-an-amazon-ec2-ebs-ami-from-a-running-instance/
What is an AMI: Amazon Machine Image
http://en.wikipedia.org/wiki/Amazon_Machine_Image
Technically, you can't start an AMI. You can start an instance that is derived from an AMI.