I would like to install DSE on m4.* instances, but it seems that the latest ComboAMI doesn't support it. Since it requires a different configuration, as it's not an instance-store backed instance, but rather an ebs-store backed instance and also a HVM instance and not PV instance.
I've started working on a new provisioning script, that basically does what the ComboAMI provisioning script does, but also adds the RAID0 creation for the EBS volumes that I attach per node, and the xfs file system creation.
My question is divided to two parts:
First of all, I was wondering if anyone in DataStax has already done something similar that he can share, since that can be a great reference for anyone that wants to do something similar.
Second of all, has anyone had any experience building a similar AMI on a m4.* instance type? I didn't manage to get something to work with the ComboAMI project.
Related
I couldn't find the answer on EC2 documentation. What is it for? If I launched an EBS backed instance, the root device for the instance would be an EBS volume. If I install a few tools/software on the instance, will those be installed on the root instance by default? Still I guess the question really came from the little understanding of the root device. Any detailed info on that?
Also if I need to launch another EBS backed instance, and also want to have the same copy of the tools/software installed on the earlier instance, how to do that?
EC2 instances have two types of storage -- ephemeral storage, and EBS based storage -- and each instance is given a specific amount of that disk space by default. Each instance has its own allotment of disk space that it can use which is independent of any other instances disk space that you launch. You can also add an EBS volume to your drive for additional storage, but, those volumes can only be attached to one specific host at a time.
If you had two EBS instances, and you wanted to ensure that they're both using the same tools and software, you'd need to enforce that using a configuration management tool, such as puppet, chef, or cfengine.
I agree with what Ian wrote. I would add that the "root device" in EC2 is analogous to the operating system partition in a personal computer. It is where the filesystem of your OS resides.
This is the best answer about 'root device' so far! AWS is still new. And most of the times, the answer is just 'cut and paste' straight from AWS document, which does not help much!
https://web.archive.org/web/20140601070714/https://skeddly.desk.com/customer/portal/articles/1346918-ebs-backed-versus-instance-store
This may have been added since you asked, but I believe this answers the question:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html
I have a Windows 2008 EC2 instance to which I have done some customizing on the EBS boot drive.
I started the instance as m1.small (or m1.large) and the instance storage does not appear as an additional drive.
I've read that the -b switch in the ec2-run-instances command allows you to create mappings for the ephymeral instance storage. The ec2-run-instances command creates a new instance, however, in my case, the instance already exists and therefore I start it as ec2-start-instances, which does not have a -b switch for ephymeral instance storage.
Is there any way I can get to the ephymeral instance storage that comes with an m1.small instance for my existing EBS-booted instance?
UPDATE: It seems that nowadays (Feb 2015) Windows machines mount ephymeral instance storage in the Z: drive.
I'm afraid this functionality isn't available (yet) for Amazon EC2, but it's a very good question in fact - the common answer used to refer to the explicated launch time requirement, see e.g. ec2-modify-instance-attribute:
Note
If you want to add ephemeral storage to an Amazon EBS-backed instance,
you must add the ephemeral storage at the time you launch the
instance. For more information, go to Overriding the AMI's Block
Device Mapping in the Amazon Elastic Compute Cloud User Guide, or to
Adding A Default Instance Store in the Amazon Elastic Compute Cloud
User Guide. [emphasis mine]
That hasn't been that much of an issue in the past, but given the recent introduction of 64-bit ubiquity implies a significant improvement of vertical scaling versatility (see EC2 Updates: New Medium Instance, 64-bit Ubiquity, SSH Client), this is suddenly a topic indeed - your question yields even more questions in turn:
What happens for the converse case, i.e. when I start a sufficiently large instance with lots of ephemeral storage and scale it down (and possibly up again) thereafter?
In case the initial block device mapping is retained somehow, should we always start with a large instance therefore? (I actually doubt that this is the case though.)
This question can only be addressed by the AWS team I guess, so you may want to file a support request or relay the question to the Amazon Elastic Compute Cloud forum at least.
I think what you're asking (but correct me if I'm wrong) is "how do I add additional storage to an EC2 instance?".
In which case, the answer is:
Select the Volumes panel in the AWS console and create a new volume of the size you want, making sure it's in the same Availability Zone as the instance you want to attach it to. Then select that new Volume, and click 'Attach' - select the instance you want to attach it to, and click OK.
Now log-on to the instance, and in Computer Management select the Disk Management plugin, format the new unassigned partition, and give it whatever drive letter you wish. It will then show up in Explorer as a standard Windows drive.
What is the different between starting an AWS Image and Instances?
Example:
I do notice when I am running AWS image using boto, I can only stop the Image while running AWS instance using boto, I can only terminate.
Think of an EC2 instance as a single running server with CPU, memory, hard disk, networking, etc. Any changes you make to that instance affect only that instance.
Think of an AMI (Amazon Machine Image) as an exact copy of the root file system that gets copied to the hard disk when you start a new instance. The AMI is a hard disk sitting on a shelf. You make an exact copy of the hard disk on the shelf, install the new hard disk in a server, and turn the server on. You can do this for as many servers as you'd like to start without affecting the master copy.
The AMI defines the initial state of each instance. Each instance changes as it runs, but you can never change the original AMI once it has been created (other than to delete it).
There are more details that refine this conceptual model, but that's the basics.
Specific to the wording in your question:
Sometimes we say we're "starting an AMI" sometimes we say we're "starting an instance". We mean the same thing. We're really starting an instance using an AMI as the template.
We never say we're "stopping/terminating an image" or "stopping an ami" as, once started, it's really the instance that's running.
You can have one or more instances running that are derived from an image (AMI). Here is a good little tutorial, that's a bit old mind you, talking about how you can convert an Instance to an AMI ... which you can then redeploy one or many times:
http://webkist.wordpress.com/2010/03/16/creating-an-amazon-ec2-ebs-ami-from-a-running-instance/
What is an AMI: Amazon Machine Image
http://en.wikipedia.org/wiki/Amazon_Machine_Image
Technically, you can't start an AMI. You can start an instance that is derived from an AMI.
I was searching for EC2 EBS storage Centos 5.4 AMI in the community AMI and eventually I found Rightscale AMI (I think they called it RightImage).
Now I have created instance using that AMI, but I found out there is some Rightscale stuff inside which is worrying me about the safety on using it. I found out there are the following files in that AMI:
/etc/init.d/rightimage
/etc/init.d/rightlink
/etc/init.d/rightscale
/home/ec2
/home/s3sync
(may be more other files I haven't found out yet)
I know I can look into the script and folder and see what they do, but since a lot of user here recommended using Rightscale Centos AMI in EC2, I hope may be there is already some gurus here know what those mentioned script and folder doing and could advice me
i)whether is it safe to delete them. (I'm more concern on whether my data in the server will be safe by using this AMI)
ii)any installed apps in RightScale AMI that should be deleted
And if you think there is other free EC2 Centos AMI that is secure and solid, do suggest as well, thanks !
In order for RightScale to properly manage instances in ec2 they use a ruby based daemon called RightLink as a communication device between their core platform and each instance that is launched. The init scripts that you saw are required for the instance to self configure itself to the point where it can be managed by RightScale properly.
/etc/init.d/rightimage is the first script that is run. Essentially it just determines the OS, arch version, and installs the correct RightLink package from the S3 bucket. Afterwards it kicks off the /opt/rightscale/bin/post_install.sh script which uses the OS init control tools to register the startup scripts to be invoked on future boots of the OS; this ensures that RightLink will always be started.
/etc/init.d/rightscale is the next script that is run. It initializes RightScale-specific (but not RightLink-specific) system state. It is responsible for caching launch settings (aka userdata) and metadata in /var/spool and installing any available patches to the RightLink agent.
/etc/init.d/rightlink is the final script that is run. It configures and enrolls the RightLink agent idempotently. If configuration and enrollment succeed, rightlink starts the sandboxed monit which starts the persistent agent process. If you're not launching the AMI using the RightScale platform this will never properly enroll because they aren't expecting it to, as such RightScale will have no communication with the instance at all.
Removing all three of these from the image shouldn't in any way harm the overall stability of the image, but from a security standpoint they shouldn't cause any problems if they are present.
If you have any further specific questions about it I'd suggest hopping on their forums at https://forums.rightscale.com/
You could also try #rightscale on freenode.
I spent the day experimenting with AWS for the first time. I've got an EC2 instance running and I mounted an Elastic Block Store (EBS) to keep the MySQL databases.
Does it make sense to also put my web application files on the EBS, or should I just deploy them to the normal EC2 file system?
When you say your web application files, I'm not sure what exactly you are referring to.
If you are referring to your deployed code, it probably doesn't make sense to use EBS. What you want to do is create an AMI with your prerequisites, then have a script to create an instance of that AMI and deploy your latest code. I highly recommend you automate and test this process as it's easy to forget about some setting you have to manually change somewhere.
If you are storing data files, that are modified by the running application, EBS may make sense. If this is something like user-uploaded images or similar, you will likely find that S3 gives you a much simpler model.
EBS would be good for: databases, lucene indexes, file based CMS, SVN repository, or anything similar to that.
EBS gives you persistent storage so if you EC2 instance fails the files still exist. Apparently their is increased IO performance but I would test it to be sure.
If your files are going to change frequently (like a DB does) and you don't want to keep syncing them to S3 (or somewhere else), then an EBS is a good way to go. If you make infrequent changes and you can manually (or scripted) sync the files as necessary then store them in S3. If you need to shutdown or you lose your instance for whatever reason, you can just pull them down when you start up the new instance.
This is also assuming that you care about cost. If cost is not an issue, using the EBS is less complicated.
I'm not sure if you plan on having a separate EBS for your DB and your web files but if you only plan on having one EBS and you have enough empty space on it for your web files, then again, the EBS is less complicated.
If it's performance you are worried about, as mentioned, it's best to test your particular app.
Our approach is to have a script pre-deployed on our AMI that fetches the latest and greatest version of the code from source control. That makes it very straightforward to launch new instances quickly, or update all running instances (we take them out of the load balancing rotation one at a time, run the script, and put them back in the rotation).
UPDATE:
Reading between the lines it looks like you're mounting a separate EBS volume to an instance-store backed instance. AWS recently introduced EBS backed instances that have a ton of benefits vs. the old instance-store ones. I still mount my MySQL data on a separate EBS partition, though, so that I can easily mount it to a different server if needed.
I strongly suggest an EBS backed instance with a separate EBS volume for the MySQL data.