I've browsed through other questions on SO concerning backing up EC2, and it has provided me with a good basis, but I'm still a bit confused as to how I should approach my solution and develop a contingency plan. Most questions are fairly specific, but I have a pretty vanilla setup, and I think this information will be beneficial to future users. Let me provide my basic setup:
Basic small instance
Pushing files to S3
Running MongoDB
Running nginx
Now, due to the ephemeral nature of EC2, it's apparent that I need to bind my EC2 instance to EBS to ensure persistent storage. The reason I'm attempting to develop a contingency plan is that I'm worried my instance may disappear at any time (due to outages, etc). If my instance were to disappear, I'm concerned that I would have to spin up a new instance and reinstall all my applications before getting everything up an running again. A few questions:
How do I backup my instance to ensure that if it were to disappear, I could quickly bring it back up (preferably without having to reinstall all previous software)? I don't need a series of backups, just the previous days (or weeks) backup to ensure that a previous working version exists that can be started quickly.
If I use EBS instead of instance storage, it essentially acts in place of my hard drive right? So if I have MongoDB installed, I'm assuming the database it is writing to cab placed on EBS?
If I go with the small instance with 160 GB of storage and I use EBS, would I need to allocate 160 GB of EBS out of the gate, or is the 160 GB for instance storage only?
So, in summary, I would like a solution (either manual or automatic) that can create a snapshot of my EC2 instance to ensure that if it were to disappear, it could be reconstructed without having to spend the time to manually reconstruct everything.
In an ideal world, if my instance were to disappear, I could spin up an version of my instance with everything intact (to a point where things were backed up). Any resources or suggestions? Thanks in advance.
OK, here we go:
For backups:
Create your instance from one of the stock AWS images. Make sure it is an EBS-backed VM - depending on the size of the VM you pick, you'll get a volume assigned with 'n' GB of space, attached as the boot volume (/dev/sda1).
Configure the VM with whatever software you need, apply patches, tune for disk fragmentation, CPU consumption (task priorities, etc), and any other configuration you need that makes the VM tailored to your requirements.
Stop the VM and take a snapshot of the EBS volume, then restart it (re-assigning the Elastic IP is there is one). This is your backup snapshot - repeat as desired at whatever frequency you like. Remember to stop the VM when you snap it to prevent the OS writing to the volume whilst you're taking a copy of it.
For recovery:
Your VM will fail, eventually. You'll break something and render it damaged or inoperative, or the hardware it's running on will suffer a fault. It will happen.
When it does, terminate it (if it didn't self-terminate) and spin-up a new VM of the same type from the AWS stock list. Wait until it shows as 'Running', and then stop it.
Detach its EBS volume and delete it.
Create a new EBS volume from whichever backup snapshot you last created, and attach that new volume to the VM as /dev/sda1.
Start the VM and assign your EIP if appropriate.
About EBS storage:
It's a chunk of storage space. If you format it to look like a standard disk, you can use it exactly as you would a physical disk. Install stuff on it, point software at it for use as storage space, whatever.
you have two options: (BUT NOT EXACTLY AS YOU WANT ;( )
1- Have an 'external' EBS attached into your EC2 instance, and manually (you can do automatically through cronjobs), make snapshots from it ! But it's not what you want, why ? If your EC2 instance disappears, you will need to re-create all your environment again and re-attach your EBS... So it's a nice way of having backups of HUGE datas on your EC2, but your enviroment is destructed...
2- The best way, but not so perfect, is after you finished configuring your EC2, make a private AMI from it, so anytime you want you can launch more instances like that , from that AMI, so everything is cloned.... BUT the worst part of it, is that every time you change a configuration from the instance, you still need to make a new AMI, and every time you make a new AMI, you need to reboot your instance to grant data integrity on your new private AMI !
I recommend you to take a closer look into RESERVED EC2 instances, that have a better stability from normal ones. BUT YOU STILL CAN HAVE HARDWARE DISASTERS as normal instances too...
Related
We run a server architecture where we have an X number of base servers which are always on. Our servers process jobs sent to them and the vast majority of our job requests come in during the workday. To facilitate this particular spike in volume, we use EC2 auto-scaling.
I prefer to launch servers through auto-scaling with as much of a configured AMI as possible as opposed to launching from a base AMI and installing packages through long Chef or Puppet scripts.
In our current build process, we implement changes to our code base late at night when only our base servers are needed and no servers are launched through auto-scaling. But every once in a while, we'll have a critical bug fix that needs to be implemented immediately during the day.
We have a rather large EBS hard drive associated with our servers (app. 400 GB) and AMI creation of a base server with the latest changes usually takes upwards of one hour. This isn't a problem for late night deployments when no additional servers need to be launched, but causes us to lose valuable time during the day because it prevents us from launching additional servers when the latest AMI isn't ready.
Is there anything out there which can speed up the AMI creation process here? I've heard of Netflix's Aminator and Boxfuse, but are there any other alternatives? Also, how do these services stack up against one another?
As far as I know instance storage on ec2 is ephemeral ie it is lost when the instance is restarted. Is this different for high storage instances? if not it would seem that you would lose any database data stored on the instance upon restart.
Ebs backed volumes should be used for storage. Either you can take standard storage or iops. This will avoid any data losses due to instance going down.
When you attach the ebs volume to instance, make sure a entry is made to fstab file. So when you reboot the instance the volume is still attached. When you stop an instance, the data is present but the hardware is released while reboot is just like your system restarts.
All ephemeral storage is lost if the instance is stopped. You can still use a database with the high IO storage instances, but you will need to replicate the data between several instances to attain persistence.
I'm writing the backend for a mobile app that does some cpu intensive work. We anticipate the app will not have heavy usage most of the time, but will have occasional spikes of high demand. I was thinking what we should do is reserve a couple of 24/7 servers to handle the steady-state of low demand traffic and then add and remove EC2 instances as needed to handle the spikes. The mobile app will first hit a simple load balancing server that does a simple round-robin user distribution among all the available processing servers. The load balancer will handle bringing new EC2 instances up and turning them back off as needed.
Some questions:
I've never written something like this before, does this sound like a good strategy?
What's the best way to handle bringing new EC2 instances up and back down? I was thinking I could just create X instances ahead of time, set them up as needed (install software, etc), and then stop each instance. The load balancer will then start and stop the instances as needed (eg through boto). I think this should be a lot faster and easier than trying to create new instances and install everything through a script or something. Good idea?
One thing I'm concerned about here is the cost of turning EC2 instances off and back on again. I looked at the AWS Usage Report and had difficulty interpreting it. I could see starting a stopped instance being a potentially costly operation. But it seems like since I'm just starting a stopped instance rather than provisioning a new one from scratch it shouldn't be too bad. Does that sound right?
This is a very reasonable strategy. I used it successfully before.
You may want to look at Elastic Load Balancing (ELB) in combination with Auto Scaling. Conceptually the two should solve this exact problem.
Back when I did this around 2010, ELB had some problems with certain types of HTTP requests that prevented us from using it. I understand those issues are resolved.
Since ELB was not an option, we manually launched instances from EBS snapshots as needed and manually added them to an NGinX load balancer. That certainly could have been automated using the AWS APIs, but our peaks were so predictable (end of month) that we just tasked someone to spin up the new instances and didn't get around to automating the task.
When an instance is stopped, I believe the only cost that you pay is for the EBS storage backing the instance and its data. Unless your instances have a huge amount of data associated, the EBS storage charge should be minimal. Perhaps things have changed since I last used AWS, but I would be surprised if this changed much if at all.
First with regards to costs, whether an instance is started from scratch or from a stopped state has no impact on cost. You are billed for the amount of compute units you use over time, period.
Second, what you are looking to do is called autoscaling. What you do is setup up a launch config that specifies an AMI you are going to use (along with any user-data configs you are using, the ELB and availiabilty zones you are going to use, min and max number of instances, etc. You set up a scaling group using that launch config. Then you set up scaling policies to determine what scaling actions are going to be attached to the group. You then attach cloud watch alarms to each of those policies to trigger the scaling actions.
You don't have servers in reserve that you attach to the ELB or anything like that. Everything is based on creating a single AMI that is used as the template for the servers you need.
You should read up on autoscaling at the link below:
http://aws.amazon.com/autoscaling/
Reading http://aws.amazon.com/autoscaling/ it looks like Amazon lets you create more virtual machines (EC2 units) automatically when the load on your existing machine gets high.
However, that's not what I want. I want a single virtual machine that becomes more powerful (more RAM, CPU, etc) when the machine load/memory usage is high. How do I do this?
vps.net appears to offer this:
http://vps.net/product/cloud-servers/
under "scale with demand", but I'd like to find an Amazon equivalent.
You can scale an EC2 instance up and down, but
Any automated trigger for this would need to be written by you, calling the EC2 API calls to perform the scaling
Moving the EC2 instance to a larger or smaller instance type requires shutting down and rebooting the server.
The basic method is:
stop (not terminate) the instance
modify-instance-attributes to change the type
start the instance
reassociate the Elastic IP address (if any).
I've written an article that provides more information, sample commands, and things to watch out for when performing this resize:
http://alestic.com/2011/02/ec2-change-type
We're running a lightweight web app on a single EC2 server instance, which is fine for our needs, but we're wondering about monitoring and restarting it if it goes down.
We have a separate non-Amazon server we'd like to use to monitor the EC2 and start a fresh instance if necessary and shut down the old one. All our user data is on Elastic Storage, so we're not too worried about losing anything.
I was wondering if anyone has any experience of using EC2 in this way, and in particular of automating the process of starting the new instance? We have no problem creating something from scratch, but it seems like it should be a solved problem, so I was wondering if anyone has any tips, links, scripts, tutorials, etc to share.
Thanks.
You should have a look at puppet and its support for AWS. I would also look at the RightScale AWS library as well as this post about starting a server with the RightScale scripts. You may also find this article on web serving with EC2 useful. I have done something similar to this but without the external monitoring, the node monitored itself and shut down when it was no longer needed then a new one would start up later when there was more work to do.
Couple of points:
You MUST MUST MUST back up your Amazon EBS volume.
They claim "better" reliability, but not 100%, and it's SEVERAL orders of magnitude off of S3's "12 9's" of durability. S3 durability >> EBS durability. That's a fact. EBS supports a "snapshots" feature which backs up your storage efficiently and incrementally to S3. Also, with EBS snapshots, you only pay for the compressed deltas, which is typically far far less than the allocated volume size. In another life, I've sent lost-volume emails to smaller customers like you who "thought" that EBS was "durable" and trusted it with the only copy of a mission-critical database... it's heartbreaking.
Your Q: automating start-up of a new instance
The design path you mention is relatively untraveled; here's why... Lots of companies run redundant "hot-spare" instances where the second instance is booted and running. This allows rapid failover (seconds) in the event of "failure" (could be hardware or software). The issue with a "cold-spare" is that it's harder to keep the machine up to date and ready to pick up where the old box left off. More important, it's tricky to VALIDATE that the spare is capable of successfully recovering your production service. Hardware is more reliable than untested software systems. TEST TEST TEST. If you haven't tested your fail-over, it doesn't work.
The simple automation of starting a new EBS instance is easy, bordering on trivial. It's just a one-line bash script calling the EC2 command-line tools. What's tricky is everything on top of that. Such a solution pretty much implies a fully 100% automated deployment process. And this is all specific to your application. Can your app pull down all the data it needs to run (maybe it's stored in S3?). Can you kill you instance today and boot a new instance with 0.000 manual setup/install steps?
Or, you may be talking about a scenario I'll call "re-instancing an EBS volume":
EC2 box dies (root volume is EBS)
Force detach EBS volume
Boot new EC2 instance with the EBS volume
... That mostly works. The gotchas:
Doesn't protect against EBS failures, either total volume loss or an availability loss
Recovery time is O(minutes) assuming everything works just right
Your services need to be configured to restart automatically. It does no good to bring the box back if Nginx isn't running.
Your DNS routes or other services or whatever need to be ok with the IP-address changing. This can be worked around with ElasticIP.
How are your host SSH keys handled? Same name, new host key can break SSH-based automation when it gets the strong-warning for host-key-changed.
I don't have proof of this (other than seeing it happen once), but I believe that EC2/EBS _already_does_this_ automatically for boot-from-EBS instances
Again, the hard part here is on your plate. Can you stop your production service today and bring it up RELIABLY on a new instance? If so, the EC2 part of the story is really really easy.
As a side point:
All our user data is on Elastic Storage, so we're not too worried about losing anything.
I'd strongly suggest to regularly snapshot your EBS (Elastic Block Storage) to S3 if you are not doing that already.
You can use an autoscale group with a min/max/desired quantity of 1. Place the instance behind an ELB and have the autoscale group be triggered by the ELB healthy node count. This allows you to have built in monitoring by cloudwatch and the ELB health check. Anytime there is an issue the instance be replaced by the autoscale service.
If you have not checked 'Protect against accidental termination' you might want to do so.
Even if you have disabled 'Detailed Monitoring' for your instance you should still see the 'StatusCheckFailed' metric for your instance over which you can configure an alarm (In the CloudWatch dashboard)
Your application (hosted in a different server) should receive the alarm and start the instance using the AWS API (or CLI)
Since you have protected against accidental termination you would never need to spawn a new instance.