I'm creating an AMI for a server with 100G files. It's been like an hour and it's still not finished. (The AMI still says pending) Is there something wrong with it? What should I do?
Just to let other people know, this process could take very, very long. My 100 GB AMI takes like 2.5 hours to create and the progress bar jumps from 0 to 100 directly after that. So don't worry.
This answer pertains to a retirement situation where there is potentially a hardware failure for an instance with a EBS root volume. In this situation, AWS recommends creating an AMI of the instance to be retired, and starting a new instance from this AMI. In my situation, the instance was not responding before starting the AMI creation process.
The AMI creation step failed to complete after >8 hours.
I next tried stopping the instance. This also failed to complete in >10 minutes, so I tried force stopping the instance (by issuing the stop command again). The force stop did complete after a few minutes. After the instance was stopped, the AMI creation succeeded in <10 minutes.
This AWS forum message seemed relevant to my case:
https://forums.aws.amazon.com/thread.jspa?messageID=372982񛃶
It fully depends on the disk size that is attached to the es2 instance.
But one can check the progress of the AMI backup job by checking the EBS snapshot, as it will also take a snapshot of the attached EBS. Once taking a snapshot is completed, it will show Available(100%), before that it will show Unavailable(X%). Once it's become Available(100%), AMI backup will complete.
Here are some image of the snapshot progress.
Snapshot progress
Related
I accidentally took snapshot from volume (100GB) that was attached to running instance (as a ROOT device). The snapshot creation took over one hour! After that I created bigger (400GB) volume from that snapshot and everything went fine. Then I:
stopped the instance
detached the old 100GB device from my instance
attached the new 400GB volume to that instance
started the instance
Everything seemed OK in AWS control panel (server running and disk status was fine) but I could not connect to that instance via SSH nor HTTP. SSH error: connection refused. So I changed back to old 100GB disk and instance worked well.
My suggestion was that something went wrong because of taking snapshot from running instance. So I deleted the snapshot and took new snapshot from stopped instance (volume). This time the creation of snapshot took only 5 minutes. Then I just created new volume from that snapshot, attached it to instance and started.. still not working.
So my questions are:
Can it be possible that even when I delete the old snapshot the data of that snapshot was partially used when making new snapshot?
How can I backup my current volume if every time I take a snapshot it uses old corrupted data
All these things were done via AWS control panel.
THANKS!!
Can it be possible that even when I delete the old snapshot the data
of that snapshot was partially used when making new snapshot?
I don't think so. It could be that when you created the first snapshot there were some resources in contention when trying to create it. Thus it took longer. Amazon is not clear about all the resources that they are using under the hood. In a way is good and bad. Good because it takes away most complexities and bad well because you can't see what the hell is going on.
How can I backup my current volume if every time I take a snapshot it
uses old corrupted data
You can either stop your instance before taking the volume snapshot or you can just create an AMI (Image) with making sure that the no reboot option is not checked.
See below:
What is the different between starting an AWS Image and Instances?
Example:
I do notice when I am running AWS image using boto, I can only stop the Image while running AWS instance using boto, I can only terminate.
Think of an EC2 instance as a single running server with CPU, memory, hard disk, networking, etc. Any changes you make to that instance affect only that instance.
Think of an AMI (Amazon Machine Image) as an exact copy of the root file system that gets copied to the hard disk when you start a new instance. The AMI is a hard disk sitting on a shelf. You make an exact copy of the hard disk on the shelf, install the new hard disk in a server, and turn the server on. You can do this for as many servers as you'd like to start without affecting the master copy.
The AMI defines the initial state of each instance. Each instance changes as it runs, but you can never change the original AMI once it has been created (other than to delete it).
There are more details that refine this conceptual model, but that's the basics.
Specific to the wording in your question:
Sometimes we say we're "starting an AMI" sometimes we say we're "starting an instance". We mean the same thing. We're really starting an instance using an AMI as the template.
We never say we're "stopping/terminating an image" or "stopping an ami" as, once started, it's really the instance that's running.
You can have one or more instances running that are derived from an image (AMI). Here is a good little tutorial, that's a bit old mind you, talking about how you can convert an Instance to an AMI ... which you can then redeploy one or many times:
http://webkist.wordpress.com/2010/03/16/creating-an-amazon-ec2-ebs-ami-from-a-running-instance/
What is an AMI: Amazon Machine Image
http://en.wikipedia.org/wiki/Amazon_Machine_Image
Technically, you can't start an AMI. You can start an instance that is derived from an AMI.
We have recently been experimenting with a micro instance to undertake a specific task. Last night, for no apparent reason, the EBS backed micro instance failed with no warning. We can no longer access the server through the Terminal (MacBook) and Monitoring show 100% utilisation (flat). We took a snapshot and created a new volume, but cannot attach this to the new micro instance, as its EBS backed. We are now in the process of setting up a new micro instance from scratch which is taking a lot of effort due to having to re-install software and configure them for our purpose
What are the best practices for running EBS backed micro instances? How can we avoid the same happening again?
You can certainly attach a new EBS boot volume to an EBS-backed instance:
Stop the instance (NOT terminate).
Snapshot its' boot volume.
Create a new volume from the snapshot.
Detach the original boot volume, and attach your new one.
Start the instance.
However, in your case, you don't know why the instance failed, so how do you know that a new boot volume created from a snapshot of the failed one will work? Did you try rebooting the instance? It sounds as though the OS might have panicked/bluescreened (you don't mention which OS is installed) or possibly just the SSH session daemon died, in which case a simple reboot might be all that is needed.
If I stop an EC2 instance, rather than terminate, will this image be saved in my account and be available to be used at a later stage? ..as I noticed terminated instances eventually dissappear, would a stopped instance be available to boot up and start when ever?
Also regarding charging, I assume the cost for having a stopped instance would be charged per GB similar to a custom AMI?
Also another possibly simple question, if I shutdown a machine over SSH or via a script, does this initiate a termination or just a stop an instance (I assume it terminates the instance).
Thanks
If you terminate the EBS backed instance, it will remove it from the list of running instance, including it's allocated EBS volume. Unless you set the instance attribute not to delete the volume.
If you only stop, it will changed to stopped status and you can start it again later.
If you shutdown a machine, it default's to stop.
A good read to protect your instance see: http://alestic.com/2010/01/ec2-instance-locking
If you stop an instance based on EBS, then the instance will terminate automatically but you'll be charged for EBS storage until you delete the EBS.
I am running a vanilla Windows install on Amazon EBS volume. The computer takes 10 minutes to boot, which may be understandable as 2 reboots are required. However, taking a snapshot is also a 10-15 minute process. Can anyone explain this? Any way to speed it up? I am a bit surprised, because I thought that snapshots are immediate replicas of the running EBS volume, in which case shouldn't they take just a couple of seconds to complete?
I will add that the console shows that "snapshot" is completed very quickly. But the "AMI" section is what seems to take 10-20 minutes. What's the difference? Is the snapshot available for use immediately, or do I need to wait for the AMI?
From the EBS product page:
Amazon EBS snapshots are incremental
backups, meaning that only the blocks
on the device that have changed since
your last snapshot will be saved. If
you have a device with 100 GBs of
data, but only 5 GBs of data has
changed since your last snapshot, only
the 5 additional GBs of snapshot data
will be stored back to Amazon S3.
Subsequent snapshots are fast because only the changed blocks need to be saved. So the time it takes scales with the amount of changes since the last snapshot.
Is the snapshot available for use
immediately, or do I need to wait for
the AMI?
Also from the product page:
New volumes created from existing
Amazon S3 snapshots load lazily in the
background. This means that once a
volume is created from a snapshot,
there is no need to wait for all of
the data to transfer from Amazon S3 to
your Amazon EBS volume before your
attached instance can start accessing
the volume and all of its data. If
your instance accesses a piece of data
which hasn’t yet been loaded, the
volume will immediately download the
requested data from Amazon S3, and
then will continue loading the rest of
the volume’s data in the background.
The creation of an AMI is a multiple-step process.
The Snapshot of the current machine is started (that's darn near instantaneous)
The snapshot copies the "changed blocks" from the base AMI to the snapshot lazily (that's pretty quick too)
The underlying Windows image is then prepared to be an AMI base image this starts with booting a "ghost" instance from the image with the snapshot as the disk image.
A SYSPREP is started to "reseal" the machine so it gets new machine SIDs.
The new image is then re-snapshotted
The AMI is marked "complete"