I accidentally took snapshot from volume (100GB) that was attached to running instance (as a ROOT device). The snapshot creation took over one hour! After that I created bigger (400GB) volume from that snapshot and everything went fine. Then I:
stopped the instance
detached the old 100GB device from my instance
attached the new 400GB volume to that instance
started the instance
Everything seemed OK in AWS control panel (server running and disk status was fine) but I could not connect to that instance via SSH nor HTTP. SSH error: connection refused. So I changed back to old 100GB disk and instance worked well.
My suggestion was that something went wrong because of taking snapshot from running instance. So I deleted the snapshot and took new snapshot from stopped instance (volume). This time the creation of snapshot took only 5 minutes. Then I just created new volume from that snapshot, attached it to instance and started.. still not working.
So my questions are:
Can it be possible that even when I delete the old snapshot the data of that snapshot was partially used when making new snapshot?
How can I backup my current volume if every time I take a snapshot it uses old corrupted data
All these things were done via AWS control panel.
THANKS!!
Can it be possible that even when I delete the old snapshot the data
of that snapshot was partially used when making new snapshot?
I don't think so. It could be that when you created the first snapshot there were some resources in contention when trying to create it. Thus it took longer. Amazon is not clear about all the resources that they are using under the hood. In a way is good and bad. Good because it takes away most complexities and bad well because you can't see what the hell is going on.
How can I backup my current volume if every time I take a snapshot it
uses old corrupted data
You can either stop your instance before taking the volume snapshot or you can just create an AMI (Image) with making sure that the no reboot option is not checked.
See below:
Related
I'm looking to setup a demo environment in Amazon that consists of a pre-configured EC2 image that resets itself back to a snapshot configuration every hour, this is would be a Linux VM.
What would be the best way to go about doing this in EC2? Does Amazon offer any tools for scheduling and reverting to the snapshot or would this need to be done from a third party VM or software?
There is no VMWare-like 'snapshot' functionality in Amazon EC2 (where you can roll-back to a point-in-time).
The network-attached disk storage system used with Amazon EC2 is called Amazon Elastic Block Store (EBS). While EBS does have a 'snapshot' function, this actually takes a backup of an EBS Volume and stores it in Amazon S3. The snapshot can then be used to create a new EBS volume, which will contain the same contents as the original disk at the time the snapshot was created.
One option would be to launch a new Amazon EC2 instance, which will automatically create a new boot disk from the indicated Amazon Machine Image (AMI). This is the way to launch new machines with the same disk content. However, this might not lend itself well to your "revert every half hour" since it requires a new machine to be started, which will also trigger a new hourly billing cycle.
You might be able to script the deletion of files or the reload of some database tables, but this will depend upon your particular system and applications.
I'm creating an AMI for a server with 100G files. It's been like an hour and it's still not finished. (The AMI still says pending) Is there something wrong with it? What should I do?
Just to let other people know, this process could take very, very long. My 100 GB AMI takes like 2.5 hours to create and the progress bar jumps from 0 to 100 directly after that. So don't worry.
This answer pertains to a retirement situation where there is potentially a hardware failure for an instance with a EBS root volume. In this situation, AWS recommends creating an AMI of the instance to be retired, and starting a new instance from this AMI. In my situation, the instance was not responding before starting the AMI creation process.
The AMI creation step failed to complete after >8 hours.
I next tried stopping the instance. This also failed to complete in >10 minutes, so I tried force stopping the instance (by issuing the stop command again). The force stop did complete after a few minutes. After the instance was stopped, the AMI creation succeeded in <10 minutes.
This AWS forum message seemed relevant to my case:
https://forums.aws.amazon.com/thread.jspa?messageID=372982񛃶
It fully depends on the disk size that is attached to the es2 instance.
But one can check the progress of the AMI backup job by checking the EBS snapshot, as it will also take a snapshot of the attached EBS. Once taking a snapshot is completed, it will show Available(100%), before that it will show Unavailable(X%). Once it's become Available(100%), AMI backup will complete.
Here are some image of the snapshot progress.
Snapshot progress
We have recently been experimenting with a micro instance to undertake a specific task. Last night, for no apparent reason, the EBS backed micro instance failed with no warning. We can no longer access the server through the Terminal (MacBook) and Monitoring show 100% utilisation (flat). We took a snapshot and created a new volume, but cannot attach this to the new micro instance, as its EBS backed. We are now in the process of setting up a new micro instance from scratch which is taking a lot of effort due to having to re-install software and configure them for our purpose
What are the best practices for running EBS backed micro instances? How can we avoid the same happening again?
You can certainly attach a new EBS boot volume to an EBS-backed instance:
Stop the instance (NOT terminate).
Snapshot its' boot volume.
Create a new volume from the snapshot.
Detach the original boot volume, and attach your new one.
Start the instance.
However, in your case, you don't know why the instance failed, so how do you know that a new boot volume created from a snapshot of the failed one will work? Did you try rebooting the instance? It sounds as though the OS might have panicked/bluescreened (you don't mention which OS is installed) or possibly just the SSH session daemon died, in which case a simple reboot might be all that is needed.
I am running a vanilla Windows install on Amazon EBS volume. The computer takes 10 minutes to boot, which may be understandable as 2 reboots are required. However, taking a snapshot is also a 10-15 minute process. Can anyone explain this? Any way to speed it up? I am a bit surprised, because I thought that snapshots are immediate replicas of the running EBS volume, in which case shouldn't they take just a couple of seconds to complete?
I will add that the console shows that "snapshot" is completed very quickly. But the "AMI" section is what seems to take 10-20 minutes. What's the difference? Is the snapshot available for use immediately, or do I need to wait for the AMI?
From the EBS product page:
Amazon EBS snapshots are incremental
backups, meaning that only the blocks
on the device that have changed since
your last snapshot will be saved. If
you have a device with 100 GBs of
data, but only 5 GBs of data has
changed since your last snapshot, only
the 5 additional GBs of snapshot data
will be stored back to Amazon S3.
Subsequent snapshots are fast because only the changed blocks need to be saved. So the time it takes scales with the amount of changes since the last snapshot.
Is the snapshot available for use
immediately, or do I need to wait for
the AMI?
Also from the product page:
New volumes created from existing
Amazon S3 snapshots load lazily in the
background. This means that once a
volume is created from a snapshot,
there is no need to wait for all of
the data to transfer from Amazon S3 to
your Amazon EBS volume before your
attached instance can start accessing
the volume and all of its data. If
your instance accesses a piece of data
which hasn’t yet been loaded, the
volume will immediately download the
requested data from Amazon S3, and
then will continue loading the rest of
the volume’s data in the background.
The creation of an AMI is a multiple-step process.
The Snapshot of the current machine is started (that's darn near instantaneous)
The snapshot copies the "changed blocks" from the base AMI to the snapshot lazily (that's pretty quick too)
The underlying Windows image is then prepared to be an AMI base image this starts with booting a "ghost" instance from the image with the snapshot as the disk image.
A SYSPREP is started to "reseal" the machine so it gets new machine SIDs.
The new image is then re-snapshotted
The AMI is marked "complete"
I am new to Amazon EC2 and just pulled down some Unix based image (I think it was an IBM image). It launched fine, worked great and was able to be stopped and snapshotted, all through the console. However, when I replicated the snapshot and launched a new image, albeit of a smaller size, the new image came up with errors. Is there something I need to do before snapshotting an image to make it restartable?
YOu should pause, the processes writing to the EBS volume, so that the EBS volume is in a consistent state.