My instance swap file is disappearing when I start my instance.
I have an Ubuntu ec2 instance, and I follow the "Four-step Process to Add Swap File" instructions at https://help.ubuntu.com/community/SwapFaq:
sudo dd if=/dev/zero of=/mnt/512MiB.swap bs=1024 count=524288
sudo chmod 600 /mnt/512MiB.swap
sudo mkswap /mnt/512MiB.swap
sudo swapon /mnt/512MiB.swap
I then changed my /etc/fstab to include:
/mnt/512MiB.swap none swap sw 0 0
Since I am using a much bigger swap, this process takes some time, and I don't want to do it every time I start. I would rather pay for the storage. However, when I start my instance, the swap has disappeared. If I type 'top', the instance does not have a swap file in use.
What should I do?
While the Amazon EC2 instance you are using has EBS backed Root Device Storage, all EC2 instance types still have the EC2 instance storage (also known as an ephemeral store) available for use as well, and the smaller instance types (e.g. m1.small and c1.medium) have it attached and mounted at /mnt by default even (the larger ones not!).
The most important characteristic of this storage type to be aware of is, that the data on the instance store volumes persists only during the life of the associated Amazon EC2 instance.
This statement is nowadays a tiny bit misleading, insofar it applies to stopping an EBS backed instance as well (not rebooting though), i.e. the moment you stop that instance, the ephemeral volume mounted at /mnt is detached and deleted and all data stored there is lost, including your swap file of course; once you start the instance again, a new ephemeral volume will be attached and mounted at /mnt.
Solution
You can still use the EC2 instance storage (which is plentiful and free of charge) if you exactly know what you are doing (see section Background below), e.g. it is a perfect option for strictly temporary data or anything that can be recreated easily on demand, like a cache for example.
A swap file is matching this requirements as well of course, so you simply need to create a script with the commands outlined in your question and execute it on instance start to recreate the swap file. You should put a guard in place though, because the instance storage survives reboots, i.e. you neither need nor should recreate the swap file on reboots, just with real stop/start cycles.
Background
The instance storage used to be the only storage option when Amazon EC2 was first introduced, but the resulting severe limitations for everyday usage have fortunately been remedied with the Amazon Elastic Block Store (EBS) you are using as well accordingly. Eric Hammond has recently provided a great summary why You Should Use EBS Boot Instances on Amazon EC2, addressing this very topic:
If you are just getting started with Amazon EC2, then use EBS boot
instances and stop reading this article. Forget that you ever heard
about instance-store and accept my apology that I just mentioned it.
Once you are completely comfortable with using EBS boot instances on
EC2, you may (or may not) want to come back here and read why you made
a good decision.
Related
I ran into some issues with my EC2 micro instance and had to terminate it and create a new one in its place. But it seems even though the old instance is no longer visible in the list, it is still using up some space on my disk. My df -h is listed below:
Filesystem Size Used Avail Use%
/dev/xvda1 7.8G 7.0G 719M 91% /
When I go to the EC22 console I see there are 3 volumes each 8gb in the list. One of them is attached (/dev/xvda) and this one is showing as "in-use". The other 2 are simply showing as "Available"
Is the terminated instance really using up my disk space? If yes, how to free it up?
I have just solved my problem by running this command:
sudo apt autoremove
and a lot of old packages are going to be removed, for instance many files like this linux-aws-headers-4.4.0-1028
Amazon Elastic Block Storage (EBS) is a service that provides virtual disks for use with Amazon EC2. It is network-attached storage that persists even when an EC2 instance is stopped or terminated.
When launching an Amazon EC2 instance, a boot volume is automatically attached to the instance. The contents of the boot volume is copied from an Amazon Machine Image (AMI), which can be chosen from a pre-populated list (including the ability to create your own AMI).
When an Amazon EC2 instance is Stopped, all EBS volumes remain attached to the instance. This allows the instance to be Started with the same configuration as when it was stopped.
When an Amazon EC2 instance is Terminated, EBS volumes might or might not be deleted, based upon the Delete on Termination setting of each volume:
By default, boot volumes are deleted when an instance is terminated. This is because the volume was originally just a copy of an AMI, so there is unlikely to be any important data on the volume. (Hint: Don't store data on a boot volume.)
Additional volumes default to "do not delete on termination", on the assumption that they contain data that should be retained. When the instance is terminated, these volumes will remain in an Available state, ready to be attached to another instance.
So, if you do not require any content on your remaining EBS volumes, simply delete them. In future, when launching instances, keep an eye on the Delete on Termination setting to make the clean-up process simpler.
Please note that the df -h command is only showing currently-attached volumes. It is not showing the volumes in Available state, since they are not visible to that instance. The concept of "Disk Space" typical refers to the space within an EBS volume, while "EBS Storage" refers to the volumes themselves. So, the 7GB of the volume that is used is related to that specific (boot) volume.
If you are running out of space on an EBS volume, see: Expanding the Storage Space of an EBS Volume on Linux. Expanding the volume involves:
Creating a snapshot
Creating a new (bigger) volume from the snapshot
Swapping the disks (requiring a Stop/Start if you are swapping a boot volume)
These 2 steps add an extra hard drive to your EC2 and format it for use:
Attach an extra hard drive (EBS: Elastic Block Storage) to an EC2
Format an EBS drive attached to an EC2
Here's pricing info. Free Tier includes 30GB. Afterward it's $1.25/month for 10GB on a General Purpose SSD (gp2).
To see how much space you are using/need:
Check your current disk use/available in Linux with df -h.
Check the size of a directory in Linux with du -sh [path].
I have a Windows 2008 EC2 instance to which I have done some customizing on the EBS boot drive.
I started the instance as m1.small (or m1.large) and the instance storage does not appear as an additional drive.
I've read that the -b switch in the ec2-run-instances command allows you to create mappings for the ephymeral instance storage. The ec2-run-instances command creates a new instance, however, in my case, the instance already exists and therefore I start it as ec2-start-instances, which does not have a -b switch for ephymeral instance storage.
Is there any way I can get to the ephymeral instance storage that comes with an m1.small instance for my existing EBS-booted instance?
UPDATE: It seems that nowadays (Feb 2015) Windows machines mount ephymeral instance storage in the Z: drive.
I'm afraid this functionality isn't available (yet) for Amazon EC2, but it's a very good question in fact - the common answer used to refer to the explicated launch time requirement, see e.g. ec2-modify-instance-attribute:
Note
If you want to add ephemeral storage to an Amazon EBS-backed instance,
you must add the ephemeral storage at the time you launch the
instance. For more information, go to Overriding the AMI's Block
Device Mapping in the Amazon Elastic Compute Cloud User Guide, or to
Adding A Default Instance Store in the Amazon Elastic Compute Cloud
User Guide. [emphasis mine]
That hasn't been that much of an issue in the past, but given the recent introduction of 64-bit ubiquity implies a significant improvement of vertical scaling versatility (see EC2 Updates: New Medium Instance, 64-bit Ubiquity, SSH Client), this is suddenly a topic indeed - your question yields even more questions in turn:
What happens for the converse case, i.e. when I start a sufficiently large instance with lots of ephemeral storage and scale it down (and possibly up again) thereafter?
In case the initial block device mapping is retained somehow, should we always start with a large instance therefore? (I actually doubt that this is the case though.)
This question can only be addressed by the AWS team I guess, so you may want to file a support request or relay the question to the Amazon Elastic Compute Cloud forum at least.
I think what you're asking (but correct me if I'm wrong) is "how do I add additional storage to an EC2 instance?".
In which case, the answer is:
Select the Volumes panel in the AWS console and create a new volume of the size you want, making sure it's in the same Availability Zone as the instance you want to attach it to. Then select that new Volume, and click 'Attach' - select the instance you want to attach it to, and click OK.
Now log-on to the instance, and in Computer Management select the Disk Management plugin, format the new unassigned partition, and give it whatever drive letter you wish. It will then show up in Explorer as a standard Windows drive.
What is the different between starting an AWS Image and Instances?
Example:
I do notice when I am running AWS image using boto, I can only stop the Image while running AWS instance using boto, I can only terminate.
Think of an EC2 instance as a single running server with CPU, memory, hard disk, networking, etc. Any changes you make to that instance affect only that instance.
Think of an AMI (Amazon Machine Image) as an exact copy of the root file system that gets copied to the hard disk when you start a new instance. The AMI is a hard disk sitting on a shelf. You make an exact copy of the hard disk on the shelf, install the new hard disk in a server, and turn the server on. You can do this for as many servers as you'd like to start without affecting the master copy.
The AMI defines the initial state of each instance. Each instance changes as it runs, but you can never change the original AMI once it has been created (other than to delete it).
There are more details that refine this conceptual model, but that's the basics.
Specific to the wording in your question:
Sometimes we say we're "starting an AMI" sometimes we say we're "starting an instance". We mean the same thing. We're really starting an instance using an AMI as the template.
We never say we're "stopping/terminating an image" or "stopping an ami" as, once started, it's really the instance that's running.
You can have one or more instances running that are derived from an image (AMI). Here is a good little tutorial, that's a bit old mind you, talking about how you can convert an Instance to an AMI ... which you can then redeploy one or many times:
http://webkist.wordpress.com/2010/03/16/creating-an-amazon-ec2-ebs-ami-from-a-running-instance/
What is an AMI: Amazon Machine Image
http://en.wikipedia.org/wiki/Amazon_Machine_Image
Technically, you can't start an AMI. You can start an instance that is derived from an AMI.
Amazon's new EBS-based EC2 instances have two options to shutdown: terminate or stop. Stopped instances can be later started again, automatically continuing from the same EBS root disk state they had when they were stopped.
But what happens when an Amazon datacenter has a hardware failure, and the EC2 instance is forced to shutdown. Does it terminate or stop? If the instance has been configured to stop by default on shutdown, can I rely on it being stopped also in this situation, and being able to start it again later?
An EC2 instance can be terminated at any time and one must account for this indeed, as already mentioned in David's answer (+1). You can arrange for a failed instance's Elastic Block Store (EBS) to remain available regardless though, see e.g. the respective FAQ What happens to my data when a system terminates?:
The data stored on a local instance store will persist only as long as
that instance is alive. However, data that is stored on an Amazon EBS
volume will persist independently of the life of the instance. If you
are using an Amazon EBS volume as a root partition, then you have set
the Delete On Terminate flag to "N" for your Amazon EBS volume to
persist outside the life of the instance. [emphasis mine]
This is explained in more detail in section 2. Delete on Termination within Eric Hammond's recommended article Three Ways to Protect EC2 Instances from Accidental Termination and Loss of Data:
Though EBS volumes created and attached to an instance at
instantiation are preserved through a “stop”/”start” cycle, by default
they are destroyed and lost when an EC2 instance is terminated. This
behavior can be changed with a delete-on-termination boolean value
buried in the documentation for the --block-device-mapping option of
ec2-run-instances.
He is referring to the ec2-run-instances documentation, and all this is meanwhile illustrated in more detail within Amazon EC2 Root Device Storage Concepts as well:
By default, the root device volume and the other volumes created when
an Amazon EBS-backed instance is launched are automatically deleted
when the instance terminates [...]. You can change
the default behavior by setting the DeleteOnTermination flag to the
value you want when you launch the instance. For an example of how to
change the flag at launch time, see Using Amazon EC2 Root Device Storage.
I assume you mean the CPU related hardware fails, rather than the network disk. The way I treat EC2 is to create a system that can go up and down without data loss. Anything important you should use an S3 bucket, not EBS.
EBS root instances seem to not have any instance storage - is this correct or some odd bug?
I'm expecting to have /dev/sda to be the EBS backed volume, and then have the other instance storage devices there as /dev/sdb etc...
Is this an incorrect assumption?
It's not a bug. Unlike Ephemeral, EBS based instances don't give you any free storage.
You can however create a new EBS volume and attach it to your instance.
Cheers
Ray
EDIT: I believe I led you astray as this seems to have changed. You still cannot attach local disk to an existing running instance but this does seem to be possible at launch time.
See:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?Using_AddingDefaultLocalInstanceStorageToAMI.html
Taken from the above url, device mapping with an 80gb EBS root volume plus ephemeral storage.
https://ec2.amazonaws.com/
?Action=RegisterImage
&Name=MyImage
&KernelId=aki-f70657b2
&RamdiskId=ari-ff0d5cba
&RootDeviceName=/dev/sda1
&BlockDeviceMapping.1.DeviceName=/dev/sda1
&BlockDeviceMapping.1.Ebs.SnapshotId=snap-12345678
&BlockDeviceMapping.1.Ebs.VolumeSize=80
&BlockDeviceMapping.1.Ebs.DeleteOnTermination=false
&BlockDeviceMapping.2.DeviceName=/dev/sdc
&BlockDeviceMapping.2.VirtualName=ephemeral0
&...auth parameters...