I see that you can convert an instance stored instance to be EBS backed as this question shows. However, I want to do the opposite, take an EBS backed instance and convert it to be Instance Store backed.
Is this possible? How do I do this?
Launch an instance-store instance from an AMI that uses the same kernel used by your EBS-backed AMI.
Create an EBS volume from the snapshot underlying the EBS AMI. (Alternatively, launch an instance of the EBS AMI and Stop the instance when it begins booting. Detach the root volume from the instance - this is usually the volume attached to /dev/sda1.)
Attach the EBS volume to the instance-store instance.
rsync the contents of the EBS volume to the root volume.
Create an instance-store AMI using the standard methods.
Detach the EBS volume and delete it. If you launched an EBS instance in step 2, terminate it.
You can try this:
# Install ec2-tools (http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip)
$ export EC2_PRIVATE_KEY=/path/to/your/key.pem
$ export EC2_CERT=/path/to/your/sert.pem
$ export AWS_ACCOUNT_ID=XXXX-XXXX-XXXX
$ export AWS_KEY=YOURAWSKEY
$ export AWS_SECRET_KEY=YOURAWSSECRETKEY
$ ec2-bundle-vol -d /tmp -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $AWS_ACCOUNT_ID
$ ec2-upload-bundle -b s3-bucket-name -m /tmp/image.manifest.xml -a $AWS_KEY -s $AWS_SECRET_KEY
$ ec2-register s3-bucket-name/image.manifest.xml -K $EC2_PRIVATE_KEY -C $EC2_CERT --region eu-west-1
# In command above you can choose another region (e.g us-west-1)
# By running command above you will see on your screen something like this: IMAGE ami-3cc1a4a
# Now you can create new instance based on your just registered ami-3cc1a4a using ec2-tools or Amazon Console
Good luck!
Related
I've installed Cloudera cluster on 4 nodes Amazon EC2 instance.
For certain time such as Monday-Friday Night, Saturday, and Sunday, I didn't need to use those 4 nodes Amazon EC2 instance for more effective cost.
How to automate start and stop an those Amazon EC2 instances using script?
Could anybody give me the example of the script to do it?
Thanks,
You can create script to stop and start instance(s) or directly run commands through crontab in linux or schedule-task in windows
for example if you want to stop instance at 11.00 pm
add below line in crontab (you will get this through crontab -e )
0 23 * * * sh stop.sh
format is
m h dom mon dow command
for start instance
aws ec2 start-instances --instance-ids i-1a1234
for stop instance
aws ec2 stop-instances --instance-ids i-1a1234
I have written a small shell script to automate the starting and loggin in to my aws instances via terminal. You can use it
#!/bin/bash
aws ec2 start-instances --instance-ids i-070107834ab273992
public_ip=aws ec2 describe-instances --instance-ids i-070107834ab2 \
--query 'Reservations[*].Instances[*].PublicDnsName' --output text
AWS_KEY="/home/debian/cs605 data management/assignment6/mumbai instance keys"
ssh -v -i "$AWS_KEY"/mumbai-instance-1.pem\
ec2-user#$public_ip
~
~
Is there a system file that authoritatively tells me if a host is an Amazon AWS EC2 instance?
Bonus point: without installing anything new, is there a command that will tell me some basic parameters of an EC2 instance?
Context:
~~~~~~~~
I have a script that gathers information.
If I run the script on an EC2 instance, some "standard" commands are not available,
for example: dmidecode (this gives me practically nothing), or virt-what.
I am aware of /usr/bin/ec2-describe-instances, or wget for metadata script, or wget
for individual components reported by metadata, but I don't want to install anything
new, and I need the tool to describe the host itself (since the script runs locally),
not to inquire another host, or pass a key that I must obtain from yet another script
that I must install first.
Maybe wget for a specific metadata info is the best tool I could use?
Thanks
I cannot guarantee this will work on any other AMI than Ubuntu 12.04 (I am using ami-8e987ef9) - please test yourself.
Here is what you might want to check out if you really want to avoid getting this info from the magic IP 169.254.169.254 and metadata as #Rico suggested:
### Datasource EC2
ubuntu#ip-10-33-59-70:~$ cat /var/lib/cloud/instance/datasource
cloudinit.DataSourceEc2.DataSourceEc2: DataSourceEc2
ubuntu#ip-10-33-59-70:~$ cat /var/lib/cloud/data/previous-datasource
cloudinit.DataSourceEc2.DataSourceEc2: DataSourceEc2
### Hostname
ubuntu#ip-10-33-59-70:~$ cat /var/lib/cloud/data/previous-hostname
ip-10-33-59-70
### Instance ID
ubuntu#ip-10-33-59-70:~$ cat /var/lib/cloud/data/previous-instance-id
i-280ace69
### Instance ID also (check out instance symlink)
ubuntu#ip-10-33-59-70:/var/lib/cloud$ ls -al | grep instance
lrwxrwxrwx 1 root root 22 Jan 29 22:00 instance -> ./instances/i-280ace69
drwxr-xr-x 3 root root 4096 Jan 29 22:00 instances
Maybe looking around /var/lib/cloud will give you some info that you need without using curl.
I would rather suggest using the magic IP with metadata though.
On Ubuntu 12.04 there's also /usr/bin/ec2metadata, a util written in Python that actually queries 169.254.169.254.
Just use the standard way of querying the metadata
curl http://169.254.169.254/latest/meta-data/instance-id
For a list of all the metadata options:
curl http://169.254.169.254/latest/meta-data/
If you get anything other than a '200' then it means it's not an EC2 instance.
From the documentation:
[ec2-user ~]$ cat /sys/hypervisor/uuid
For HVM:
[ec2-user ~]$ sudo dmidecode --string system-uuid
[ec2-user ~]$ sudo cat /sys/devices/virtual/dmi/id/product_uuid
If the strings start with "ec2" or "EC2", the OS is running on an EC2 instance.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html
I am having AWS EC2 instance through which i am creating new AWS EC2 instances using command "ec2-run-instances".
This new instance is pre-configured with EC2 command line API and S3cmd API.
While creating instance i am passing user data to new instance in which i have written code for transferring file from that instance to AWS s3 bucket as follows.
s3cmd put res.doc s3://BucketName/DocFiles/res.doc
but it not transfers res.doc to bucket.
After that i came to know that this script uploads that files which are exist on first EC2 instance from which i creates new instances.
So how can i solve this problem?
Script file is here:-
str=$"#! /bin/bash"
str+=$"\ncd /home"
str+=$"\nmkdir pravin"
str+=$"\ns3cmd put res.doc s3://BuckectName/DocFiles/res.csv"
ud=`echo -e "$str" |base64`
ec2-run-instances ami-784c2823 -t t1.micro -g group -n 1 -k key1 -d "$ud"
Let's say I have an AMI with an attached EBS Volume.
I also have a snapshot.
I want to "restore" the EBS Volume to the snapshot.
What's the best process to do this?
I don't know of a way that you can 'restore' an attached volume, but the way i would do it is to create a volume from the snapshot, then detach the original and attach the new one.
If you have a running EC2 instance, and you want to restore it to the state captured in an earlier snapshot, then you need to stop the instance, detach its current volume, create a new volume from the snapshot, attach the new volume to your instance, and restart your instance. Furthermore, there are a couple subtleties around specifying the availability zone of the new volume, and the device name when detaching/re-attaching the volume.
The logic might easier to see if you do it from the command line, instead of from the AWS web UI.
The following bash script is not fit for production use, since it lacks any error-checking and it just uses sleep instead of polling to ensure AWS commands have completed. But it does perform all these steps successfully:
#!/bin/bash
set -e
# IN PARAMS
INSTANCE_ID=<YOUR_INSTANCE_ID_HERE>
SNAPSHOT_ID=<YOUR_SNAPSHOT_ID_HERE>
# OUT PARAMS
VOLUME_ID=
# begin execution
echo "Gathering information about the instance"
DEVICE_NAME=`ec2-describe-instance-attribute ${INSTANCE_ID} --block-device-mapping | awk '{print $2}'`
OLD_VOLUME_ID=`ec2-describe-instance-attribute ${INSTANCE_ID} --block-device-mapping | awk '{print $3}'`
echo "Found instance ${INSTANCE_ID} has volume ${OLD_VOLUME_ID} on device ${DEVICE_NAME}"
echo "Creating new volume from snapshot"
AVAILABILITY_ZONE=`ec2-describe-availability-zones --filter state=available | head -n 1 | awk '{print $2}'`
VOLUME_ID=`ec2-create-volume --availability-zone ${AVAILABILITY_ZONE} --snapshot ${SNAPSHOT_ID} | awk '{print $2}'`
echo "Created new volume: ${VOLUME_ID}"
sleep 20
echo "Stopping the instance"
ec2-stop-instances $INSTANCE_ID
sleep 20
echo "Detaching current volume"
ec2-detach-volume $OLD_VOLUME_ID --instance $INSTANCE_ID --device $DEVICE_NAME
sleep 20
echo "Attaching new volume"
ec2-attach-volume $VOLUME_ID --instance $INSTANCE_ID --device $DEVICE_NAME
sleep 20
echo "Starting the instance"
ec2-start-instances $INSTANCE_ID
I have touched up the script provided by #algal to use the aws cli and polling instead of sleep. It will also look for the latest snapshot of the given volume.
#!/bin/bash
set -e
# IN PARAMS
RECOVERY_INSTANCE_ID=
SNAPSHOT_VOLUME_ID=
echo "Gathering information about the instance"
BLOCK_DEVICE_MAPPING=`aws ec2 describe-instance-attribute --instance-id ${RECOVERY_INSTANCE_ID} --attribute blockDeviceMapping`
DEVICE_NAME=`echo ${BLOCK_DEVICE_MAPPING} | jq '.BlockDeviceMappings[0].DeviceName' | tr -d '"'`
OLD_VOLUME_ID=`echo ${BLOCK_DEVICE_MAPPING} | jq '.BlockDeviceMappings[0].Ebs.VolumeId' | tr -d '"'`
AVAILABILITY_ZONE=`aws ec2 describe-instances --filters "Name=instance-id,Values='${RECOVERY_INSTANCE_ID}'" | jq '.Reservations[0].Instances[0].Placement.AvailabilityZone' | tr -d '"'`
LATEST_SNAPSHOT_ID=`aws ec2 describe-snapshots --filter "Name=volume-id,Values='${SNAPSHOT_VOLUME_ID}'" | jq '.[]|max_by(.StartTime)|.SnapshotId' | tr -d '"'`
echo "Found instance ${RECOVERY_INSTANCE_ID} in ${AVAILABILITY_ZONE} has volume ${OLD_VOLUME_ID} on device ${DEVICE_NAME}"
echo "Creating new volume from snapshot ${LATEST_SNAPSHOT_ID}"
NEW_VOLUME_ID=`aws ec2 create-volume --region eu-west-1 --availability-zone ${AVAILABILITY_ZONE} --snapshot-id ${LATEST_SNAPSHOT_ID} | jq '.VolumeId' | tr -d '"'`
echo "Created new volume ${NEW_VOLUME_ID}"
aws ec2 wait volume-available --volume-ids $NEW_VOLUME_ID
echo "Stopping the instance"
aws ec2 stop-instances --instance-ids $RECOVERY_INSTANCE_ID
aws ec2 wait instance-stopped --instance-ids $RECOVERY_INSTANCE_ID
echo "Detaching current volume"
aws ec2 detach-volume --volume-id $OLD_VOLUME_ID --instance-id $RECOVERY_INSTANCE_ID
aws ec2 wait volume-available --volume-ids $OLD_VOLUME_ID
echo "Attaching new volume"
aws ec2 attach-volume --volume-id $NEW_VOLUME_ID --instance-id $RECOVERY_INSTANCE_ID --device $DEVICE_NAME
aws ec2 wait volume-in-use --volume-ids $NEW_VOLUME_ID
echo "Starting the instance"
aws ec2 start-instances --instance-ids $RECOVERY_INSTANCE_ID
If you'd like to stay up to date with this script or contribute:
https://github.com/karimtabet/ebs_snapshot_recovery
To replace a volume attached to an instance with a new volume created from a snapshot:
Create a volume from the snapshot in the same availability zone the instance is in (right click on snapshot and click "create volume from snapshot")
Best to stop the instance to avoid any application from crashing. Wait until instance is stopped.
Write down the exact device name of the original volume (it is written in the AWS console under instances view or volumes view)
Detach the old volume, delete it afterwards if you don't need it.
Attach the newly created volume (from the snapshot) to the instance with the same device name.
Start the instance again
Make a volume from the snapshot to mount the volume on an existing EC2 machine and copy files from it.
Check the EC2 machine.
Pick an instance. EC2 tab | INSTANCES | Instances.
Make a note of the EC2 machine’s availability zone.
Create a volume.
Find the snapshot you want to copy files from and tick the box. ELASTIC BLOCK STORE | Snapshots
Click the Create Volume button and fill in the fields.
o The Size must be bigger than the snapshot size (free micro-instances get an 8GB volume).
o The Availability Zone must be the same as the EC2 machine’s.
o The Snapshot is already selected, more or less like snap12345678 - my description.
Click the Yes, Create button. A new line appears in the Volumes table. ELASTIC BLOCK STORE | Volumes
Attach the volume.
Click the Attach Volume button and fill in the fields.
The Volume value is already there.
Pick your machine name i-12345678 (running) from the drop-down list of Instances.
The Devices field shows the first available device name, like /dev/sdf. Does anyone bother changing this value?
Click the Yes, Create button. A new device magically appears on the EC2 machine.
Close the AWS console.
I created a micro ec2 instance. Installed all the necessary web software, mysql and git. Created an AMI out of that instance. Since that instance was using EBS as its root device, it took an EBS snapshot as well when I was creating my AMI.
I terminated that running instance. I then tried creating an instance out that amazon machine image (AMI), the new instance started along with a new EBS volume getting attached to the instance.
Now when I use my key pair to login to that instance via my ssh key to its public DNS address with a command as
ssh -i aws/mykey.pem ubuntu#thepublicdnsname
it says
ssh: connect to host <thepublickdnsname> port 22: Connection refused
Why is this happening. I was able to connect to my first instance with the same keys via ssh. Now the new instance is the exact copy and im not able to login to it. Any help on this ... ? Am I missing something?
I used the same key pairs to create the second new instance from the AMI.
I found that it takes variable amount of time for an EC2 instance to come up and get initialized. One is the time between calling ec2-run-instances till the instance state changes from "pending" to "running". After that there is additional time when ssh server becomes ready. That time can be a couple of minutes.
I had the same issue: my problem was that I had a volume attached to my instance, then I detached the volume and deleted it. I followed aws docs to mount my instance and edited /etc/fstab. This was the problem: when the volume is detached and you try to reboot (or stop and start) the instance it goes to this file and tries to attach the nonexistent volume and ssh daemon is not started.
The solution is: I had to create another instance, detach the volume from problematic instance then edit the mounted_point/etc/fstab file to comment the line where its trying to mount the nonexistent instance, the reattach volume to problematic instance and then it worked everything fine.
This is probably not the answer to the original question, but as this is at the top of Google for connection issues to EC2, be sure to configure your security group to allow SSH2 from your machine as per:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AccessingInstances.html#authorizing-access-to-an-instance
AWS takes time to bring up an instance from AMI. If you try to connect too quickly, and too frequently, the box can't respond. The full script below launches an AMI, determines IP address, and waits until the system is ready to connect. It would work very well for spot instances close to or below current pricing, as the time required to connect can vary widely.
The following loop caused a connection refused error, when the sleep statement was commented out, and it began too quickly after the instance started. It also consumed alot of CPU on the script server, and made huge error logs.
`nc -z $ip_address -w 20 22` 1>/dev/null 2>&1; result=$?;
while [ $result -eq 1 ]
do
#echo $ip_address booting
`nc -z $ip_address -w 30 22` 1>/dev/null 2>&1; result=$?;
sleep 30
done
Here is a complete script to start an instance, tag it, wait for it to boot fully, and connect.
instance_id=$(aws ec2 run-instances --region us-east-1 --count 1 --instance-type $AMItype --image-id $ami --security-group-ids $sg_group --output text --query 'Instances[*].InstanceId' )
aws ec2 create-tags --resources $instance_id --tags "Key=Name, Value=$AMIname
#delay until AWS says instance is running
start_state=0
while [ $start_state -ne 16 ]
do
start_state=$(aws ec2 start-instances --instance-ids $instance_id --query 'StartingInstances[*].PreviousState[*].Code[*]' )
start_state=$(echo $start_state | tr -d '" []')
sleep 10
done
ip_address=$(aws ec2 describe-instances --instance-ids $instance_id --output text --query 'Reservations[*].Instances[*].PrivateIpAddress')
`nc -z $ip_address -w 20 22` 1>/dev/null 2>&1; result=$?;
while [ $result -eq 1 ]
do
#waiting for routing updates and connectivity
`nc -z $ip_address -w 30 22` 1>/dev/null 2>&1; result=$?;
sleep 30
done
Another potential cause for a connection refused port 22 error is misspelling the public dns name. For example, part of mine contained .compute. and I put .computer. and it gave me a port 22 error instead of something more sensible like the host does not exist.
Did you check ip address of the instance? Mine changes every time I run it, unless I chose fixed ip.
I had a different (and honestly very silly) issue. Posting this answer here in case it helps someone else.
As part of my debugging I launched a couple of new instances and none of them could connect so I just rebooted my machine!
It works now! :D
I encountered similar issue, I noticed the instance was created with virtualization of paravirtual and after recrerating with hvm the issue was resolved.
I was having trouble accessing my instance from the internet even though I had all the right security groups, port rules, etc.
The solution was simply to go to the "instances" tab and reboot my instance. A couple minutes later when I retried accessing it from a browser, everything worked as expected.
Somehow, the security changes I made weren't applied until I rebooted the instance.