Connection Refused to my Running Amazon EC2 instance - amazon-ec2

I created a micro ec2 instance. Installed all the necessary web software, mysql and git. Created an AMI out of that instance. Since that instance was using EBS as its root device, it took an EBS snapshot as well when I was creating my AMI.
I terminated that running instance. I then tried creating an instance out that amazon machine image (AMI), the new instance started along with a new EBS volume getting attached to the instance.
Now when I use my key pair to login to that instance via my ssh key to its public DNS address with a command as
ssh -i aws/mykey.pem ubuntu#thepublicdnsname
it says
ssh: connect to host <thepublickdnsname> port 22: Connection refused
Why is this happening. I was able to connect to my first instance with the same keys via ssh. Now the new instance is the exact copy and im not able to login to it. Any help on this ... ? Am I missing something?
I used the same key pairs to create the second new instance from the AMI.

I found that it takes variable amount of time for an EC2 instance to come up and get initialized. One is the time between calling ec2-run-instances till the instance state changes from "pending" to "running". After that there is additional time when ssh server becomes ready. That time can be a couple of minutes.

I had the same issue: my problem was that I had a volume attached to my instance, then I detached the volume and deleted it. I followed aws docs to mount my instance and edited /etc/fstab. This was the problem: when the volume is detached and you try to reboot (or stop and start) the instance it goes to this file and tries to attach the nonexistent volume and ssh daemon is not started.
The solution is: I had to create another instance, detach the volume from problematic instance then edit the mounted_point/etc/fstab file to comment the line where its trying to mount the nonexistent instance, the reattach volume to problematic instance and then it worked everything fine.

This is probably not the answer to the original question, but as this is at the top of Google for connection issues to EC2, be sure to configure your security group to allow SSH2 from your machine as per:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AccessingInstances.html#authorizing-access-to-an-instance

AWS takes time to bring up an instance from AMI. If you try to connect too quickly, and too frequently, the box can't respond. The full script below launches an AMI, determines IP address, and waits until the system is ready to connect. It would work very well for spot instances close to or below current pricing, as the time required to connect can vary widely.
The following loop caused a connection refused error, when the sleep statement was commented out, and it began too quickly after the instance started. It also consumed alot of CPU on the script server, and made huge error logs.
`nc -z $ip_address -w 20 22` 1>/dev/null 2>&1; result=$?;
while [ $result -eq 1 ]
do
#echo $ip_address booting
`nc -z $ip_address -w 30 22` 1>/dev/null 2>&1; result=$?;
sleep 30
done
Here is a complete script to start an instance, tag it, wait for it to boot fully, and connect.
instance_id=$(aws ec2 run-instances --region us-east-1 --count 1 --instance-type $AMItype --image-id $ami --security-group-ids $sg_group --output text --query 'Instances[*].InstanceId' )
aws ec2 create-tags --resources $instance_id --tags "Key=Name, Value=$AMIname
#delay until AWS says instance is running
start_state=0
while [ $start_state -ne 16 ]
do
start_state=$(aws ec2 start-instances --instance-ids $instance_id --query 'StartingInstances[*].PreviousState[*].Code[*]' )
start_state=$(echo $start_state | tr -d '" []')
sleep 10
done
ip_address=$(aws ec2 describe-instances --instance-ids $instance_id --output text --query 'Reservations[*].Instances[*].PrivateIpAddress')
`nc -z $ip_address -w 20 22` 1>/dev/null 2>&1; result=$?;
while [ $result -eq 1 ]
do
#waiting for routing updates and connectivity
`nc -z $ip_address -w 30 22` 1>/dev/null 2>&1; result=$?;
sleep 30
done

Another potential cause for a connection refused port 22 error is misspelling the public dns name. For example, part of mine contained .compute. and I put .computer. and it gave me a port 22 error instead of something more sensible like the host does not exist.

Did you check ip address of the instance? Mine changes every time I run it, unless I chose fixed ip.

I had a different (and honestly very silly) issue. Posting this answer here in case it helps someone else.
As part of my debugging I launched a couple of new instances and none of them could connect so I just rebooted my machine!
It works now! :D

I encountered similar issue, I noticed the instance was created with virtualization of paravirtual and after recrerating with hvm the issue was resolved.

I was having trouble accessing my instance from the internet even though I had all the right security groups, port rules, etc.
The solution was simply to go to the "instances" tab and reboot my instance. A couple minutes later when I retried accessing it from a browser, everything worked as expected.
Somehow, the security changes I made weren't applied until I rebooted the instance.

Related

shell script to start ec2 instances and ssh and introducing delay in second command

I have written a small shell script to automate the starting and loggin in to my aws instances via terminal.
#!/bin/bash
aws ec2 start-instances --instance-ids i-070107834ab273992
public_ip=aws ec2 describe-instances --instance-ids i-070107834ab273992 \
--query 'Reservations[*].Instances[*].PublicDnsName' --output text
AWS_KEY="/home/debian/cs605 data management/assignment6/mumbai instance keys"
ssh -v -i "$AWS_KEY"/mumbai-instance-1.pem\
ec2-user#$public_ip
~
~
The problem is public_ip variable I want it to be used in line ssh
1) how do I get value of a variable to use in a command.
2) The instance takes some time to boot when it is switched on from power off to power on so how do I keep checking that instances has been powered on after aws start instance command in the script or retrieve the public ip once it has started fully and then ssh into it.
I am not good at python know just basics so is there a pythonic way of doing it.If there is an example script some where that would be better for me to have a look at it.
You do not set the variable public_ip in your script. It would not surprise me if the script complained about "ec2: command not found".
To set the variable:
public_ip=$(aws ec2 describe-instances --instance-ids i-070107834ab273992 --query 'Reservations[*].Instances[*].PublicDnsName' --output text)
(disclaimer: I have not used aws so I assume that the command is correct).
The information on whether an instance is running should be available with
aws ec2 describe-instance-status
You may want to apply some filters and/or grep for a specific result. You could try polling with a while loop:
while ! aws ec2 describe-instance-statusv --instance-ids i-070107834ab273992 | grep 'something that characterizes running' ; do
sleep 5
done

User is reporting that they've unable to SSH into an EC2 instance in AWS?

The user's are doing the following:
$ ssh -i /Users/user1/key.pem centos#10.12.10.10
The error message received while trying to access is as follows:
$ ssh -i /Users/user1/key.pem centos#10.12.10.10
centos#10.12.10.10 : Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
A novel solution to this particular problem was presented by the AWS support and I felt compelled to share it here, since I hadn't seen it previously. In the past the method most of my colleagues have used revolved around stopping the instance and then mounting the EBS volume attached to it to another EC2 instance as a secondary volume and then mount it and edit it as needed.
These howtos show that traditional method:
https://n2ws.com/blog/how-to-guides/connect-aws-ebs-volume-another-instance
AWS Knowledge Center Videos: How do I recover access to my EC2 instances if I lost my SSH key pair?
Using User Data
This new method to me was to utilize #userdata to for a sequence of commands to run within the EC2 instance as it boots.
Before you begin, please create a snapshot of your volume (Volume ID: vol-XXXX) that's attached to the EC2 you're unable to SSH into, just as a precaution. A snapshot is essentially a backup of your volume. You can find the steps for creating a snapshot in this documentation.
Stop your instance.
In the console, select your instance, go to Actions → Instance Settings → View/Change User Data
Inside User Data, put the information below:
#cloud-config
bootcmd:
- [ chmod, 700, /home/centos ]
- [ chmod, 700, /home/centos/.ssh ]
- [ chmod, 600, /home/centos/.ssh/* ]
- [ chmod, 600, /etc/ssh/ssh_host_*_key ]
- [ chmod, 711, /var/empty/sshd ]
- [ chmod, 600, /home/centos/.ssh/authorized_keys ]
- [ sh, -c, "chown -R centos:centos /home/centos" ]
Start your instance.
Once the instance is restarted, attempt to access the instance via ssh.
Notes
The above user-data script will correct any ownership and permissions issues within the instance itself.
If you are still experiencing an issue after using the above user-data script, please let me know and I will get back to you.
If the previous procedure worked, please repeat the steps 1-4 and delete the content from the User-Data box to avoid the procedure repeats again every instance reboot.

How to access a Docker container on the host by a name (not IP)?

I want to run MS SQL server (docker image: microsoft/mssql-server-windows-developer) in a docker container. Using Windows on the host and the container. Afterwards, the database should be accessible from the host (using SQL Management Studio) by a useful, name (so that the instructions can be re-used). However, docker generates a seemingly random IP, which is not as useful, especially as it resets on every call to run.
So, I would like to give the container a hostname that is accessible from the host machine (e.g. by SQL Management Studio). I'd like to avoid a mere IP here, but it would suffice, if no better solution presents itself.
Creating a network in docker did not work, as this functionality apparently is only supported under Linux.
--network-alias also failed.
The run command looks like this:
docker run -d -p 1433:1433 -e sa_password=1234qwerT -e ACCEPT_EULA=Y --name docker_sql microsoft/mssql-server-windows-developer
This is very similar to this question here: How to get a Docker container's IP address from the host?
I think you can achieve what you want by way of a 2 step process:
Obtain the container id for your container as part of your docker run command.
Use docker inspect to get the container's IP address.
If you really don't want to use the IP address, then you can always add the IP address to your hosts file, but simply using the IP address as a shell variable should be almost as useful.
So, for example, from a bash shell:
CID=$(docker run -d ubuntu /bin/sh -c 'while /bin/true; do sleep 10 ; done')
IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $ID)
Now you can use $IP within scripts as you see fit. (Substitute the CID=... line with whatever docker run command you are using to start your container).
As per bluescores' comment and stumbling upon this related question, I tried and verified that connecting to localhost is possible - so there actually is no need to configure a name for the container-sql-server or to configure its IP.
The general problem might persist for other applications, but for what I want to achieve currently, localhost will suffice.

Cloudformation helper script (bash) is not running as expected

I am trying to setup an Autoscaling group with rolling updates which will kick-in in case of any changes to ec2 launch configurations. I am using a sample template from aws labs. This works fine for Amazon linux AMI. I have changed the template to work for Ubuntu. Now everything runs fine I can see from /var/log/cfn-init.log that my script was initiated properly, below is the script I am using to check health of my ec2 instance:
verify_instance_health:
commands:
ELBHealthCheck:
command: !Sub
'until [[ $state == *InService* ]]; do state=$(aws --region ${AWS::Region} elb describe-instance-health
--load-balancer-name ${ElasticLoadBalancer}
--instances $(curl -s http://169.254.169.254/latest/meta-data/instance-id)
--query InstanceStates[0].State);echo $(date): [$state] >> /tmp/health.log;sleep 10;done'
If I run the script directly from terminal it runs fine and works as expected, but when this runs via cloudformation show how the condition:
until [[ $state = InService ]]
Is never evaluated to be true which makes it keep on running. I have verified from /tmp/health.log that output is "InService". Anyone has faced this issue? What can cause this script to behave differently.

Execute Command after VM Booted

I've gotten sick of how many steps it takes me to get started in the morning. Yes it only takes me a few minutes to start up my whole environment, but I'd really rather just run a single command on boot-up and be ready to go immediately.
I'm writing an app on Rails connected to SqlServer. To develop for it I have a local version of the DB I use on a VM. My manual process goes like this:
Run VirtualBox.
Start the VM.
When the VM is done booting:
Open terminal
Run `rails s`
When rails is done starting:
open browser
navigate to localhost:3000 and start developing
Run Sublime
I'd love to do this in one script:
VirtualBox Windows7 &
sublime &
google-chrome &
But I can't figure out how to run this command only once the VM is done booting:
gnome-terminal --working-directory=git/my_project --tab -e 'rails s' --tab -e 'git status'
Also, it'd be nice (but not necessary) to have chrome start after rails s has succeeded.
Is this even possible?
I'm not opposed to polling, but it feels like this is something VirtualBox should be able to do a bit more naturally.
EDIT
From Comment:
I'm using Host-Only network with two Bridged Interfaces (one for wireless and one for wired) available. (It allows me to use the VM whether or not I'm connected to a network, and lets me freely switch between wired and wireless without noticing the difference).
Here is how I would do:
In the VM, create a script which will find the default gateway, & keep pinging to it. & add it to user's startup. (needs parsing of ipconfig /all which can be done with vbscript/python.)
In host, look at the network interface between host & VM. Find the default gateway on host (parse route -n output in bash script). Since both use same physical interface, the gateway would be same (assuming NAT & ONE physical interface). Use tcpdump, to wait for the ping packets to the gateway.
"Default gateway" was chosen because that was something host & VM can find out independent of each other. Other alternative was to hard-code host's address.
After the host tcpdump on host exits, it means that the VM is alive & booted upto windows desktop.
I looked into this line of inquiry before, and I think Devil's Pie is the closest you can get to setting that up:
http://burtonini.com/blog/computers/devilspie
You could try starting with this (VBoxManager startvm):
How to automatically start and shut down VirtualBox machines?
and then look at some working scripts to add to your init.d and/or rc.local once your VM is up to finish the rest of the job in order:
Get To Know Linux: The /etc/init.d Directory
I needed to orchestrate something similar. I'm using a Windows VM (guest) as a proxy (it runs a Windows-only corporate VPN client) for my Linux laptop (host). The approach is to fully automate the guest and wait until it's ready:
The host must have no funky routes (yet)
The VM starts and runs a powershell script (via Windows Task Scheduler, run-on-startup) that connects the VPN client and sets up ICS (Internet Connection Sharing, basically routing).
The host now adds funky routes that send some traffic via the VM's host-only interface. If it added these routes too soon, step 2 would not work.
The VM also runs squid (http proxy) and its port is NAT port forwarded from the host, so localhost:3128 actually goes to the guest. So a curl using this proxy goes to the corporate network and indicates whether the guest is fully up and connected.
(Squid is also useful as a backup to this complicated but very convenient mechanism, I can still ssh via corkscrew, etc)
So, I run this script on the host (simplified version shown):
#!/bin/bash
VM=vm #Name of the Virtual Machine
SCRIPT_DIR=/some/dir
PROXY_ADDRESS=localhost:3128
REMOTE_CURL_HOST=any.corporate.hostname
function waitloop() {
echo -n "Waiting to hear from $REMOTE_CURL_HOST "
while ! curl -s -m 5 --proxy $PROXY_ADDRESS $REMOTE_CURL_HOST > /dev/null ; do
echo -n .
sleep 10
done
echo "!"
}
# a separate script that takes down my routes, you may not need this.
bash $SCRIPT_DIR/network-config-vboxnet0.sh down
# error is OK if it's already running
vboxmanage startvm $VM
waitloop && bash $SCRIPT_DIR/network-config-vboxnet0.sh up && echo "Completed"
Essentially, the script waits until curl works through the VM.

Resources