I am trying to setup an Autoscaling group with rolling updates which will kick-in in case of any changes to ec2 launch configurations. I am using a sample template from aws labs. This works fine for Amazon linux AMI. I have changed the template to work for Ubuntu. Now everything runs fine I can see from /var/log/cfn-init.log that my script was initiated properly, below is the script I am using to check health of my ec2 instance:
verify_instance_health:
commands:
ELBHealthCheck:
command: !Sub
'until [[ $state == *InService* ]]; do state=$(aws --region ${AWS::Region} elb describe-instance-health
--load-balancer-name ${ElasticLoadBalancer}
--instances $(curl -s http://169.254.169.254/latest/meta-data/instance-id)
--query InstanceStates[0].State);echo $(date): [$state] >> /tmp/health.log;sleep 10;done'
If I run the script directly from terminal it runs fine and works as expected, but when this runs via cloudformation show how the condition:
until [[ $state = InService ]]
Is never evaluated to be true which makes it keep on running. I have verified from /tmp/health.log that output is "InService". Anyone has faced this issue? What can cause this script to behave differently.
Related
In my company when we SSH to our AWS EC2 instances we are required to use the aws CLI session-manager plugin for auth. Using this SSH config snippet works:
Host my-aws-host
ProxyCommand bash -c "aws ssm start-session --target 'i-0abc123def456hij' \
--document-name AWS-StartSSHSession --parameters 'portNumber=22' \
--region us-west-1 --profile MAIN"
However, when the EC2 instance is relaunched, which happens semi-regularly, the 'target' instance ID changes. When this happens, all users need to update their SSH config with the new ID. We don't have any sort of DNS that resolves these instances to a static hostname unfortunately, and so would need to somehow publish the new instance ID to all interested users.
So instead I wrote a bash script (ssh_proxy_command.sh) that first queries our AWS account to grab the current instance ID based on a known tag value, and use that for the target - here's a cut-down version:
#!/bin/bash
INSTANCE_ID=$(aws ec2 describe-instances --region us-west-1 \
--filters Name=tag:Name,Values=my-server-nametag* \
--query "Reservations[*].Instances[*].{Instance:InstanceId}" --output text)
aws ssm start-session --target $INSTANCE_ID --document-name AWS-StartSSHSession --parameters 'portNumber=22' --region us-west-1 --profile MAIN
Now the SSH config looks like
Host my-aws-host
ProxyCommand bash -c "/path/to/my/ssh_proxy_command.sh %h"
This has been working fine. However, we have just started running multiple instances built from the same base image (AMI), and which use the same tags, etc. so the given describe-instances query now returns multiple instance IDs. So I tried wrapping the output returned by the query in a bash select loop, thinking I could offer the user a list of instance IDs and let them choose the one they want. This works when running the script directly, but not when it's used as the ProxyCommand. In the latter case when it reaches the select statement it prints out the options as expected, but doesn't wait for the user input - it just continues straight to the end of the script with an empty $INSTANCE_ID variable, which makes the aws ssm command fail.
I'm guessing this is a side-effect of the way SSH runs its ProxyCommands — from the ssh_config man page:
[the proxy command] is executed using the user's shell ‘exec’ directive [...]
I'm hoping I can find a way around this problem while still using SSH config and ProxyCommand, rather than resorting to a complete stand-alone wrapper around the ssh executable and requiring everyone use that. Any suggestions gratefully accepted...
host my-aws-host
ProxyCommand aws ssm start-session --target $(aws ec2 describe-instances --filter "Name=tag:Name,Values=%h" --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" --output text) --document-name AWS-StartSSHSession --parameters portNumber=%p
The above will dynamically filter for your target-id based on host name (%h) provided so you can login using ssh my-aws-host. I personally have a prefix for all my machines in AWS so I ssh config is as follows:
host your-custom-prefix-*
ProxyCommand aws ssm start-session --target $(aws ec2 describe-instances --filter "Name=tag:Name,Values=%h" --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" --output text) --document-name AWS-StartSSHSession --parameters portNumber=%p
This works only when the name of your machines in AWS match the host name provided.
I have written a small shell script to automate the starting and loggin in to my aws instances via terminal.
#!/bin/bash
aws ec2 start-instances --instance-ids i-070107834ab273992
public_ip=aws ec2 describe-instances --instance-ids i-070107834ab273992 \
--query 'Reservations[*].Instances[*].PublicDnsName' --output text
AWS_KEY="/home/debian/cs605 data management/assignment6/mumbai instance keys"
ssh -v -i "$AWS_KEY"/mumbai-instance-1.pem\
ec2-user#$public_ip
~
~
The problem is public_ip variable I want it to be used in line ssh
1) how do I get value of a variable to use in a command.
2) The instance takes some time to boot when it is switched on from power off to power on so how do I keep checking that instances has been powered on after aws start instance command in the script or retrieve the public ip once it has started fully and then ssh into it.
I am not good at python know just basics so is there a pythonic way of doing it.If there is an example script some where that would be better for me to have a look at it.
You do not set the variable public_ip in your script. It would not surprise me if the script complained about "ec2: command not found".
To set the variable:
public_ip=$(aws ec2 describe-instances --instance-ids i-070107834ab273992 --query 'Reservations[*].Instances[*].PublicDnsName' --output text)
(disclaimer: I have not used aws so I assume that the command is correct).
The information on whether an instance is running should be available with
aws ec2 describe-instance-status
You may want to apply some filters and/or grep for a specific result. You could try polling with a while loop:
while ! aws ec2 describe-instance-statusv --instance-ids i-070107834ab273992 | grep 'something that characterizes running' ; do
sleep 5
done
There is a method to setup an EC2 machine to execute Kafka starting script on startup?
I use also java Aws SDK, so I accept both solution for a program java that run command on EC2 instance and solutions for a bash script mode that run kafka script at startup.
A script can be passed in the User Data property.
If you are using the Amazon Linux AMI, and the first line of the script begins with #!, then the script will be executed the first time that the instance is started.
For details, see: Running Commands on Your Linux Instance at Launch
Adding a script under User Data in CloudFormation only runs once, right when the instance is launched but not when the instance is restarted which is what I needed for my use case. I use the rc.local approach as commented above and here. The following in effect appends my script to the rc.local file and performs as expected:
Resources:
VM:
Type: 'AWS::EC2::Instance'
Properties:
[...]
UserData:
'Fn::Base64': !Sub |
#!/bin/bash -x
echo 'INSTANCEID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id)"' >> /etc/rc.local
#echo 'INSTANCEID=$(ls /var/lib/cloud/instances)' >> /etc/rc.local
echo 'echo "aws ec2 stop-instances --instance-ids $INSTANCEID --region ${AWS::Region}" | at now + ${Lifetime} minutes' >> /etc/rc.local
Additional tip: You can inspect the user data (the current script) and modify it using the AWS console by following these instructions: View and update the instance user data.
What is the OS of EC2 instance?
You could use userdata script at instance launch time. Remember this is just 1time activity
If your requirement is to start the script everytime you reboot EC2 instance then you could make use of rc.local file on Linux instances which is loaded at OS boot time.
rc.local didn't work for me.
I used crontab following this guide, which sets up a cron job that can run a script on start up.
https://phoenixnap.com/kb/crontab-reboot
It's essentially
crontab -e
<select editor>
#reboot <script to run>
If you are running a windows EC2 you'll want to read: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-windows-user-data.html
Example:
<script>
echo Current date and time >> %SystemRoot%\Temp\test.log
echo %DATE% %TIME% >> %SystemRoot%\Temp\test.log
</script>
I want to fill some tags of the EC2 spot instance, however as it is impossible to do it directly in spot request, I do it via user data script. All is going fine when I specify region statically, but it is not universal approach. When I try to detect current region from instance userdata, the region variable is always empty. I do it in a following way:
#!/bin/bash
region=$(ec2-metadata -z | awk '{print $2}' | sed 's/[a-z]$//')
aws ec2 create-tags \
--region $region \
--resources `wget -q -O - http://169.254.169.254/latest/meta-data/instance-id` \
--tags Key=sometag,Value=somevalue Key=sometag,Value=somevalue
I tried to made a delay before region populating
/bin/sleep 30
but this had no result.
However, when I run this script manually after start, the tags are added fine. What is going on?
Why all in all aws-cli doesn't get default region from profile? I have aws configure properly configured inside the instance, but without --region clause it throws error that region is not specified.
I suspect the ec2-metadata command is not available when your userdata script is executed. Try getting the region from the metadata server directly (which is what ec2-metadata does anyway)
region=$(curl -fsq http://169.254.169.254/latest/meta-data/placement/availability-zone | sed 's/[a-z]$//')
AWS CLI does use the region from default profile.
You can now use this endpoint to get only the instance region (no parsing needed):
http://169.254.169.254/latest/meta-data/placement/region
So in this case:
region=`curl -s http://169.254.169.254/latest/meta-data/placement/region`
I ended up with
region=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | python -c "import json,sys; print"
which worked fine. However, it would be fine if somebody explain the nuts-and-bolts.
I created a micro ec2 instance. Installed all the necessary web software, mysql and git. Created an AMI out of that instance. Since that instance was using EBS as its root device, it took an EBS snapshot as well when I was creating my AMI.
I terminated that running instance. I then tried creating an instance out that amazon machine image (AMI), the new instance started along with a new EBS volume getting attached to the instance.
Now when I use my key pair to login to that instance via my ssh key to its public DNS address with a command as
ssh -i aws/mykey.pem ubuntu#thepublicdnsname
it says
ssh: connect to host <thepublickdnsname> port 22: Connection refused
Why is this happening. I was able to connect to my first instance with the same keys via ssh. Now the new instance is the exact copy and im not able to login to it. Any help on this ... ? Am I missing something?
I used the same key pairs to create the second new instance from the AMI.
I found that it takes variable amount of time for an EC2 instance to come up and get initialized. One is the time between calling ec2-run-instances till the instance state changes from "pending" to "running". After that there is additional time when ssh server becomes ready. That time can be a couple of minutes.
I had the same issue: my problem was that I had a volume attached to my instance, then I detached the volume and deleted it. I followed aws docs to mount my instance and edited /etc/fstab. This was the problem: when the volume is detached and you try to reboot (or stop and start) the instance it goes to this file and tries to attach the nonexistent volume and ssh daemon is not started.
The solution is: I had to create another instance, detach the volume from problematic instance then edit the mounted_point/etc/fstab file to comment the line where its trying to mount the nonexistent instance, the reattach volume to problematic instance and then it worked everything fine.
This is probably not the answer to the original question, but as this is at the top of Google for connection issues to EC2, be sure to configure your security group to allow SSH2 from your machine as per:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AccessingInstances.html#authorizing-access-to-an-instance
AWS takes time to bring up an instance from AMI. If you try to connect too quickly, and too frequently, the box can't respond. The full script below launches an AMI, determines IP address, and waits until the system is ready to connect. It would work very well for spot instances close to or below current pricing, as the time required to connect can vary widely.
The following loop caused a connection refused error, when the sleep statement was commented out, and it began too quickly after the instance started. It also consumed alot of CPU on the script server, and made huge error logs.
`nc -z $ip_address -w 20 22` 1>/dev/null 2>&1; result=$?;
while [ $result -eq 1 ]
do
#echo $ip_address booting
`nc -z $ip_address -w 30 22` 1>/dev/null 2>&1; result=$?;
sleep 30
done
Here is a complete script to start an instance, tag it, wait for it to boot fully, and connect.
instance_id=$(aws ec2 run-instances --region us-east-1 --count 1 --instance-type $AMItype --image-id $ami --security-group-ids $sg_group --output text --query 'Instances[*].InstanceId' )
aws ec2 create-tags --resources $instance_id --tags "Key=Name, Value=$AMIname
#delay until AWS says instance is running
start_state=0
while [ $start_state -ne 16 ]
do
start_state=$(aws ec2 start-instances --instance-ids $instance_id --query 'StartingInstances[*].PreviousState[*].Code[*]' )
start_state=$(echo $start_state | tr -d '" []')
sleep 10
done
ip_address=$(aws ec2 describe-instances --instance-ids $instance_id --output text --query 'Reservations[*].Instances[*].PrivateIpAddress')
`nc -z $ip_address -w 20 22` 1>/dev/null 2>&1; result=$?;
while [ $result -eq 1 ]
do
#waiting for routing updates and connectivity
`nc -z $ip_address -w 30 22` 1>/dev/null 2>&1; result=$?;
sleep 30
done
Another potential cause for a connection refused port 22 error is misspelling the public dns name. For example, part of mine contained .compute. and I put .computer. and it gave me a port 22 error instead of something more sensible like the host does not exist.
Did you check ip address of the instance? Mine changes every time I run it, unless I chose fixed ip.
I had a different (and honestly very silly) issue. Posting this answer here in case it helps someone else.
As part of my debugging I launched a couple of new instances and none of them could connect so I just rebooted my machine!
It works now! :D
I encountered similar issue, I noticed the instance was created with virtualization of paravirtual and after recrerating with hvm the issue was resolved.
I was having trouble accessing my instance from the internet even though I had all the right security groups, port rules, etc.
The solution was simply to go to the "instances" tab and reboot my instance. A couple minutes later when I retried accessing it from a browser, everything worked as expected.
Somehow, the security changes I made weren't applied until I rebooted the instance.