I am using AWS opsworks for statically configuring a simple stack, consisting of two layers (Rails app server and mySQL db).
After having successfully configured and started the stack and deploying my app, I would like to automate the start activity for the stack as part of my pipeline. AWS CLI provides the features to start stacks, retrieve the instance id's of the individual servers and then poll AWS for the completion status ("instance-running"), using the EC2 wait command.
The script below is what I am using (the first command starts the stack, the second command retrieves the instance id's for the two hosts, the third initiates the wait command for these two servers):
#!/bin/bash
aws opsworks --region us-east-1 start-stack --stack-id 9e1b0534-5b38-4fa5-b30c-f849dda8f46b
instance_id=$(aws opsworks --region us-east-1 describe-instances --stack-id 9e1b0534-5b38-4fa5-b30c-f849dda8f46b --query "Instances[].Ec2InstanceId" --output text)
aws ec2 wait --region ap-southeast-1 instance-running --instance-ids $instance_id
When running this script, I always get a "InvalidInstanceID" exception on one of the two id's even though it is definitely the right ID. Secondly, if running the last command in the shell directly while starting the stack in parallel via the AWS console, it turns out that the wait command returns BEFORE the servers are actually up and running (which is the whole point of the exercise).
Lastly, I couldn't find any information about time outs which seem to be quite essential for a blocking async operation. where can the wait timeout be defined ?
Any idea whether there is a glitch in my code, or some specific consideration that I need to take into account?
The aws opsworks describe-instances command is using the --region us-east-1, but the aws ec2 wait command is using --region ap-southeast-1. Are you sure that your instances you're waiting for are in ap-southeast-1 as opposed to us-east-1?
Related
I am writing bash script to install missing patches on ec2 instance using session manager.
I can start the session using script but I am not sure how can I run command on it using script?
instanceid = "i-098xxxx"
echo $instanceid
echo "instance id"
# Creating AMI
echo "-------------------------------Creation AMI-----------------------------------------"
aws ec2 create-image --instance-id $instanceid --name "test ami" --description "Delete when
testing" --no-reboot --profile xyz
# Start Session Manager
aws ssm start-session --target $instanceid --profile xyz
# Command to check for critical security patches and update/install it on ec2 instance
echo "-------------------------------installing missing patches-----------------------------
------------"
sudo yum update-minimal --sec-severity=critical,important --bugfix
I want to run sudo yum update-minimal --sec-severity=critical,important --bugfix command in ec2 instance using script.
Can someone guide me on this?
As an alternative to using run-command, if for some reason you need to use a session, you can do this by using the AWS-StartNonInteractiveCommand document, for example:
aws ssm start-session \
--document-name 'AWS-StartNonInteractiveCommand' \
--parameters '{"command": ["sudo yum -y update amazon-ssm-agent"]}' \
--target "$instanceid"
To see the other parameters accepted by this document, run
aws ssm describe-document --name AWS-StartNonInteractiveCommand
The AWS Systems Manager Session Manager provides an SSH-like connection via a web browser.
However, if you wish to automate the execution of commands on instances, you should use the AWS Systems Manager Run Command, which can run commands on a single instance or hundreds of instances, and return results from each run.
I think AWS Systems Manager is best way to resolve this case.
https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html
Trying to run few steps of CI/CD in a EC2 instance. Please don't ask for reasons.
Need to:
1) Start an instance using AWS CLI. Set few environment variables.
2) Run few bash commands.
3) Stream the command from the above commands into the console of the caller script.
4) If any of the commands fail, need to fail the calling script as well.
5) Terminate the instance.
There is a SO thread which indicates that streaming the output is not as easy. [1]
What I would do, if I had to implement this task:
Start the instance using the cli command aws ec2 run-instances and using an AMI which has the AWS SSM agent preinstalled. [2]
Run your commands using AWS SSM. [3] This has the benefit that you can run any number of commands you want - whenever you want (i.e. the commands must not be specified at instance launch, but can be chosen afterwards). You also get the status code of each command.[4]
Use the CloudWatch integration in SSM to stream your command output to CloudWatch logs. [5]
Stream the logs from CloudWatch to your own instance. [6]
Note: Instead of streaming the command output via CloudWatch, you could also periodically poll the SSM API by using aws ssm get-command-invocation. [7]
Reference
[1] How to check whether my user data passing to EC2 instance working or not?
[2] Working with SSM Agent - AWS Systems Manager
[3] Walkthrough: Use the AWS CLI with Run Command - AWS Systems Manager
[4] Understanding Command Statuses - AWS Systems Manager
[5] Streaming AWS Systems Manager Run Command output to Amazon CloudWatch Logs | AWS Management Tools Blog
[6] how to view aws log real time (like tail -f)
[7] get-command-invocation — AWS CLI 1.16.200 Command Reference
Approach 1.
Start an instance using AWS CLI.
aws ec2 start-instances --instance-ids i-1234567890abcdef0
Set few environment variables.
Use user dat of ec2 to set env. & run commands
..
Run your other logic / scripts
To terminate the instance run below command in the same instance.
instanceid=`curl http://169.254.169.254/latest/meta-data/instance-id`
aws ec2 terminate-instances --instance-ids $instanceid
Approach 2.
Use python boto3 or kitchen chef ci.
I am trying to use Amazon's ECS cli to create a cluster. I keep getting the error:
reason="The key pair 'my-key-pair' does not exist" resourceType="AWS::AutoScaling::LaunchConfiguration"
I have also run:
ecs-cli configure profile --profile-name grantspilsbury --access-key foo --secret-key bar
ecs-cli configure --cluster cluster_test --region us-east-1 --config-name myclusterconfig
I have added my-key-pair to ECS and to EC2.
The full log is:
~ $ ecs-cli up --keypair my-key-pair --capability-iam --size 2 --instance-type t2.small --force
INFO[0002] Created cluster cluster=default region=us-east-1
INFO[0003] Waiting for your CloudFormation stack resources to be deleted...
INFO[0003] Cloudformation stack status stackStatus="DELETE_IN_PROGRESS"
INFO[0038] Waiting for your cluster resources to be created...
INFO[0038] Cloudformation stack status stackStatus="CREATE_IN_PROGRESS"
INFO[0101] Cloudformation stack status stackStatus="CREATE_IN_PROGRESS"
INFO[0164] Cloudformation stack status stackStatus="CREATE_IN_PROGRESS"
ERRO[0197] Failure event reason="The key pair 'my-key-pair' does not exist" resourceType="AWS::AutoScaling::LaunchConfiguration"
FATA[0197] Error executing 'up': Cloudformation failure waiting for 'CREATE_COMPLETE'. State is 'ROLLBACK_IN_PROGRESS'
I ran into the same issue. My issue was I was giving it the full path to the pem file rather than the name of the key on EC2 (link to Ohio region). Turning
ecs-cli up --keypair /home/me/keyPair.pem --capability-iam --size 2 --instance-type t2.medium --cluster-config ec2-tutorial --force
into
ecs-cli up --keypair keyPair --capability-iam --size 2 --instance-type t2.medium --cluster-config ec2-tutorial --force
works as long as there is a key pair on EC2 named keyPair
My problem was that I was passing the filename (keypair.pem) instead of the name of the keypair on AWS. Make sure you pass the keypair name as you see on AWS and not the name of the file.
It's possible that your key is in a different region from the one where you're attempting to create the image. Jeff's answer gave me the clue that my keypair was in the default instance (Ohio), but I was creating my instance in my local region.
I was following this tutorial until I found the same issue:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-ec2.html
It had nothing to do with the region, the problem was my Key-Pair wasn't uploaded to the ECS. Maybe it is obvious, but if you are AWS beginner like me, the tutorial instructions are not clear about this.
Previous step: Create Key-Pair
If you haven't already done it, first you have to create the Key-Pair that is stored on a ".pem" file. Following this instructions with the AWS Console, was quite easy: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html
Solution:
The ".pem" file has to be uploaded. Can be done using the AWS console, following the menu:
EC2 > Network & Security > Key Pairs > "Create Key Pair"
Menu URL: https://eu-central-1.console.aws.amazon.com/ec2/v2/home?region=eu-central-1#CreateKeyPair:
(Be careful about the 2 regions defined on this URL)
As it is described on:
https://www.cloudbooklet.com/how-to-add-new-user-with-key-pair-in-aws-ec2/
That means that keypair name does not exist in your EC2 account.
Simply create it with s per the aws guide
i have a shell script which need to be installed over 100 Ubuntu instances/servers.What is the best way to install the same script on all instance without logging into each one.
You can use AWS System Manager , according to AWS Documentation :
You can send commands to tens, hundreds, or thousands of instances by
using the targets parameter (the Select Targets by Specifying a
Tag option in the Amazon EC2 console). The targets parameter accepts
a Key,Value combination based on Amazon EC2 tags that you specified
for your instances. When you execute the command, the system locates
and attempts to run the command on all instances that match the
specified tags
You can Target Instance by tag :
aws ssm send-command --document-name name --targets Key=tag:tag_name,Values=tag_value [...]
or
Targeting Instance IDs:
aws ssm send-command --document-name name --targets Key=instanceids,Values=ID1,ID2,ID3 [...]
Read the AWS Documentation for Details.
Thanks
You have several different options when trying to accomplish this task.
Like Kush mentioned, AWS System manager is great, but is a tightly coupled AWS service.
Packer - You could use Packer to create an AMI of the servers, and have the script installed on them, or just executed whatever the script is doing.
Configuration Management.
Ansible/Puppet/Chef. - These tools allow you to manage thousands of servers with only a couple of commands. My preference would be for Ansible, it is light weight, the syntax is only yaml, connects over ssh, and still allows use of placing shell scripts, if need be.
Can anyone please tell me how to run a shell script on all the ec2 instances that are part of an auto scaling group?
The scenario is that I have a script that I want to run on many ec2 instances that are turned on automatically as part of auto scaling group. The native approach is to SSH to each instance and run the script. I am looking for a way by which it can run automatically on all the instances when I run it on one of the ec2 instance or any better way of doing this?
Thanks in advance.
You'll want to add that shell script to the userdata in a NEW Launch Config and then update the autoscaling group.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts
Updating the Launch Config
if you want to change the launch configuration for your Auto Scaling group, you must create a launch configuration and then update your Auto Scaling group with the new launch configuration. When you change the launch configuration for your Auto Scaling group, any new instances are launched using the new configuration parameters, but existing instances are not affected.
https://docs.aws.amazon.com/autoscaling/latest/userguide/LaunchConfiguration.html
You can implement in many different ways...
Use awscli to get all instances in auto scaling group
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name myTestGroup
SSHkit is interesting tool run commands on remote servers
or
Invest your time to automate your infrastructure in proper tools like puppet, chef. With puppet mcollective you can do magic, including what you have asked about.
Update:
When you add instance to autoscaling group new tag name=aws:autoscaling:groupName, value=name_of_assigned_autoscaling_group is added, thus its easy to find it searching for this tag.
$ asgName=testASG
$ aws ec2 describe-instances --filters Name=tag-key,Values='aws:autoscaling:groupName' Name=tag-value,Values=$asgName --output text --query 'Reservations[*].Instances[*].[InstanceId,PublicIpAddress]'
The output you will get from the command above is the instance name and public IP:
i-4c42aabc 52.1.x.y
You can use this is your script...
I would do it using Chef. Opsworks (an aws product) is chef plus a lot of things that will do exactly what you want and give you even more flexibility.
Run Command perhaps?
You can use it to invoke scripts as well:
*Nice cheat: Select Run Command on the left-side menu of the EC2 section. Click the "Run Command" button. Then setup your AWS-RunShellScript. Type in your code. Then at the bottom there is a dropdown labelled: "AWS Command Line Interface command", select the correct platform and copy/paste the command into a script.
$Command_Id = aws ssm send-command --document-name "AWS-RunPowerShellScript" --targets '{\"Key\":\"tag:Name\",\"Values\":[\"RunningTests\"]}' --parameters '{\"commands\":[\"Start-Process \\\"C:\\\\path\\\\to\\\\scripts\\\\LOCAL_RUN.ps1\\\"\"]}' --comment "Run Tests Locally" --timeout-seconds 3800 --region us-east-1 --query 'Command.CommandId' --output text | Out-String
Per the question: Use your Auto Scaling Group Name instead of "RunningTests." Or, in the console: In your "Run Command" setup. Select the "Specifiying a Tag" radio button, then "Name" and your Auto Scaling Group.
*Note: The command above is windows powershell, but you can convert your script to Linux/OS or whatever by selecting the correct platform when in the Run Command setup.
**Note: Ensure your User on that instance has the AmazonSSMFullAccess permission setup to run the commands.
***Note:The SSM Agent comes installed on Windows Instances by default. If you are running Linux or something else, you might need to install the SSM Agent.
For a simplistic implementation you can use python fabric (http://www.fabfile.org/). It is a tool to run commands from a local or bashin instance to list of servers.
Here is a repo which has basic scaffolding and examples. Lot of CICD tools are having features to target this requirements, but I found fabric the easiest to implement fo simple setup.
https://github.com/techsemicolon/python-fabric-ec2-aws