How to run shell script on multiple EC2 instances using AWS CLI? - shell

I'm trying to do a simple AWS CLI command that can run a shell command to multiple instances.
I know first I need to get the list of instances ids:
aws ec2 describe-instances --filter "Name=tag:Group,Values=Development" --query 'Reservations[].Instances[].[InstanceId]' --output text
I then will have to assign them to an array. Then loop through each instance id and send the command.
Do we have an option for aws to send a shell command to an instance with a specific id?
Something like this:
aws ssm send-command --instance-ids "i-xxxxxxxxxxxxxxxx" --document-name "shellscript"
I keep getting this error:
An error occurred (InvalidInstanceId) when calling the SendCommand operation:
I've made sure that the SSM agent is running on that specific instance and made sure everything is correct according to these docs pages.

You can use ssm send-command.
A sample command to see ip address of instance:
aws ssm send-command --instance-ids "your id's" --document-name "AWS-RunShellScript" --comment "IP config" --parameters "commands=ifconfig" --output text
Modify command as per your needs.
In case you've got the error, this can happen when you don't have SSM setup on the instance you're trying to access. For a list of instances where you can run SSM commands, run:
aws ssm describe-instance-information --output text
See: InvalidInstanceId: An error occurred (InvalidInstanceId) when calling the SendCommand operation.

I was able to create a script with Python using Boto3.
import boto3
import botocore
import paramiko
tagkey = 'Environment'
tagvalue = 'DEV'
# list_instances functions returns a list of ip addresses containing a set of tags
def list_instances(tagkey, tagvalue):
ec2client = boto3.client('ec2')
response = ec2client.describe_instances(
Filters=[
{
'Name': 'tag:'+tagkey,
'Values': [tagvalue]
}
]
)
instancelist = []
for reservation in (response["Reservations"]):
for instance in reservation["Instances"]:
instancelist.append(instance["PublicDnsName"])
return instancelist
# Results of the function get stored in a list.
list = list_instances(tagkey, tagvalue)
key = paramiko.RSAKey.from_private_key_file("/home/ec2-user/key.pem")
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
# Looping through all the instannces in the list
for instance_ip in list[:]:
# Connect/ssh to an instance
try:
# Here 'ec2-user' is user name and 'instance_ip' is public IP of EC2
client.connect(hostname=instance_ip, username="ec2-user", pkey=key)
# Execute a command after connecting/ssh to an instance
stdin, stdout, stderr = client.exec_command("touch test")
# close the client connection once the job is done
print "Command sent:",instance_ip
except Exception, e:
print e

Related

AWS CLI: How to use variable to filter EFS

I want to use the value of the DOMAIN_ID variable to filter the EFS to get a FileSystemId. I used the commands below. The first command works and it stores the domain ID. The second one returns an empty list, even though the DOMAIN_ID variable is present.
DOMAIN_ID=$(aws sagemaker list-domains --query 'Domains[0].DomainId')
aws efs describe-file-systems --query 'FileSystems[?CreationToken==`$DOMAIN_ID`].FileSystemId'
Output:
[]
Expected output:
<Some EFS identifier>
This works (escaping backticks) -
aws efs describe-file-systems --query "FileSystems[?CreationToken==\`$DOMAIN_ID\`].FileSystemId"
You can also use describe-domain command instead -
$ DOMAIN_ID=$(aws sagemaker list-domains --query 'Domains[0].DomainId' | tr -d '"')
$ aws sagemaker describe-domain --domain-id $DOMAIN_ID --query 'HomeEfsFileSystemId'

Create EC2 instance and get public IP when available

I have a monotonous work of creating an ec2 instance from an AMI and then ssh into it and run a crypto bot to get a sub domain SSL certificates for it and then run some frontend and backend onto it.
I would like to automate this...
I am able to create the EC2 instance
aws ec2 run-instances --image-id ami-0bef9cfcxxxxxx --count 1 --instance-type t2.medium --key-name MyKey --security-group-ids SG-1 --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=SubDomain}]'
but how do I wait till the instance is UP and get the public IP for the next steps?
Any suggestions would help,
Thanks!
Assuming you know the target instance id, you can query your instance and get the current state, instanceId, name (assuming you tagged them with a name) and public IP via something like:
aws ec2 describe-instances \
--instance-ids YOUR_TARGET_INSTANCE_ID \
--query "Reservations[*].Instances[*].[State.Name, InstanceId, Tags[?Key==\`Name\`]|[0].Value, PublicIpAddress]" \
--output text
You could run this query in a loop until the status comes back as running and then capture the public ip address:
while :
do
output=$(aws ec2 describe-instances \
--instance-ids YOUR_TARGET_INSTANCE_ID \
--query "Reservations[*].Instances[*].[State.Name, InstanceId, \
Tags[?Key==\`Name\`]|[0].Value, PublicIpAddress]" \
--output text)
if [[ "${output}" == running* ]] ; then
cut -d $'\t' -f4- <<<"${output}"
break
fi
done
Note:
The aws ec2 queries are all using the default AWS profile. You can specify a different profile via --profile your_target_profile_name if needed.
If you do not know your target instance id, you can remove the --instance-id portion of the aws ec2 query and list all of your instances (in your default region). You could choose the required instance id from that list.

How to list public IPs of all compute instances in OCI?

I need to get the public IPs of all instances of my OCI tenant.
Here i saw a python scripts to do this from OCI Console Cloud Shell : https://medium.com/oracledevs/writing-python-scripts-to-run-from-the-oci-console-cloud-shell-a0be1091384c
But I want to create a bash script, that uses OCI CLI commands to fetch the required data.
How can I achieve this using OCI CLI commands?
OCI CLI structured-search and query feature can be used to fetch the OCID of instances, and instance command can be used fetch the instance details.
The output would be in json format by default.
You can use jq to filter needed data from the output json and create an array with it.
(OCI tool supports JMESPath queries)
Here is the snippet from bash script that uses OCI CLI commands to get public IPs of all compute instances in the compartment :
Pre-requisites: OCI CLI should be installed and configured properly to authenticate with the correct tenant and compartment
# Fetch the OCID of all the running instances in OCI and store to an array
instance_ocids=$(oci search resource structured-search --query-text "QUERY instance resources where lifeCycleState='RUNNING'" --query 'data.items[*].identifier' --raw-output | jq -r '.[]' )
# Iterate through the array to fetch details of each instance one by one
for val in ${instance_ocids[#]} ; do
echo $val
# Get name of the instance
instance_name=$(oci compute instance get --instance-id $val --raw-output --query 'data."display-name"')
echo $instance_name
# Get Public Ip of the instance
public_ip=$(oci compute instance list-vnics --instance-id $val --raw-output --query 'data[0]."public-ip"')
echo $public_ip
done
References :
https://blogs.oracle.com/cloud-infrastructure/post/exploring-the-search-and-query-features-of-oracle-cloud-infrastructure-command-line-interface
https://docs.oracle.com/en-us/iaas/tools/oci-cli/2.9.9/oci_cli_docs/cmdref/compute/instance.html

Error trying to get command status with aws cli

I seem to keep getting an error whenever I try to use bash to automate getting the status of a job.
My current bash script currently looks like this:
#!/bin/bash
aws ec2 start-instances --instance-ids=$1;
start=$(aws ec2 describe-instance-status --instance-id $1)
status=$(echo $start | jq '.InstanceStatuses[0].InstanceState.Name')
#wait for ec2 instance to start running before launching command
while [ "$status" != "\"running\"" ]
do
start=$(aws ec2 describe-instance-status --instance-id $1)
status=$(echo $star | jq '.InstanceStatueses[0].InstanceState.Name')
done
sh_command_id=$(aws ssm send-command --instance-ids=$1 --document-name "AWS-RunShellScript" --parameters 'commands=["echo Helloworld","sleep 60"]');
command_id=$(echo $sh_command_id | jq '.Command.CommandId')
full_status=$(aws ssm list-commands --command-id $command_id)
echo $command_id;
aws ec2 stop-instances --instance-ids=$1;
When the script gets to aws ssm list-commands --command-id $command_id I get this error.
An error occurred (ValidationException) when calling the ListCommands operation: 2
validation errors detected: Value '"67fb9aed-00bf-4741-ae1a-736ddbfba498"' at 'commandId'
failed to satisfy constraint: Member must satisfy regular expression pattern: ^[A-Fa-
f0-9]{8}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{12}$.; Value
'"67fb9aed-00bf-4741-ae1a-736ddbfba498"' at 'commandId' failed to satisfy constraint:
Member must have length less than or equal to 36.
When running everything individually in terminal I get the same error. However, I do not get an error when I mannually type in the commandId as so: full_status=$(aws ssm list-commands --command-id 67fb9aed-00bf-4741-ae1a-736ddbfba498).
Is there some aws formatting I am missing here?
You might be able to avoid the use of jq by using the aws cli built in --query 'your.json.query' to specify your JSON query and then the --output text to return plain text. It has been a while since I checked so your mileage may vary.
I was able to verify that the following works for checking an ec2 is running:
check_instance() {
local instance_id="${1}"
local status="_"
while [ "${status}" != "running" ] ; do
status=$(aws ec2 describe-instance-status \
--instance-ids ${instance_id} \
--query "InstanceStatuses[*].InstanceState.Name" \
--output text)
done
echo "${instance_id} is running"
}

How can I get the the ID of an AWS security group if I know the name?

I'm using the AWS CLI and I want to get the ID of security group whose name I know (kingkajou_sg). How can I do it?
When I ask it to list all the security groups, it does so happily:
$ aws ec2 describe-security-groups | wc -l
430
When I grep through this information, I see that the SG in question is listed:
$ aws ec2 describe-security-groups | grep -i kingkajou_sg
"GroupName": "kingkajou_sg",
However, when I try to get the information about only that security group, it won't let me. Why?
$ aws ec2 describe-security-groups --group-names kingkajou_sg
An error occurred (InvalidGroup.NotFound) when calling the
DescribeSecurityGroups operation: The security group 'kingkajou_sg' does not exist in default VPC 'vpc-XXXXXXXX'
Can someone please provide me the one line command that I can use to extract the Security group's ID given its name? You can assume that the command will be run from within an EC2 which is in the same VPC as the Security group.
From the API Documentation:
--group-names (list)
[EC2-Classic and default VPC only] One or more security group names. You can specify either the security group name or the security group ID. For security groups in a nondefault VPC, use the group-name filter to describe security groups by name.
If you are using a non-default VPC, use the Filter
aws ec2 describe-security-groups --filter Name=vpc-id,Values=<my-vpc-id> Name=group-name,Values=kingkajou_sg --query 'SecurityGroups[*].[GroupId]' --output text
If it's in a VPC and you know the name & region and vpc id, you can try it like below:
aws ec2 describe-security-groups --region eu-west-1 --filter Name=vpc-id,Values=vpc-xxxxx Name=group-name,Values=<your sg name> --query 'SecurityGroups[*].[GroupId]' --output text
You just need to add --query 'SecurityGroups[*].[GroupId]' option with aws cli command.
aws ec2 describe-security-groups --group-names kingkajou_sg --query 'SecurityGroups[*].[GroupId]' --output text
To get the IDs of all security groups with a name matching exactly a specified string (default in this example) without specifying a VPC ID, use the following:
aws ec2 describe-security-groups --filter Name=group-name,Values=default --output json | jq -r .SecurityGroups[].GroupId
Note: this works for security groups even if they are not in the default VPC.
Small shell script to list security with search string as a variable. and we can tag the security groups.
https://ideasofpraveen.blogspot.com/2022/09/aws-cli-get-security-group-id-with-name.html.
If you want boto3 script to integrate with lambda for automations .
https://ideasofpraveen.blogspot.com/2022/09/aws-cli-get-security-group-id-with-name_15.html

Resources