JMESPath to create a dictionary (map) out of Tags key and value pairs from AWS CLI output - amazon-ec2

I am trying to extract a few properties of AWS EC2 instance using JMESPath. I just started learning JMESPath using this useful guide. But because my knowledge is limited, I was only able to get the output like below.
% aws ec2 describe-instances --query "Reservations[].Instances[].[InstanceId,InstanceType,KeyName,LaunchTime,State.Name,Tags[].[Key,Value]]" --instance-ids i-123456789101123
[
[
"i-123456789101123",
"r5.8xlarge",
"My_EMR",
"2022-12-19T16:11:42.000Z",
"running",
[
[
"tag-key-1",
"tag-value-1"
],
[
"tag-key-2",
"tag-value-2"
],
[
"tag-key-3",
"tag-value-3"
],
[
"project",
"my-project"
]
]
]
]
Ideally, I would like to format the output using JMESPath like below.
[
[
"i-123456789101123",
"r5.8xlarge",
"My_EMR",
"2022-12-19T16:11:42.000Z",
"running",
{
"tag-key-1": "tag-value-1",
"tag-key-2": "tag-value-2",
"tag-key-3": "tag-value-3",
"project": "my-project"
}
]
]
I have tried a bunch of different JMESPath expressions like below, but none of them seem to work. I'd greatly appreciate any suggestion. Thank you in advance!
# JMESPath statements that I've tried so far (actually there are more, but these are the 'promising' ones that I tried, but failed to work)
% aws ec2 describe-instances --query "Reservations[].Instances[].[InstanceId,InstanceType,KeyName,LaunchTime,State.Name,map(Tags[].Key,Tags[].Value)]"
% aws ec2 describe-instances --query "Reservations[].Instances[].[InstanceId,InstanceType,KeyName,LaunchTime,State.Name,map(Tags[].Key,Tags[].Value)]"
% aws ec2 describe-instances --query "Reservations[].Instances[].{InstanceId: InstanceId, InstanceType: InstanceType, KeyName: KeyName, LaunchTime: LaunchTime, StateName: State.Name, Tags[].Key: Tags[].Value}"
% aws ec2 describe-instances --query "Reservations[].Instances[].{InstanceId: InstanceId, InstanceType: InstanceType, KeyName: KeyName, LaunchTime: LaunchTime, StateName: State.Name, Tags[*].Key: Tags[*].Value}"

Related

aws cli to loop through rds instances

if I run
aws rds describe-db-instances --query 'DBInstances[*].[DBInstanceIdentifier,AutoMinorVersionUpgrade]' --output text
It gives 1000's of IDs and true or false for AutoMinorVersionUpgrade:
db-test-admin-1-dev True
db-test-admin-product1-dev True
db-test-admin-legacy-dev True
db-byodb-test False
db-byod-nightly True
db-debug-oc-issue-dev True
db-test1-dev True
db-test-instance-dev True
db-exa-dev False
I want to set AutoMinorVersionUpgrade to false to all, so I do:
for rds in $(aws rds describe-db-instances --query 'DBInstances[*].DBInstanceIdentifier' --output text);
do aws rds modify-db-instance --db-instance-identifier $rds --no-auto-minor-version-upgrade ;
done
How can I skip running modify-db-instance if the rds instance already has AutoMinorVersionUpgrade=False?
You can use --query to conditionally return some items. The --query option support jmespath expressions.
for rds in $(aws rds describe-db-instances --query 'DBInstances[?AutoMinorVersionUpgrade==`true`].DBInstanceIdentifier' --output text); do
aws rds modify-db-instance --db-instance-identifier $rds --no-auto-minor-version-upgrade
done

How to sort JSON output from AWS CLI using JMESPath

I try to sort this output from AWS CLI by ImageId, and I executed command below.
aws ec2 describe-images --profile xxxxxxxxxx \
--filter Name=tag:Name,Values=Backup*some-string* \
--query "Images[*].[Tags[?Key=='Name'].Value[]|[0],ImageId]"
output is:
[
[
"Backup-20191215T174530Z-utc-some-string",
"ami-004"
],
[
"Backup-20191219T174631Z-utc-some-string",
"ami-002"
],
[
"Backup-20191208T174534Z-utc-some-string",
"ami-001"
],
[
"Backup-20191222T174530Z-utc-some-string",
"ami-003"
],
[
"Backup-20191221T174530Z-utc-some-string",
"ami-005"
]
]
I found sort_by functions of JMESPath could be a solution but that is too hard to understand.
aws ec2 describe-images --profile xxxxxxxxxx \
--filter "Name=tag:Name,Values=Backup*some-string*" \
--query "sort_by(Images[*].[Tags[?Key=='Name'].Value[]|[0],ImageId], &[0])"
or
aws ec2 describe-images --profile xxxxxxxxxx \
--filter "Name=tag:Name,Values=Backup*some-string*" \
--query "Images[*].[Tags[?Key=='Name'].Value[]|[0],ImageId] | sort_by(#, &[0])"
is working fine for me. & (expression type operator) is needed.
The idea is, In my solution below, I am sorting the output first by the ImageId and then applying the projections.
aws ec2 describe-images --filter Name=tag:Environment,Values=Staging --output json --query "(sort_by(Images[], &ImageId))[*].[ImageId, Tags[?Key=='Environment'].Value | [0]]"

Getting value out from square brackets in "aws ec2 describe-instances" output

I am saving output of my command in a variable and using it in other command
instance_id=$(aws ec2 describe-instances --query Reservations[*].Instances[*].[InstanceId] --filters "Name=tag:staging,Values=staging")
I get this output [ [ [ "i-09140824c1b7f9ea7" ] ] ]
How to I remove brackets from the output and use it in the variable in this command
aws ec2 associate-address --instance-id $instance_id --allocation-id allocid
I am new to bash so any help would be appreciated.
You can extract the value and decode it from JSON to plain text with the following:
instance_id_json='[ [ [ "i-09140824c1b7f9ea7" ] ] ]'
instance_id=$(jq -r '.[][][]' <<<"$instance_id_json")
Consider changing your original code to:
instance_id_json=$(aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters 'Name=tag:staging,Values=staging')
...note that we're putting the query in quotes (since it has glob characters), not just the filter (and since the filter is intended to be literal text with no expansions, we're defaulting to single-quotes rather than double as a matter of form).

Shell command to return value in json output

How to return a particular value using shell command?
In the following example I would like to query to return the value of "StackStatus" which is "CREATE_COMPLETE"
Here is the command:
aws cloudformation describe-stacks --stack-name stackname
Here is the output:
{
"Stacks": [{
"StackId": "arn:aws:cloudformation:ap-southeast-2:64560756805470:stack/stackname/8c8e3330-9f35-1er6-902e-50fae94f3fs42",
"Description": "Creates base IAM roles and policies for platform management",
"Parameters": [{
"ParameterValue": "64560756805470",
"ParameterKey": "PlatformManagementAccount"
}],
"Tags": [],
"CreationTime": "2016-10-31T06:45:02.305Z",
"Capabilities": [
"CAPABILITY_IAM"
],
"StackName": "stackname",
"NotificationARNs": [],
"StackStatus": "CREATE_COMPLETE",
"DisableRollback": false
}]
}
The aws cli supports the --query option to get parts. In addition you could also pipe to another command line tool, jq to do similar query.
But in aws notation to get the 1st result:
aws cloudformation describe-stacks --stack-name stackname --query 'Stacks[0].StackStatus' --output text
Based on the above output, Stacks is an array of objects (a key/value), so hence need the [0] to get the 1st element of the array, and then .StackStatus is a key in this object containing a string as value. The --output text simply presents the output as simple text value instead of a json-looking object.
Edited per Charles comment.

Method of finding instances attached to ELB

I am using the Amazon AWS ELB command line tools. Is there a way of finding out the instances attached to a particular Elastic Load Balancer (ELB)?
2013/12/18: To update this and since the links are dead!
I installed the new AWS cli tools:
$ pip install awscli
Then ran:
$ aws configure
AWS Access Key ID [None]: my-key
AWS Secret Access Key [None]: my-secret
Default region name [None]: us-east-1
Default output format [None]:
This data is saved into ~/.aws/config.
Then I can find instances connected to a loadbalancer like so:
$ aws elb describe-load-balancers --load-balancer-name "my-name"
{
"LoadBalancerDescriptions": [
{
"Subnets": [],
"CanonicalHostedZoneNameID": "ID",
"CanonicalHostedZoneName": "my-name-foo.us-east-1.elb.amazonaws.com",
"ListenerDescriptions": [
{
"Listener": {
"InstancePort": 80,
"LoadBalancerPort": 80,
"Protocol": "HTTP",
"InstanceProtocol": "HTTP"
},
"PolicyNames": []
},
{
"Listener": {
"InstancePort": 80,
"SSLCertificateId": "arn:aws:iam::x:server-certificate/x-ssl-prod",
"LoadBalancerPort": 443,
"Protocol": "HTTPS",
"InstanceProtocol": "HTTP"
},
"PolicyNames": [
"AWSConsole-SSLNegotiationPolicy-api-production"
]
}
],
"HealthCheck": {
"HealthyThreshold": 10,
"Interval": 30,
"Target": "HTTP:80/healthy.php",
"Timeout": 5,
"UnhealthyThreshold": 2
},
"BackendServerDescriptions": [],
"Instances": [
{
"InstanceId": "i-FIRST-INSTANCEID"
},
{
"InstanceId": "i-SECOND-INSTANCEID"
}
],
"DNSName": "my-name-foo.us-east-1.elb.amazonaws.com",
"SecurityGroups": [],
"Policies": {
"LBCookieStickinessPolicies": [],
"AppCookieStickinessPolicies": [],
"OtherPolicies": [
"AWSConsole-SSLNegotiationPolicy-my-name"
]
},
"LoadBalancerName": "my-name",
"CreatedTime": "2013-08-05T16:55:22.630Z",
"AvailabilityZones": [
"us-east-1d"
],
"Scheme": "internet-facing",
"SourceSecurityGroup": {
"OwnerAlias": "amazon-elb",
"GroupName": "amazon-elb-sg"
}
}
]
}
The data is in LoadBalancerDescriptions.Instances.
My loadbalancer is called my-name — this is the name you selected when you created it.
Old answer below!
I'm not familiar with the cli tool, but I used the API.
I'd check these two requests:
DescribeLoadBalancers
DescribeInstanceHealth
The cli tool probably has something to resemble these?
HTH!
Assuming you have aws-cli and jq installed, you can use the following command to get associated ec2 instance ids:
aws elb describe-load-balancers --load-balancer-name my-elb \
| jq -r '.LoadBalancerDescriptions[].Instances[].InstanceId'
This will return the ec2 ids associated with that ELB.
Side note: I recommend you setup aws cli profiles so you don't have to fiddle with environment variables and region params (as much).
Because I love answers that can be used with a minimum of search/replace and copy paste
Prerequisites : aws-cli configured
pip install awscli
aws configure
Configure : your ELB name
$ELB_NAME = "Your-elb-name"
Copy-n-Paste in terminal
for ID in $(aws elb describe-load-balancers --load-balancer-name $ELB_NAME \
--query LoadBalancerDescriptions[].Instances[].InstanceId \
--output=text);
do
aws ec2 describe-instances --instance-ids $ID \
--query Reservations[].Instances[].PublicIpAddress \
--output text
done
Will output a list of Public IPs. You could also just execute the query inside the parenthesis of the for ID in $(...) to just get the instance IDs
Want something different ?
Feel free to have a look at the structure of
aws elb describe-load-balancers --load-balancer-name $ELB_NAME
aws ec2 describe-instances --instance-ids $INSTANCE_ID
and change the query accordingly!
If anyone arrives here from a search as to why the elb-describe-lbs command returns nothing when they have ELBs up and running, what I realised was I needed to add EC2_REGION=eu-west-1 to my environment variables (or use elb-describe-lbs --region command)
If you want to see all your ELB's and the instances attached use JMESPath like this:
aws elb describe-load-balancers --query "LoadBalancerDescriptions[*].{ID:LoadBalancerName,InstanceId:Instances[*].InstanceId}[*]. {ELB:ID,InstanceId:InstanceId[*]}" --output=json
Result
[
{
"ELB": "my_name",
"InstanceId": [
"i-0cc72"
]
},
{
"ELB": "my_name2",
"InstanceId": [
"i-02ff5f",
"i-09e467"
]
}
]
If you know the name of the ELB and want to see what is attached use JMESPath like this:
aws elb describe-load-balancers --load-balancer-name "my_name" --query "LoadBalancerDescriptions[].{ID:LoadBalancerName,InstanceId:Instances[].InstanceId}[].{ELB:ID,InstanceId:InstanceId[]}" --output=json
Result:
[
{
"ELB": "my_name",
"InstanceId": [
"i-02ff5f72",
"i-09e46743"
]
}
]
replace INSTANCEID with actual instance id
aws elb describe-load-balancers --query "LoadBalancerDescriptions[*].{ID:LoadBalancerName,InstanceId:Instances[?InstanceId=='INSTANCEID'].InstanceId}[*].{ID:ID,InstanceId:InstanceId[0]}" --output=text | grep INSTANCEID | awk '{print $1}'
In node.js you can do this by using aws-sdk.
var AWS = require('aws-sdk')
var options = {
accessKeyId: 'accessKeyId',
secretAccessKey: 'secretAccessKey',
region: 'region'
}
var elb = new AWS.ELB(options)
elb.describeLoadBalancers({LoadBalancerNames: ['elbName']}, function(err, data) {
if (err) {
console.log('err: ', err)
}
else {
console.log('data: ', data.LoadBalancerDescriptions)
}
})
data.LoadBalancerDescriptions is an array and each element in the array is an object with the property Instances that has the instance id.
You can loop trough all you load balancer instance ids as follows:
while read -r lb ; do echo -e "\n\n start lb: $lb " ; \
echo run cmd on lb: $lb ; echo " stop lb: $lb" ; \
done < <(aws elb describe-load-balancers --query \
'LoadBalancerDescriptions[].Instances[].InstanceId' \
--profile dev|perl -nle 's/\s+/\n/g;print')
You can loop trough your load balancers names as follows :
# how-to loop trough all your load balancer names
while read -r lb ; do \
echo -e "\n\n start lb: $lb " ; \
echo run cmd on lb: $lb ; \
echo " stop lb: $lb" ; \
done < <(aws elb describe-load-balancers --query \
'LoadBalancerDescriptions[].LoadBalancerName' \
--profile rnd|perl -nle 's/\s+/\n/g;print')
Provided that you have configured your aws cli :
src: http://docs.aws.amazon.com/cli/latest/topic/config-vars.html
cat << "EOF" > ~/.aws/config
[profile dev]
output = text
region = us-east-1
[profile dev]
output = text
region = us-east-1
[default]
output = text
region = Global
EOF
And configured your security credentials:
# in aws admin console :
# Services => iam => users => <<your_username>> => Security Credentials => Access Keys
# configure the aws cli
cat << "EOF" > ~/.aws/credentials
[dev]
aws_access_key_id = <<your_aws_access_key_id_in_the_dev_environment>>
aws_secret_access_key = <<your_aws_secret_access_key_in_dev_env>>
[dev]
aws_access_key_id = <<your_aws_access_key_id_in_the_dev_environment>>
aws_secret_access_key = <<your_aws_secret_access_key_in_dev_env>>
[default]
aws_access_key_id = <<your_aws_access_key_id_in_the_dev_environment>>
aws_secret_access_key = <<your_aws_secret_access_key_in_dev_env>>
EOF
aws elb describe-load-balancers --load-balancer-name "LB_NAME" | grep "InstanceId" | awk '{print $2}' | sed 's/\"//g'
First do elb-describe-lbs to get a list of your load balancers and their names.
Then do elb-describe-instance-health <LB_NAME> to get a list of instances behind that load balancer. LB_NAME is the value of the 2nd column in the output of elb-describe-lbs.
You can use AWS command line tools with some bash piping:
elb-describe-instance-health loadbalancer_name --region eu-west-1 | awk '{ print $2 }' | xargs ec2-describe-instances --region eu-west-1 | grep ^INSTANCE | awk '{ print $4 }'
This will give you the public DNS name for every instance attached to the ELB, you can change the awk columns respectively to get other details.

Resources