I try to sort this output from AWS CLI by ImageId, and I executed command below.
aws ec2 describe-images --profile xxxxxxxxxx \
--filter Name=tag:Name,Values=Backup*some-string* \
--query "Images[*].[Tags[?Key=='Name'].Value[]|[0],ImageId]"
output is:
[
[
"Backup-20191215T174530Z-utc-some-string",
"ami-004"
],
[
"Backup-20191219T174631Z-utc-some-string",
"ami-002"
],
[
"Backup-20191208T174534Z-utc-some-string",
"ami-001"
],
[
"Backup-20191222T174530Z-utc-some-string",
"ami-003"
],
[
"Backup-20191221T174530Z-utc-some-string",
"ami-005"
]
]
I found sort_by functions of JMESPath could be a solution but that is too hard to understand.
aws ec2 describe-images --profile xxxxxxxxxx \
--filter "Name=tag:Name,Values=Backup*some-string*" \
--query "sort_by(Images[*].[Tags[?Key=='Name'].Value[]|[0],ImageId], &[0])"
or
aws ec2 describe-images --profile xxxxxxxxxx \
--filter "Name=tag:Name,Values=Backup*some-string*" \
--query "Images[*].[Tags[?Key=='Name'].Value[]|[0],ImageId] | sort_by(#, &[0])"
is working fine for me. & (expression type operator) is needed.
The idea is, In my solution below, I am sorting the output first by the ImageId and then applying the projections.
aws ec2 describe-images --filter Name=tag:Environment,Values=Staging --output json --query "(sort_by(Images[], &ImageId))[*].[ImageId, Tags[?Key=='Environment'].Value | [0]]"
Related
I am trying to extract a few properties of AWS EC2 instance using JMESPath. I just started learning JMESPath using this useful guide. But because my knowledge is limited, I was only able to get the output like below.
% aws ec2 describe-instances --query "Reservations[].Instances[].[InstanceId,InstanceType,KeyName,LaunchTime,State.Name,Tags[].[Key,Value]]" --instance-ids i-123456789101123
[
[
"i-123456789101123",
"r5.8xlarge",
"My_EMR",
"2022-12-19T16:11:42.000Z",
"running",
[
[
"tag-key-1",
"tag-value-1"
],
[
"tag-key-2",
"tag-value-2"
],
[
"tag-key-3",
"tag-value-3"
],
[
"project",
"my-project"
]
]
]
]
Ideally, I would like to format the output using JMESPath like below.
[
[
"i-123456789101123",
"r5.8xlarge",
"My_EMR",
"2022-12-19T16:11:42.000Z",
"running",
{
"tag-key-1": "tag-value-1",
"tag-key-2": "tag-value-2",
"tag-key-3": "tag-value-3",
"project": "my-project"
}
]
]
I have tried a bunch of different JMESPath expressions like below, but none of them seem to work. I'd greatly appreciate any suggestion. Thank you in advance!
# JMESPath statements that I've tried so far (actually there are more, but these are the 'promising' ones that I tried, but failed to work)
% aws ec2 describe-instances --query "Reservations[].Instances[].[InstanceId,InstanceType,KeyName,LaunchTime,State.Name,map(Tags[].Key,Tags[].Value)]"
% aws ec2 describe-instances --query "Reservations[].Instances[].[InstanceId,InstanceType,KeyName,LaunchTime,State.Name,map(Tags[].Key,Tags[].Value)]"
% aws ec2 describe-instances --query "Reservations[].Instances[].{InstanceId: InstanceId, InstanceType: InstanceType, KeyName: KeyName, LaunchTime: LaunchTime, StateName: State.Name, Tags[].Key: Tags[].Value}"
% aws ec2 describe-instances --query "Reservations[].Instances[].{InstanceId: InstanceId, InstanceType: InstanceType, KeyName: KeyName, LaunchTime: LaunchTime, StateName: State.Name, Tags[*].Key: Tags[*].Value}"
Below are the two different AWS CLI commands, running on bash:
IMAGES_TO_DELETE=$( aws ecr list-images --region $ECR_REGION --repository-name $ECR_REPO --filter "tagStatus=UNTAGGED" --query 'imageIds[*]' --output json )
aws ecr batch-delete-image --region $ECR_REGION --repository-name $ECR_REPO --image-ids "$IMAGES_TO_DELETE" || true
where first command stores JSON format [ {"imageDigest": "sha256:..."}, {"imageDigest": "sha256:..."}, ... ] in IMAGES_TO_DELETE
In a scenario, where I can run single nested command:
aws ecr batch-delete-image --region us-west-2 --repository-name "somedockerimage" --image-ids “$(aws ecr list-images --region us-west-2 --repository-name "somedockerimage" --filter "tagStatus=UNTAGGED" --query 'imageIds[*]' --output text)” || true
gives error:
Error parsing parameter '--image-ids': Expected: '=', received: '“' for input:
“sha256:cccccccccc983f4185f48fb968634dae8c4276ee1a7ffffffffffffff
^
aws ecr batch-delete-image --region us-west-2 --repository-name "somedockerimage" --image-ids “$(aws ecr list-images --region us-west-2 --repository-name "somedockerimage" --filter "tagStatus=UNTAGGED" --query 'imageIds[*]' --output json)” || true
gives error:
Error parsing parameter '--image-ids': Expected: '=', received: '“' for input:
“[
^
What is the nested syntax to replace "$IMAGES_TO_DELETE" in the second command?
Change the smart quotes “...” to straight quotes "...".
I am trying to get spaces into the tags parameter for the aws cli and it works if I hardcode it but not if I use bash variables. What is going on and how do I fix it?
This works with out spaces:
aws cloudformation deploy \
--template-file /path_to_template/template.json \
--stack-name my-new-stack \
--tags Key1=Value1 Key2=Value2
This works with out spaces but with variables:
tags="Key1=Value1 Key2=Value2"
aws cloudformation deploy \
--template-file /path_to_template/template.json \
--stack-name my-new-stack \
--tags $tags
This works with spaces:
aws cloudformation deploy \
--template-file /path_to_template/template.json \
--stack-name my-new-stack \
--tags 'Key1=Value1' 'Key Two=Value2'
This does not work, spaces and variables:
tags="'Key1=Value1' 'Key Two=Value2'"
aws cloudformation deploy \
--template-file /path_to_template/template.json \
--stack-name my-new-stack \
--tags $tags
Attempting to fix bash expansion, also does not work, spaces and variables:
tags="'Key1=Value1' 'Key Two=Value2'"
aws cloudformation deploy \
--template-file /path_to_template/template.json \
--stack-name my-new-stack \
--tags "$tags"
Attempting to fix bash expansion, also does not work, spaces and variables:
tags="'Key1=Value1' 'Key Two=Value2'"
aws cloudformation deploy \
--template-file /path_to_template/template.json \
--stack-name my-new-stack \
--tags "$(printf '%q' $tags)"
Error:
Invalid parameter: Tags Reason: The given tag(s) contain invalid
characters (Service: AmazonSNS; Status Code: 400; Error Code:
InvalidParameter; Request ID
Would you please try:
tags=('Key1=Value1' 'Key Two=Value2')
aws cloudformation deploy \
--template-file /path_to_template/template.json \
--stack-name my-new-stack \
--tags "${tags[#]}"
Stealing some ideas from https://github.com/aws/aws-cli/issues/3274 I was able to get this working by doing the following
deploy=(aws cloudformation deploy
...
--tags $(cat tags.json | jq '.[] | (.Key + "=" + .Value)'))
eval $(echo ${deploy[#]})
With a tags.json file structure of
[
{
"Key": "Name With Spaces",
"Value": "Value With Spaces"
},
{
"Key": "Foo",
"Value": "Bar"
}
]
Try this :
tags="'Key1=Value1' 'Key Two=Value2'"
aws cloudformation deploy \
--template-file /path_to_template/template.json \
--stack-name my-new-stack \
--tags "$tags"
# ^ ^
# double quotes
Learn how to quote properly in shell, it's very important :
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
As of 2022-02 this was still an issue described
here
here also
and a little here
#esolomon is correct you have to array expansion. His answer which works just fine:
deploy=(aws cloudformation deploy
...
--tags $(cat tags.json | jq '.[] | (.Key + "=" + .Value)'))
eval $(echo ${deploy[#]})
The actual problem is a result of the shell environment (bin/bash here) that is used in combination with how python cli executable's handling of values. Since the aws cloudformation deploy does not standardize the input but expects the shell program to standardize array input this was causing my problem.
So my errors with the --debug flag turned on produced the first response which is the error and the second response is the expected input into aws cloudformation deploy
Error input:
2022-02-10 17:32:28,137 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['cloudformation', 'deploy', '--region', 'us-east-1', ..., '--parameter-overrides', 'PARAM1=dev PARAM2=blah', '--tags', "TAG1='Test Project' TAG2='blah'...", '--debug']
Expected input:
2022-02-10 17:39:40,390 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['cloudformation', 'deploy', '--region', 'us-east-1', ..., '--parameter-overrides', 'PARAM1=dev', 'PARAM2=blah', '--tags', "TAG2='Test Project'", 'TAG2=blah',..., '--debug']
I was unexpectedly sending in a string instead of array of strings this error resulted in several errors depending on how I sent it:
example TAG: TAG1=Test Project
['Project'] value passed to --tags must be of format Key=Value
the means IFS needs to be set to something other than the default ' \t\n', solution below
An error occurred (ValidationError) when calling the CreateChangeSet operation: 1 validation error detected: Value 'Test Project Tag2=Value2 ...' at 'tags.1.member.value' failed to satisfy constraint: Member must have length less than or equal to 256
the error starts after the first = this error means that I am sending in one long string instead of array items, as seen here when doing [*] instead of [#] aws cloudformation deploy ... --tags "${TAGS[*]}" diff between [*] and [#]
To fix this the most important thing was that IFS needed to be set to anything other than ' \t\n' and secondly I still need to do array expansion with [#] and could not input a string. The --parameter-overrides for me did not have this problem even though similar variable loading BECAUSE it did not have a string.
This was my solution, my params and tags input is all over the place, spaces + sometimes arrays + bad indenting thus the sed:
export IFS=$'\n'
# Build up the parameters and Tags
PARAMS=($(jq '.[] | .ParameterKey + "=" + if .ParameterValue|type=="array" then .ParameterValue | join(",") else .ParameterValue end ' parameters-${environment}.json \
| sed -e 's/"//g' \
| sed -e $'s/\r//g' | tr '\n' ' '))
TAGS=("$(jq -r '.[] | [.Key, .Value] | "\(.[0])=\(.[1])"' tags-common.json)")
TAGS=($TAGS "$(jq -r '.[] | [.Key, .Value] | "\(.[0])=\(.[1])"' tags-${environment}.json)")
aws cloudformation deploy \
--region "${REGION}" \
--no-fail-on-empty-changeset \
--template-file "stack-name-cfn-transform.yaml" \
--stack-name "stack-name-${environment}" \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides "${params[#]}" \
--tags "${TAGS[#]}" \
--profile "${PROFILE}"
parameters file
[
{
"ParameterKey": "Environment",
"ParameterValue": "dev"
}
]
tags file - both common and environment specific tag files have same format
[
{
"Key": "TAG1",
"Value": "Test Project"
},
{
"Key": "Name With Spaces",
"Value": "Value With Spaces"
},
{
"Key": "Foo",
"Value": "Bar"
}
]
I resolved this scenario using options below:
"scripts": { "invoke": "sam ... --parameter-overrides \"$(jq -j 'to_entries[] | \"\\(.key)='\\\\\\\"'\\(.value)'\\\\\\\"''\\ '\"' params.json)\"" }
Or
sam ... --parameter-overrides "$(jq -j 'to_entries[] | "\(.key)='\\\"'\(.value)'\\\"''\ '"' params.json)"
I am trying to automate some part of my work by creating a bash function that let's me easily ssm into one of our instances. To do that, I only need to know the instance id. Then I run aws ssm start-session with the proper profile. Here's the function:
function ssm_to_cluster() {
local instance_id=$(aws ec2 describe-instances --filters \
"Name=tag:Environment,Values=staging" \
"Name=tag:Name,Values=my-cluster-name" \
--query 'Reservations[*].Instances[*].[InstanceId]' \
| grep i- | awk '{print $1}' | tail -1)
aws ssm start-session --profile AccountProfile --target $instance_id
}
When I run this function, I always get an error like the following:
An error occurred (TargetNotConnected) when calling the StartSession operation: "i-0599385eb144ff93c" is not connected.
However, then I take that instance id and run it from my terminal directly, it works:
aws ssm start-session --profile MyProfile --target i-0599385eb144ff93c
Why is this?
You're sending instance ID as "i-0599385eb144ff93c" instead of i-0599385eb144ff93c.
Modified function that should work -
function ssm_to_cluster() {
local instance_id=$(aws ec2 describe-instances --filters \
"Name=tag:Environment,Values=staging" \
"Name=tag:Name,Values=my-cluster-name" \
--query 'Reservations[*].Instances[*].[InstanceId]' \
| grep i- | awk '{print $1}' | tail -1 | tr -d '"')
aws ssm start-session --profile AccountProfile --target $instance_id
}
I am using the Amazon AWS ELB command line tools. Is there a way of finding out the instances attached to a particular Elastic Load Balancer (ELB)?
2013/12/18: To update this and since the links are dead!
I installed the new AWS cli tools:
$ pip install awscli
Then ran:
$ aws configure
AWS Access Key ID [None]: my-key
AWS Secret Access Key [None]: my-secret
Default region name [None]: us-east-1
Default output format [None]:
This data is saved into ~/.aws/config.
Then I can find instances connected to a loadbalancer like so:
$ aws elb describe-load-balancers --load-balancer-name "my-name"
{
"LoadBalancerDescriptions": [
{
"Subnets": [],
"CanonicalHostedZoneNameID": "ID",
"CanonicalHostedZoneName": "my-name-foo.us-east-1.elb.amazonaws.com",
"ListenerDescriptions": [
{
"Listener": {
"InstancePort": 80,
"LoadBalancerPort": 80,
"Protocol": "HTTP",
"InstanceProtocol": "HTTP"
},
"PolicyNames": []
},
{
"Listener": {
"InstancePort": 80,
"SSLCertificateId": "arn:aws:iam::x:server-certificate/x-ssl-prod",
"LoadBalancerPort": 443,
"Protocol": "HTTPS",
"InstanceProtocol": "HTTP"
},
"PolicyNames": [
"AWSConsole-SSLNegotiationPolicy-api-production"
]
}
],
"HealthCheck": {
"HealthyThreshold": 10,
"Interval": 30,
"Target": "HTTP:80/healthy.php",
"Timeout": 5,
"UnhealthyThreshold": 2
},
"BackendServerDescriptions": [],
"Instances": [
{
"InstanceId": "i-FIRST-INSTANCEID"
},
{
"InstanceId": "i-SECOND-INSTANCEID"
}
],
"DNSName": "my-name-foo.us-east-1.elb.amazonaws.com",
"SecurityGroups": [],
"Policies": {
"LBCookieStickinessPolicies": [],
"AppCookieStickinessPolicies": [],
"OtherPolicies": [
"AWSConsole-SSLNegotiationPolicy-my-name"
]
},
"LoadBalancerName": "my-name",
"CreatedTime": "2013-08-05T16:55:22.630Z",
"AvailabilityZones": [
"us-east-1d"
],
"Scheme": "internet-facing",
"SourceSecurityGroup": {
"OwnerAlias": "amazon-elb",
"GroupName": "amazon-elb-sg"
}
}
]
}
The data is in LoadBalancerDescriptions.Instances.
My loadbalancer is called my-name — this is the name you selected when you created it.
Old answer below!
I'm not familiar with the cli tool, but I used the API.
I'd check these two requests:
DescribeLoadBalancers
DescribeInstanceHealth
The cli tool probably has something to resemble these?
HTH!
Assuming you have aws-cli and jq installed, you can use the following command to get associated ec2 instance ids:
aws elb describe-load-balancers --load-balancer-name my-elb \
| jq -r '.LoadBalancerDescriptions[].Instances[].InstanceId'
This will return the ec2 ids associated with that ELB.
Side note: I recommend you setup aws cli profiles so you don't have to fiddle with environment variables and region params (as much).
Because I love answers that can be used with a minimum of search/replace and copy paste
Prerequisites : aws-cli configured
pip install awscli
aws configure
Configure : your ELB name
$ELB_NAME = "Your-elb-name"
Copy-n-Paste in terminal
for ID in $(aws elb describe-load-balancers --load-balancer-name $ELB_NAME \
--query LoadBalancerDescriptions[].Instances[].InstanceId \
--output=text);
do
aws ec2 describe-instances --instance-ids $ID \
--query Reservations[].Instances[].PublicIpAddress \
--output text
done
Will output a list of Public IPs. You could also just execute the query inside the parenthesis of the for ID in $(...) to just get the instance IDs
Want something different ?
Feel free to have a look at the structure of
aws elb describe-load-balancers --load-balancer-name $ELB_NAME
aws ec2 describe-instances --instance-ids $INSTANCE_ID
and change the query accordingly!
If anyone arrives here from a search as to why the elb-describe-lbs command returns nothing when they have ELBs up and running, what I realised was I needed to add EC2_REGION=eu-west-1 to my environment variables (or use elb-describe-lbs --region command)
If you want to see all your ELB's and the instances attached use JMESPath like this:
aws elb describe-load-balancers --query "LoadBalancerDescriptions[*].{ID:LoadBalancerName,InstanceId:Instances[*].InstanceId}[*]. {ELB:ID,InstanceId:InstanceId[*]}" --output=json
Result
[
{
"ELB": "my_name",
"InstanceId": [
"i-0cc72"
]
},
{
"ELB": "my_name2",
"InstanceId": [
"i-02ff5f",
"i-09e467"
]
}
]
If you know the name of the ELB and want to see what is attached use JMESPath like this:
aws elb describe-load-balancers --load-balancer-name "my_name" --query "LoadBalancerDescriptions[].{ID:LoadBalancerName,InstanceId:Instances[].InstanceId}[].{ELB:ID,InstanceId:InstanceId[]}" --output=json
Result:
[
{
"ELB": "my_name",
"InstanceId": [
"i-02ff5f72",
"i-09e46743"
]
}
]
replace INSTANCEID with actual instance id
aws elb describe-load-balancers --query "LoadBalancerDescriptions[*].{ID:LoadBalancerName,InstanceId:Instances[?InstanceId=='INSTANCEID'].InstanceId}[*].{ID:ID,InstanceId:InstanceId[0]}" --output=text | grep INSTANCEID | awk '{print $1}'
In node.js you can do this by using aws-sdk.
var AWS = require('aws-sdk')
var options = {
accessKeyId: 'accessKeyId',
secretAccessKey: 'secretAccessKey',
region: 'region'
}
var elb = new AWS.ELB(options)
elb.describeLoadBalancers({LoadBalancerNames: ['elbName']}, function(err, data) {
if (err) {
console.log('err: ', err)
}
else {
console.log('data: ', data.LoadBalancerDescriptions)
}
})
data.LoadBalancerDescriptions is an array and each element in the array is an object with the property Instances that has the instance id.
You can loop trough all you load balancer instance ids as follows:
while read -r lb ; do echo -e "\n\n start lb: $lb " ; \
echo run cmd on lb: $lb ; echo " stop lb: $lb" ; \
done < <(aws elb describe-load-balancers --query \
'LoadBalancerDescriptions[].Instances[].InstanceId' \
--profile dev|perl -nle 's/\s+/\n/g;print')
You can loop trough your load balancers names as follows :
# how-to loop trough all your load balancer names
while read -r lb ; do \
echo -e "\n\n start lb: $lb " ; \
echo run cmd on lb: $lb ; \
echo " stop lb: $lb" ; \
done < <(aws elb describe-load-balancers --query \
'LoadBalancerDescriptions[].LoadBalancerName' \
--profile rnd|perl -nle 's/\s+/\n/g;print')
Provided that you have configured your aws cli :
src: http://docs.aws.amazon.com/cli/latest/topic/config-vars.html
cat << "EOF" > ~/.aws/config
[profile dev]
output = text
region = us-east-1
[profile dev]
output = text
region = us-east-1
[default]
output = text
region = Global
EOF
And configured your security credentials:
# in aws admin console :
# Services => iam => users => <<your_username>> => Security Credentials => Access Keys
# configure the aws cli
cat << "EOF" > ~/.aws/credentials
[dev]
aws_access_key_id = <<your_aws_access_key_id_in_the_dev_environment>>
aws_secret_access_key = <<your_aws_secret_access_key_in_dev_env>>
[dev]
aws_access_key_id = <<your_aws_access_key_id_in_the_dev_environment>>
aws_secret_access_key = <<your_aws_secret_access_key_in_dev_env>>
[default]
aws_access_key_id = <<your_aws_access_key_id_in_the_dev_environment>>
aws_secret_access_key = <<your_aws_secret_access_key_in_dev_env>>
EOF
aws elb describe-load-balancers --load-balancer-name "LB_NAME" | grep "InstanceId" | awk '{print $2}' | sed 's/\"//g'
First do elb-describe-lbs to get a list of your load balancers and their names.
Then do elb-describe-instance-health <LB_NAME> to get a list of instances behind that load balancer. LB_NAME is the value of the 2nd column in the output of elb-describe-lbs.
You can use AWS command line tools with some bash piping:
elb-describe-instance-health loadbalancer_name --region eu-west-1 | awk '{ print $2 }' | xargs ec2-describe-instances --region eu-west-1 | grep ^INSTANCE | awk '{ print $4 }'
This will give you the public DNS name for every instance attached to the ELB, you can change the awk columns respectively to get other details.