I am saving output of my command in a variable and using it in other command
instance_id=$(aws ec2 describe-instances --query Reservations[*].Instances[*].[InstanceId] --filters "Name=tag:staging,Values=staging")
I get this output [ [ [ "i-09140824c1b7f9ea7" ] ] ]
How to I remove brackets from the output and use it in the variable in this command
aws ec2 associate-address --instance-id $instance_id --allocation-id allocid
I am new to bash so any help would be appreciated.
You can extract the value and decode it from JSON to plain text with the following:
instance_id_json='[ [ [ "i-09140824c1b7f9ea7" ] ] ]'
instance_id=$(jq -r '.[][][]' <<<"$instance_id_json")
Consider changing your original code to:
instance_id_json=$(aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters 'Name=tag:staging,Values=staging')
...note that we're putting the query in quotes (since it has glob characters), not just the filter (and since the filter is intended to be literal text with no expansions, we're defaulting to single-quotes rather than double as a matter of form).
Related
I am trying to extract a few properties of AWS EC2 instance using JMESPath. I just started learning JMESPath using this useful guide. But because my knowledge is limited, I was only able to get the output like below.
% aws ec2 describe-instances --query "Reservations[].Instances[].[InstanceId,InstanceType,KeyName,LaunchTime,State.Name,Tags[].[Key,Value]]" --instance-ids i-123456789101123
[
[
"i-123456789101123",
"r5.8xlarge",
"My_EMR",
"2022-12-19T16:11:42.000Z",
"running",
[
[
"tag-key-1",
"tag-value-1"
],
[
"tag-key-2",
"tag-value-2"
],
[
"tag-key-3",
"tag-value-3"
],
[
"project",
"my-project"
]
]
]
]
Ideally, I would like to format the output using JMESPath like below.
[
[
"i-123456789101123",
"r5.8xlarge",
"My_EMR",
"2022-12-19T16:11:42.000Z",
"running",
{
"tag-key-1": "tag-value-1",
"tag-key-2": "tag-value-2",
"tag-key-3": "tag-value-3",
"project": "my-project"
}
]
]
I have tried a bunch of different JMESPath expressions like below, but none of them seem to work. I'd greatly appreciate any suggestion. Thank you in advance!
# JMESPath statements that I've tried so far (actually there are more, but these are the 'promising' ones that I tried, but failed to work)
% aws ec2 describe-instances --query "Reservations[].Instances[].[InstanceId,InstanceType,KeyName,LaunchTime,State.Name,map(Tags[].Key,Tags[].Value)]"
% aws ec2 describe-instances --query "Reservations[].Instances[].[InstanceId,InstanceType,KeyName,LaunchTime,State.Name,map(Tags[].Key,Tags[].Value)]"
% aws ec2 describe-instances --query "Reservations[].Instances[].{InstanceId: InstanceId, InstanceType: InstanceType, KeyName: KeyName, LaunchTime: LaunchTime, StateName: State.Name, Tags[].Key: Tags[].Value}"
% aws ec2 describe-instances --query "Reservations[].Instances[].{InstanceId: InstanceId, InstanceType: InstanceType, KeyName: KeyName, LaunchTime: LaunchTime, StateName: State.Name, Tags[*].Key: Tags[*].Value}"
I have the following bash script that I execute in order to create new Glue Job via CLI:
#!/usr/bin/env bash
set -e
NAME=$1
PROFILE=$2
SCRIPT_LOCATION='s3://bucket/scripts/'$1'.py'
echo [*]--- Creating new job on AWS
aws glue create-job --profile $PROFILE --name $NAME --cli-input-json | jq '.Command.ScriptLocation = '\"$SCRIPT_LOCATION\"'' ./resources/config.json
I'm using jq as i need one of the values to be replaced on runtime before i pass the .json as --cli-input-json argument. How can i pass json with replaced value to this command? As of now, it prints out the json content (although with value already replaced).
Running the command above causes the following error:
[*]--- Creating new job on AWS
{
"Description": "Template for Glue Job",
"LogUri": "",
"Role": "arn:aws:iam::11111111111:role/role",
"ExecutionProperty": {
"MaxConcurrentRuns": 1
},
"Command": {
"Name": "glueetl",
"ScriptLocation": "s3://bucket/scripts/script.py",
"PythonVersion": "3"
},
"DefaultArguments": {
"--TempDir": "s3://temp/admin/",
"--job-bookmark-option": "job-bookmark-disable",
"--enable-metrics": "",
"--enable-glue-datacatalog": "",
"--enable-continuous-cloudwatch-log": "",
"--enable-spark-ui": "true",
"--spark-event-logs-path": "s3://assets/sparkHistoryLogs/"
},
"NonOverridableArguments": {
"KeyName": ""
},
"MaxRetries": 0,
"AllocatedCapacity": 0,
"Timeout": 2880,
"MaxCapacity": 0,
"Tags": {
"KeyName": ""
},
"NotificationProperty": {
"NotifyDelayAfter": 60
},
"GlueVersion": "3.0",
"NumberOfWorkers": 2,
"WorkerType": "G.1X"
}
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:
aws help
aws <command> help
aws <command> <subcommand> help
aws.exe: error: argument --cli-input-json: expected one argument
The command line
aws glue create-job --profile $PROFILE --name $NAME --cli-input-json | jq '.Command.ScriptLocation = '\"$SCRIPT_LOCATION\"'' ./resources/config.json
executes the command
aws glue create-job --profile $PROFILE --name $NAME --cli-input-json,
takes its standard output and uses it as input to
jq '.Command.ScriptLocation = '\"$SCRIPT_LOCATION\"'' ./resources/config.json
(which will ignore the input and read from the file given as argument). Please also note that blanks or spaces in $SCRIPT_LOCATION will break your script, because it is not quoted (your quotes are off).
To use the output of one command in the argument list of another command, you must use Command Substitution: outer_command --some-arg "$(inner_command)".
So your command should become:
aws glue create-job --profile $PROFILE --name $NAME --cli-input-json "$(jq '.Command.ScriptLocation = "'"$SCRIPT_LOCATION"'"' ./resources/config.json)"
# or simplified with only double quotes:
aws glue create-job --profile $PROFILE --name $NAME --cli-input-json "$(jq ".Command.ScriptLocation = \"$SCRIPT_LOCATION\"" ./resources/config.json)"
See https://superuser.com/questions/1306071/aws-cli-using-cli-input-json-in-a-pipeline for additional examples.
Although, I have to admit I am not 100% certain that the JSON content can be passed directly on the command line. From looking at the docs and some official examples, it looks like this parameter expects a file name, not a JSON document's content. So it could be possible that your command in fact needs to be:
# if "-" filename is specially handled:
jq ".Command.ScriptLocation = \"$SCRIPT_LOCATION\"" ./resources/config.json | aws glue create-job --profile $PROFILE --name $NAME --cli-input-json -
# "-" filename not recognized:
jq ".Command.ScriptLocation = \"$SCRIPT_LOCATION\"" ./resources/config.json > ./resources/config.replaced.json && aws glue create-job --profile $PROFILE --name $NAME --cli-input-json file://./resources/config.replaced.json
Let us know which one worked.
The az keyvault secret list --vault-name "lofa" results in a list similar to what is below (but has way more elements of each type):
[
{
"attributes": { ... },
"id": "https://lofa.vault.azure.net/secrets/conn-string",
"tags": {
"file-encoding": "utf-8"
}
},
{
"attributes": { ... },
"id": "https://lofa.vault.azure.net/secrets/a-password",
"tags": null
},
{
"attributes": { ... },
"id": "https://lofa.vault.azure.net/secrets/another-password",
"tags": {
"testThis": "miez",
"what": "else"
}
}
]
Tried to filter the "easy" targets first (i.e., JSON objects where the keys contain no hyphens), and it worked as expected:
$ az keyvault secret list --vault-name "lofa" --query "[? tags.testThis=='vmi']"
The same didn't work for file-encoding keys (resulting in invalid jmespath_type value):
$ az keyvault secret list --vault-name "lofa" --query "[?tags.file-encoding=='utf-8']"
So tried single quotes next, but no joy:
$ az keyvault secret list --vault-name "lofa" --query "[?tags.'file-encoding'=='utf-8']"
And if you do want a solution that does not forces you to escape any quotes, you can use:
simple quotes for your shell script parameters
az keyvault secret list --vault-name 'lofa' --query '[]'
double quotes for your key, since it contains a dash
az keyvault secret list --vault-name 'lofa' --query '[?tags."file-encoding"]'
backticks for your literal, utf-8
az keyvault secret list --vault-name 'lofa' --query '[?tags."file-encoding"==`utf-8`]'
The solution was using
escaped double quotes (\") for JSON object keys (in this case, for file-encoding), and
single quotes for JSON object values (in this case, for utf-8)
az keyvault secret list --vault-name "lofa" --query "[?tags.\"file-encoding\"=='utf-8']"
NOTE: Not sure why this is, or what is happening exactly but when trying to substitute the single quotes with escaped double quotes as well for the value part,
az keyvault secret list --vault-name "lofa" --query "[?tags.\"file-encoding\"==\"utf-8\"]"
it resulted in a list that had all the entries except the ones that had tags.file-encoding.
I try to sort this output from AWS CLI by ImageId, and I executed command below.
aws ec2 describe-images --profile xxxxxxxxxx \
--filter Name=tag:Name,Values=Backup*some-string* \
--query "Images[*].[Tags[?Key=='Name'].Value[]|[0],ImageId]"
output is:
[
[
"Backup-20191215T174530Z-utc-some-string",
"ami-004"
],
[
"Backup-20191219T174631Z-utc-some-string",
"ami-002"
],
[
"Backup-20191208T174534Z-utc-some-string",
"ami-001"
],
[
"Backup-20191222T174530Z-utc-some-string",
"ami-003"
],
[
"Backup-20191221T174530Z-utc-some-string",
"ami-005"
]
]
I found sort_by functions of JMESPath could be a solution but that is too hard to understand.
aws ec2 describe-images --profile xxxxxxxxxx \
--filter "Name=tag:Name,Values=Backup*some-string*" \
--query "sort_by(Images[*].[Tags[?Key=='Name'].Value[]|[0],ImageId], &[0])"
or
aws ec2 describe-images --profile xxxxxxxxxx \
--filter "Name=tag:Name,Values=Backup*some-string*" \
--query "Images[*].[Tags[?Key=='Name'].Value[]|[0],ImageId] | sort_by(#, &[0])"
is working fine for me. & (expression type operator) is needed.
The idea is, In my solution below, I am sorting the output first by the ImageId and then applying the projections.
aws ec2 describe-images --filter Name=tag:Environment,Values=Staging --output json --query "(sort_by(Images[], &ImageId))[*].[ImageId, Tags[?Key=='Environment'].Value | [0]]"
Messing around with a simple aws cli query to check for the existence of a Lambda function and echo the associated role if it exists:
#!/bin/bash
fname=$1
role=$(aws lambda list-functions --query 'Functions[?FunctionName == `$fname`].Role' --output text)
echo "$fname role: $role"
However, $fname appears to be resolving to an empty string in the aws command. I've tried escaping the back ticks, swapping ` to ' and a miriad of other thrashing edits (and yes, I'm passing a string on the cl when invoking the script :)
How do I properly pass a variable into JMESPath query inside a bash script?
Because the whole JMESPath expression is enclosed in single quotes, bash is not expanding the $fname variable. To fix this you can surround the value with double quotes and then use single quotes (raw string literals) for the $fname var:
aws lambda list-functions --query "Functions[?FunctionName == '$fname'].Role" --output text
Swapping the backticks to single quotes, didn't work for me... :(
But escaping the backticks works :)
Here are my outputs:
aws elbv2 describe-listeners --load-balancer-arn $ELB_ARN --query "Listeners[?Port == '$PORT'].DefaultActions[].TargetGroupArn | [0]"
null
aws elbv2 describe-listeners --load-balancer-arn $ELB_ARN --query "Listeners[?Port == \`$PORT\`].DefaultActions[].TargetGroupArn | [0]"
"arn:aws:elasticloadbalancing:ap-southeast-2:1234567:targetgroup/xxx"