AWS CLI command aws emr list-clusters returns the following json. Is there a way through bash or groovy that I can use the Name to get the Id? I can't just use the Id becauseI am deleting clusters and rebuilding them with only the same name. So I know the name of the cluster is "my-cluster" and I would like to use it somehow to get the Id of the cluster. End case is I actually want the Master public DNS of the cluster.
{
"Clusters": [
{
"Status": {
"Timeline": {
"ReadyDateTime": 1433200405.353,
"CreationDateTime": 1433199926.596
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting after step completed"
}
},
"NormalizedInstanceHours": 6,
"Id": "j-3SD91U2E1L2QX",
"Name": "my-cluster"
},
{
"Status": {
"Timeline": {
"ReadyDateTime": 1433200405.353,
"CreationDateTime": 1433199926.596
},
"State": "WAITING",
"StateChangeReason": {
"Message": "Waiting after step completed"
}
},
"NormalizedInstanceHours": 6,
"Id": "j-3S281UEK8L2LW",
"Name": "my-cluster2"
}
]
}
You can use the query parameter to achieve what you are trying. The command will look like below:
aws emr list-clusters --query 'Clusters[?Name==`my-cluster`].Id' --output text
You can find the complete documentation for the query parameter here.
To answer the complete question, one could put the above approach into a long one-liner over several lines.
aws emr describe-cluster \
--output text \
--cluster-id $(aws emr list-clusters \
--active \
--query 'Clusters[?Name==`my-cluster`].Id' \
--output text) \
--query Cluster.MasterPublicDnsName
You can get the ID using jq bash command as follows
aws emr list-clusters --active | jq '.["Clusters"][0]["Id"]'
It will return the first active cluster Id as output
Related
The az webapp identity show command: https://learn.microsoft.com/en-us/cli/azure/webapp/identity?view=azure-cli-latest#az-webapp-identity-show
Will return the following result:
{
"principalId": null,
"tenantId": null,
"type": "UserAssigned",
"userAssignedIdentities": {
"/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx": {
"clientId": "xxx",
"principalId": "xxx"
}
}
}
Now I am only intrested in returning the /subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxx value. How can I filter this value using --query option? I have tried
--query userAssignedIdentities
"/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx": {
"clientId": "xxx",
"principalId": "xxx"
}
}
But there is something more required to get the first key here. What?
Use the below Azure CLI cmdlet
az webapp identity show -name <WebAppName> -g <ResoureGroupName> --query "keys(userAssignedIdentities)" -o tsv
Alternatively, you can use the below PowerShell cmdlet to pull the resourceId of the userAssignedIdentities that is assigned to Azure web app.
$identity=Get-AzWebApp -Name <WebAppName> -ResourceGroupName "<ResourceGroupName>" | select -ExpandProperty Identity
$identity.UserAssignedIdentities.Keys
I want to use jq to get value's out of a the Traefik API service. The output of the service looks like this:
{
"loadBalancer": {
"servers": [
{
"url": "http://172.18.0.19:3000"
},
{
"url": "http://172.18.0.17:3000"
},
{
"url": "http://172.20.0.3:3000"
}
],
"healthCheck": {
"path": "/health",
"interval": "10s",
"timeout": "10s",
"followRedirects": true
},
"passHostHeader": true
},
"status": "enabled",
"serverStatus": {
"http://172.18.0.17:3000": "UP",
"http://172.18.0.19:3000": "UP",
"http://172.20.0.3:3000": "DOWN"
},
"provider": "docker",
"type": "loadbalancer"
}
I want to get the value of the serverStatus dynamically. First assigning a variable with the correct IP address from docker inspect. I'm using curl with jq to check if the container is being serviced.
The problem is when getting it with a variable I cannot seem to escape the special characters in the url key. Nothing is returned.
This is the command I use is:
TRAEFIK_URL="http://someurl.com/"
TRAEFIK_URL="'Authorization: Basic hashkey"
NEW_IPADDRESS=$(docker container inspect some_container_name | jq -r .[].NetworkSettings.Networks.proxy.IPAddress)
NEW_ENV_URL="http://${NEW_IPADDRESS}:3000"
curl --location --request GET "$TRAEFIK_URL" --header "$TRAEFIK_AUTH_HEADER" | jq -r '.serverStatus["${NEW_ENV_URL}"]'
I've tried using this command and some other obvious options but all didn't work.
jq -r .serverStatus[$NEW_ENV_URL]
I do get the correct value when I'm not using a variable to get the status using:
jq -r '.serverStatus["http://172.18.0.17:3000"]'
Any help is welcome.
Use the option --arg to pass arguments to jq.
jq --arg url http://172.18.0.17:3000 -r '.serverStatus[$url]'
Always use this option to avoid security problems by injections.
One of the preferred ways to pass specific environment variables into jq is using the --arg command-line option; in your case:
jq -r --arg url "${NEW_ENV_URL}" '.serverStatus[$url]'
I'm trying to put an item to DynamoDB using CLI, but I'm getting an error: Unknown options: {, "S":, "do_something", },, "CreatedOn":, {, "N":, "201412250053", }, }, "TestRunId":
ENTRY='{ "TestRunId": { "S": "do_something" }, "CreatedOn": {"N": "201412250053"} }'
JSON=$(echo ${ENTRY} | jq .)
aws dynamodb put-item --table-name TestResultsDDBTable --item ${JSON} --region ${REGION}
If you run it in windows cmd, try this:
aws dynamodb put-item --table-name TestResultsDDBTable --item "{
\"TestRunId\": { \"S\": \"do_something\" }, \"CreatedOn\": {\"N\":
\"201412250053\"} }" --region ${REGION}
I was bothered by this question for one afternoon, :)
and I finally found a solution in here: https://acloud.guru/forums/aws-dynamodb/discussion/-KUX8EQjmN7F2pNKvoWW/when-i-am-trying-to-run-below-command-aws-dynamodb-put-item-table-name-weatherst
I'm using the gcloud describe command to get metadata information about instances.What's the best way to filter the json response with jq to get the name of the instance - if it contains "kafka" as a key.
.name + " " + .metadata.items[]?.key | select(contains("kafka"))'
Basically if items contains kafka print name.This is just a small excerpt from the json file.
"metadata": {
"fingerprint": "xxxxx=",
"items": [
{
"key": "kafka",
"value": "xxx="
},
{
"key": "some_key",
"value": "vars"
}
],
"kind": "compute#metadata"
},
"name": "instance-name",
"networkInterfaces": [
{
"accessConfigs": [
{
"kind": "compute#accessConfig",
"name": "External NAT",
"natIP": "ip",
"type": "ONE_TO_ONE_NAT"
}
],
"kind": "compute#networkInterface",
"name": "",
"network": xxxxx
}
],
I'm sure this is possible with jq, but in general working with gcloud lists is going to be easier using the built-in formatting and filtering:
$ gcloud compute instances list \
--filter 'metadata.items.key:kafka' \
--format 'value(name)'
--filter tells you which items to pick; in this case, it grabs the instance metadata, looks at the items, and checks the keys for those containing kafka (use = instead to look for keys that are exactly kafka).
--format tells you to grab just one value() (as opposed to a table, JSON, YAML) from each matching item; that item will be the name of the instance.
You can learn more by running gcloud topic filters, gcloud topic formats, and gcloud topic projections.
Here is a simple jq solution using if and any:
if .metadata.items | any(.key == "kafka") then . else empty end
| .name
I have a response from aws like this
{
"Owner": {
"DisplayName": "2414218.aws",
"ID": "xxxxxxxx"
},
"Grants": [
{
"Grantee": {
"DisplayName": "2414218.aws",
"ID": "yyyyyyyyyy"
},
"Permission": "FULL_CONTROL"
},
{
"Grantee": {
"URI": "http://acs.amazonaws.com/groups/global/AllUsers"
},
"Permission": "READ"
}
]
}
I am looking to update the files so AllUsers is removed (get-object-acl should look like the below)
{
"Owner": {
"DisplayName": "2414218.aws",
"ID": "xxxxxxxx"
},
"Grants": [
{
"Grantee": {
"DisplayName": "2414218.aws",
"ID": "yyyyyyyyyy"
},
"Permission": "FULL_CONTROL"
}
]
}
How would I do this, not knowing what other grantees are available? I am specifically looking to remove a grantee when I see http://acs.amazonaws.com/groups/global/AllUsers
I am currently using aws s3api get-object-acl --bucket mhe-deployments-prod --key $keyFile | jq '.' to locate the policies
Here is a jq filter which will remove all the .Grants array elements for which .Grantee.URI is "http://acs.amazonaws.com/groups/global/AllUsers":
.Grants |= map( select(.Grantee.URI != "http://acs.amazonaws.com/groups/global/AllUsers") )
Output: as requested
You can do using the built in --query option in the AWS CLI. The benefit being you don't need any external tools:
aws s3api get-object-acl --bucket $BUCKET --key $KEY \
--query "{Owner: Owners, \
Grants: Grants[?Grantee.URI != 'http://acs.amazonaws.com/groups/global/AllUsers']}"
Probably not the best but, this worked
#!/usr/local/bin/bash
# aws ~/.aws/credentials and s3cmd must be configured first with proper creds
target=''
for key in $(aws s3 ls s3://$target --recursive |awk '{print$4}') ; do
[ "${key: -1}" == "/" ] || {
award=$(aws s3api get-object-acl --bucket $target --key $key |jq '.Grants[].Grantee | .URI' |grep -v 'null' |grep AllUsers)
[ ! -z "${award}" ] && {
policy=$(aws s3api get-object-acl --bucket $target --key $key)
echo "$target: $key\n$policy\n\n" >> /tmp/policy-backup.json
echo -e "Working on: $key"
s3cmd setacl s3://$target/$key --acl-private ## s3cmd must be comfigured to your env
}
}
done
In this problem the specified grant can also be removed directly with del. e.g
del(
.Grants[]
| select(.Grantee.URI == "http://acs.amazonaws.com/groups/global/AllUsers")
)