I've got a group of AWS instances that I'm parsing via aws ec describe-instances. I'm looking to trim out all the records whose IP's do not start with '10.10'.
aws ec2 describe-instances --no-paginate --filter "Name=instance-state-name,Values=running" --query 'Reservations[].Instances[].{Private:PrivateIpAddress,PublicDNS:PublicDnsName,PublicIP:PublicIpAddress}' | jq '.[] | select( .Private | contains("10.10"))'
This gets me the exact opposite of what I want. It seems logical that I should be able to negate the contains in some way - but I've not been able to glean it from the documentation, nor through experimentation. My jq proficiency is middling, so perhaps I'm using the wrong operator or function here.
While i WOULD like an answer to this specific jq question - I'll accept an answer that utilizes JMESPath through the --query switch yield the same result.
Jeff Marcado's answer in the comments will be accepted if he writes it up as a full fledged answer. In the meantime, since I had hit a wall with trying to get JQ to do it, I experimented with the --query syntax for AWS to get this.
It might be a bit better, since this catches only objects that start with 10.10, whereas the jq from above will catch any object that contains 10.10, so things like 10.100. or 110.100, etc... will get through. That's assuming there is not a similar operator to "starts_with" in jq. Probably there is. Regardless, I'm putting this here because it worked for my end goal and may be helpful to someone else at some point.
aws ec2 describe-instances \
--no-paginate --filter "Name=instance-state-name,Values=running" \
--query 'Reservations[].Instances[?starts_with(PrivateIpAddress, `10.10.`) == `false`]' |
jq '.[] | .[] | {PrivateIpAddress, PublicIpAddress, PublicDnsName}'
Related
Given the input document:
{"a":1}
{"b":2}
{"c":3,"d":4}
What is the difference between the following jq programs (if any)? They all seem to produce the same output.
jq '[., inputs] | map(to_entries[].value)'
jq -n '[inputs] | map(to_entries[].value)'
jq -s 'map(to_entries[].value)'
In other words, the following (simplified/reduced) invocations seem identical:
jq '[.,inputs]'
jq -n '[inputs]'
jq -s '.'.
How are they different? Are there scenarios where one works, but the others don't? Did older versions of jq not support all of them? Is it performance related? Or simply a matter of readability and personal preference?
Bonus points (added later to the question): does the same hold true for the following programs?
jq '., inputs | to_entries[].value'
jq -n 'inputs | to_entries[].value'
jq -s '.[] | to_entries[].value'
jq 'to_entries[].value'
With jq '-n [inputs] ....' and jq '[.,inputs] ....', you are loading the whole file into memory.
A more memory-efficient way to achieve the result as an array is:
jq -n '[inputs | to_entries[].value]'
Those first three programs are equivalent, both functionally and in terms of resource utilization, but they obscure the difference between array-oriented and stream-oriented programming.
In a nutshell, think sed and awk. For more details, see e.g. my A Stream-oriented Introduction to jq, and i.p. the section On the importance of inputs.
Bonus points: does the same hold true for the following programs:
Referring to the last four numbered examples in the Q: (4), (5) and (7) are essentially equivalent; (6) is just silly.
If you're looking for a reason why all these variations exist, please bear in mind that input and inputs were late additions in the development of jq. Perhaps they were late additions because jq was originally envisioned as a very simple and for the most part "purely functional" language.
Adding even more cases for the sake of completeness:
From the manual:
--raw-input/-R:
Don't parse the input as JSON. Instead, each line of text is passed to the filter as a string. If combined with --slurp, then the entire input is passed to the filter as a single long string.
This means that on one hand
jq -R -n '[inputs]' and
jq -R '[., inputs]'
both produce an array of strings, as each item provided by inputs (and . if it wasn't silenced by -n) corresponds to a line of text from the input document(s), whereas on the other hand
jq -R -s '.'
slurps all characters from the input document(s) into exactly one long string, newlines included.
I am using this code to get my AWS region. I want working on them in a while loop.
awsRegionList=$(aws ec2 describe-regions | jq -r '.Regions[] | .RegionName')
while [I can't find the expression work with my variable]:
do
echo " working on : (I want here the regionName)"
done
In bash you need to use a for loop to iterate over a list, instead of a while loop:
awsRegionList=$(aws ec2 describe-regions | jq -r '.Regions[] | .RegionName')
for region in $awsRegionList
do
echo " working on : ${region}"
done
I'm using below jq statement with grep in my code to print a value.
jq '.Subnets[0].Tags' subnet.txt | grep -q "${add}usea1 internal us-east"
This works fine for some values however, few values need grep to be "${add}use* internal us-east", can i use asterisk so that all my values can be printed.
I get error when i include asterisk. any suggestions?
You have not followed the mcve guidelines, but as #shellter pointed out, the problem description suggests you just have to use the proper (grep) regex:
grep -q "${add}use.* internal us-east"
However, since you are using jq in any case, it would probably be better to perform the filtering by extending the jq filter, for example as follows:
jq --arg add "$add" '
.Subnets[0].Tags
| select(test("\($add)use.* internal us-east"))
' subnet.txt
I am using below command to list the UserPool Names.
aws cognito-idp list-user-pools --max-results 60 --region us-west-2 --query 'UserPools[*].{Names:Name}'
Now my all UserPool Names contains cust_ as prefix. And i want to remove that from whole list.
I know i can achieve this using jq.
But how?
Any help will be highly appreciated.
Thanks!
If you output with --output text, it will become a text list.
You could then use standard Linux tools such as piping it through | cut -c6-
This will provide character #6 onwards for each line.
Full command would be something like:
aws cognito-idp list-user-pools --max-results 60 --region us-west-2 --query 'UserPools[*].[{Names:Name}]' --output text | cut -c6-
I'm trying to write a scalable and reusable script to provision ec2s using ansible. As part of this, I would like to be able to determine which Route53 hosted zone my machine is a part of, so I can add it as a record set for a private zone. I don't want to have to enter the zone ... I want to be able to figure it out using the ec2.
For a given ec2, I can get the instance. From the instance, I get get VPC-ID. I know that VPC-IDs are associated with Route53 hosted zones, but I can't seem to find an AWS CLI command to figure out the hosted zone from the VPC-ID.
I've found the command'route53 list-vpc-association-authorizations --hosted-zone-id=' command, which has to be run on each individual zone, but the result is an empty array for a zone that I know for a fact is associated with a VPC.
Can anyone help me to derive the correct private hosted zone, given that I know the VPC ID and ec2 instance id?
Thanks
Maybe too simple for people, but this works:
aws route53 list-hosted-zones --output text | grep 'MYDOMAIN' | awk '{print $3}' | cut -c13-
...Just lists the domains in AWS in column format, searches for your domain and then cuts out the zone id with awk and cut.
Took me a while, but I figured it out:
getHostedZone(){
ZONE_IDS=$(aws route53 --region $2 list-hosted-zones | jq ".HostedZones | map(.Id)")
while IFS= read -r; do
ZONE=$(aws route53 --region $2 get-hosted-zone --id $REPLY)
hasVPCs=$(echo $ZONE | jq 'has("VPCs")')
VPCs=$(echo $ZONE | jq ".VPCs")
if [ "$hasVPCs" == true ]
then
VPC=$(echo $VPCs | jq ".[] | select(.VPCId == \"$1\")")
if [ -n "$VPC" ]
then
HOSTED_ZONE=$(echo $REPLY | sed 's/^\/hostedzone\///g')
fi
fi
done < <(echo $ZONE_IDS | jq -r '.[]')
echo $HOSTED_ZONE
}
Called with:
ZONE_ID=$(getHostedZone $VPC_ID $EC2_REGION)