I have a number of ec2 instances running in AWS, and I've extracted this information into a file.
aws ec2 describe-instances > instances.json
I also have another file ipAddressList
cat ipAddressList
10.100.39.4
10.100.56.20
10.100.78.11
10.100.78.12
I would like to extract the ImageId for these 4 instances.
I'm able to get the ImageId for individual ip addresses using this command
cat instances.json | jq '.Reservations[] | .Instances[] | select(.PrivateIpAddress == "10.100.39.41") | .ImageId'
But I would like to put this into a bash loop to extract the ImageId's for all 4 instances at once.
I've tried
for i in `cat ipAddressList` ; do jq '.Reservations[] | .Instances[] | select(.PrivateIpAddress == \$i) | .ImageId' instances.json ; done
But it throws an error.
What am I doing wrong please?
Don't inject data into code, use the --arg and --argjson options provided by jq to do that safely:
for i in `cat ipAddressList`
do jq --arg i "$i" '
.Reservations[] | .Instances[]
| select(.PrivateIpAddress == $i) | .ImageId
' instances.json
done
On top of that, jq provides options to read in files as raw text, so you could shift the entire loop into jq logic, resulting in just one invocation of jq:
jq --rawfile ips ipAddressList '
.Reservations[].Instances[]
| select(IN(.PrivateIpAddress; ($ips / "\n")[])).ImageId
' instances.json
You're really close with your solution.
What you need is
for i in `cat ipAddressList` ; do jq '.Reservations[] | .Instances[] | select(.PrivateIpAddress == "'$i'") | .ImageId' instances.json ; done
And you should be fine.
Related
I have below command:
ExpirationDate=$(date -d '+60 days' +'%Y-%m-%d')
VaultName="abc"
getapp=$(az keyvault secret list --vault-name $VaultName --query "[].{SecretName:name,ExpiryDate:attributes.expires} [?ExpiryDate<='$ExpirationDate']" | jq '.[].SecretName' | tr -d '"')
getserviceprincipal=$(az keyvault secret list --vault-name $VaultName --query "[].{Type:contentType,ExpiryDate:attributes.expires} [?ExpiryDate<='$ExpirationDate']" | jq '.[].Type' | tr -d '"')
## get length of $distro array
len=${#getapp[#]}
## Use bash for loop
for (( i=0; i-le$len-1; i++ ))
do
echo "${getapp[$i]}"
./resetpassword.sh -a ${getapp[$i]} -s ${getserviceprincipal[$i]} -y
echo "${getserviceprincipal[$i]}"
done
in this command I want store all value of vault name getapp and similarly getserviceprincipal. Example If I have more then 2 vault in getapp variable then script is not working due to $getapp is not storing variable in array.
Is anyone help me to put out this simple solutions!! Thanks In Advance..
readarray -t getapp < <( az keyvault ... | tr -d '"' ) should do the trick here.
Note that this requires newlines to be valid delimiters. If there can be newlines in your data then you'll have to pick a different delimiter with the -d delim option. If there isn't any single delimiter that works everywhere then bash may not be the best choice for this.
Since you are using jq, I think you could so something like that:
declare -a getapp=()
declare -a getserviceprincipal=()
# note: be sure to check that the resulting bash is valid!
eval(az keyvault secret list \
--vault-name $VaultName \
--query "[].{SecretName:name,ExpiryDate:attributes.expires} [?ExpiryDate<='$ExpirationDate']" \
| jq --raw-output '.[] | #sh "getapp+=( \(.SecretName) ) ; getserviceprincipal+=( \(.Type) );' ")
If all goes well, this will result in getapp and getserviceprincipal being filled as array: https://jqplay.org/s/BbHMn9i79KB
Note:
as you can see, you don't need to invoke your command (az) twice.
you can also extract the jq expression to a file using the --from-file option, which may help when reading it and handling shell quotes.
Background
I want to be able to pass a json file to WP CLI, to iteratively create posts.
So I thought I could create a JSON file:
[
{
"post_type": "post",
"post_title": "Test",
"post_content": "[leaflet-map][leaflet-marker]",
"post_status": "publish"
},
{
"post_type": "post",
"post_title": "Number 2",
"post_content": "[leaflet-map fitbounds][leaflet-circle]",
"post_status": "publish"
}
]
and iterate the array with jq:
cat posts.json | jq --raw-output .[]
I want to be able to iterate these to execute a similar function:
wp post create \
--post_type=post \
--post_title='Test Map' \
--post_content='[leaflet-map] [leaflet-marker]' \
--post_status='publish'
Is there a way I can do this with jq, or similar?
The closest I've gotten so far is this:
> for i in $(cat posts.json | jq -c .[]); do echo $i; done
But this seems to take issue with the (valid) spaces in the strings. Output:
{"post_type":"post","post_title":"Test","post_content":"[leaflet-map][leaflet-marker]","post_status":"publish"}
{"post_type":"post","post_title":"Number
2","post_content":"[leaflet-map
fitbounds][leaflet-circle]","post_status":"publish"}
Am I way off with this approach, or can it be done?
Use a while to read entire lines, rather than iterating over the words resulting from the command substitution.
while IFS= read -r obj; do
...
done < <(jq -c '.[]' posts.json)
Maybe this would work for you:
Make a bash executable, maybe call it wpfunction.sh
#!/bin/bash
wp post create \
--post_type="$1"\
--post_title="$2" \
--post_content="$3" \
--post_status="$4"
Then run jq on your posts.json and pipe it into xargs
jq -M -c '.[] | [.post_type, .post_title, .post_content, .post_status][]' \
posts.json | xargs -n4 ./wpfunction`
I am experimenting to see how this would handle post_content that contained quotes...
First generate an array of the arguments you wish to pass then convert to a shell compatible form using #sh. Then you could pass to xargs to invoke the command.
$ jq -r '.[] | ["post", "create", (to_entries[] | "--\(.key)=\(.value|tojson)")] | #sh' input.json | xargs wp
I am using JQ module the parse some of the data and then running the final loop over it to parse few more data.
cluster_list=`databricks --profile hq_dev clusters list --output JSON | jq 'select(.clusters != null) | .clusters[] | [.cluster_name,.autotermination_minutes,.state,.cluster_id] | #csv' | grep -v "job-"`
for cluster in ${cluster_list[#]}
do
cluster_id=`echo $cluster| cut -d "," -f 4 | sed 's/\"//g' | sed 's/\\\//g'`
cluster_name=`echo "${cluster}"| cut -d "," -f 1| sed 's/\"//g' | sed 's/\\\//g'`
echo $cluster_name
done
cluster_list contains following value.
"\"Test Space Cluster\",15,\"TERMINATED\",\"ddd-dese23-can858\""
"\"GatewayCluster\",15,\"TERMINATED\",\"ddd-ddsd-ddsds\""
"\"delete_later\",15,\"TERMINATED\",\"1120-195800-93839\""
"\"GatewayCluster_old\",15,\"TERMINATED\",\"0108-2y7272-393893\""
it prints following.
Test
Space
Cluster
GatewayCluster
delete_later
GatewayCluster_old
Desired output
it shouldn't break to newline if there is a space, I am doing few more action by the name I am getting here.
Test Space Cluster
GatewayCluster
delete_later
GatewayCluster_old
Your script seems a bit overly complex to achieve your goal. Better use read to store each value in a separate variable, and set a comma for the input field separator IFS:
databricks --profile hq_dev clusters list --output JSON |
jq 'select(.clusters != null) | .clusters[] |
[.cluster_name,.autotermination_minutes,.state,.cluster_id] | #csv' |
grep -v "job-" |
sed 's/\\\?"//g' |
while IFS=, read name autotermination_minutes state id ; do
echo $name
done
Note: I didn't touch your jq command. The sed line I put aims to remove quotes, protected or not. You can tune jq to remove these quotes with -r, as said in the man page:
INVOKING JQ
[...]
--raw-output / -r::
With this option, if the filter´s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.
I'm trying to create a ohmyzsh function for Salesforce's DX CLI based on Wade Wegner's guide here. In order to get the value I want I need to expand how he is using JQ which I've never heard of before. I get the premise for this use case but I'm struggling with one abstraction point (within the aliasConfig json). Here's my script so far
get_sfdx_defaultusername() {
config="$(cat .sfdx/sfdx-config.json 2> /dev/null)";
globalConfig="$(cat ~/.sfdx/sfdx-config.json)";
aliasConfig="$(cat ~/.sfdx/alias.json)";
defaultusername="$(echo ${config} | jq -r .defaultusername)"
defaultusernamealias="NEED HELP HERE"
globaldefaultusername="$(echo ${globalConfig} | jq -r .defaultusername)"
if [ ! $defaultusernamealias = "null" ]
then
echoString=$echoString$defaultusernamealias"$txtylw (alias)"
elif [ ! $defaultusername = "null" ]
then
echoString=$echoString$defaultusername"$txtylw (local)"
else
echoString=$echoString$globaldefaultusername"$txtylw (global)"
fi
echo $echoString"\n"
}
The alias.json looks like this:
{
"orgs": {
"HubOrg": "myemail#domain.com",
"my-scrath-org": "test-jdj1iflkor4k#mydomain.net"
}
}
Using the ${defaultusername} I know the value in this case to be "test-jdj1iflkor4k#mydomain.net", therefore I need it to set the value of defaultusernamealias to "my-scrath-org"
NOTE: The closest answer I found was this, but unfortunately I still couldn't get what I needed with it.
Congratulations on figuring out how to use to_entries.
One small suggestion is to avoid using shell interpolation to "construct" the jq program. A much better way to achieve the desired goal is to pass in the relevant values on the command-line. In your case, the following would be appropriate:
$ jq --arg username "$defaultusername" '
.orgs | to_entries[] | select(.value == $username ).key'
Another small point is to avoid using echo to send JSON to STDIN. There are several possibilities, including these patterns:
if you are using bash: jq .... <<< "$JSON"
use printf "%s" "$JSON" | jq ...
jq -n --argjson JSON "$JSON" '$JSON | ...'
In your case, the last of these alternatives would look like this:
$ jq --arg username "$defaultusername" --argjson JSON "$aliasConfig" '
$JSON
| .orgs | to_entries[] | select(.value == $username ).key'
I think I got it figured out here:
get_sfdx_defaultusername() {
config="$(cat .sfdx/sfdx-config.json 2> /dev/null)";
globalConfig="$(cat ~/.sfdx/sfdx-config.json)";
aliasConfig="$(cat ~/.sfdx/alias.json)";
defaultusername="$(echo ${config} | jq -r .defaultusername)"
defaultusernamealias="$(echo ${aliasConfig} | jq -r '.orgs | to_entries[] | select(.value =="'$defaultusername'").key' )"
globaldefaultusername="$(echo ${globalConfig} | jq -r .defaultusername)"
if [ ! $defaultusernamealias = "null" ]
then
echoString=$echoString$defaultusernamealias"$txtylw (alias)"
elif [ ! $defaultusername = "null" ]
then
echoString=$echoString$defaultusername"$txtylw (local)"
else
echoString=$echoString$globaldefaultusername"$txtylw (global)"
fi
echo $echoString"\n"
}
This allows me to show my current defaultusername org like so:
In case anyone is interested in using this or contributing to it, I published a github repo here
I am using the following command to list all the images within my private registry:
curl -s http://internal.private.registry.com/v2/_catalog | jq -r '.repositories[0:2] | to_entries | map( .value )[]'
Output:
centos
containersol/consul-server
I am using the following command to list all the tags of a particular image within my private registry:
curl -s GET http://internal.private.registry.com/v2/centos/tags/list | jq -r '.tags | to_entries | map( .value )[]'
Output:
6.6
6
7.1.1503
Now, I am trying to list all tags for every image present in catalog using the following bash script:
#!/bin/sh
image_name=$(curl -s http://internal.private.registry.com/v2/_catalog | jq -r '.repositories[0:2] | to_entries | map( .value )[]')
while read -r line; do
${line}$_image_taglist=$(curl -s GET http://internal.private.registry.com/v2/cybs/${line}/tags/list | jq -r '.tags | to_entries | map( .value )[]')
while read -r tag; do
echo "$tag"
done <<< "${line}$_image_taglist"
done <<< "$image_name"
However, I am getting the following error while executing ${line}$_image_taglist=$(curl -s GET http://internal.private.registry.com/v2/cybs/${line}/tags/list | jq -r '.tags | to_entries | map( .value )[]') :
jq: error (at <stdin>:1): null (null) has no keys
Also, please note that I am expecting that dynamic variable ${line}$_image_taglist should resolve to centos_image_taglist and containersol/consul-server_image_taglist
jq: error (at :1): null (null) has no keys
Is caused by . being null. In this case . refers to stdin which means that you get no response from http://internal.private.registry.com/v2/cybs/${line}/tags/list.
You cannot define dynamic named variables like this: ${line}$_image_taglist. You should use an associative array if your bash version supports it:
declare -A arr
arr["$line"]=$(curl ... | jq ...)
But in your case I can't really see why you need it, since you are just iteration over it. Why not use a pipe or process substitution?
while read -r tag; do
echo "$tag"
done < <(curl ... | jq ...)