I have a json file which i have obtained using curl command and it looks as below.
{
"Storages": [
{
"Creation": "2020-04-21T14:01:54",
"Modified": "2020-04-21T14:01:54",
"Volume": "/dev/null",
"id": 10000,
"version": "20190925-230722"
},
{
"Creation": "2020-04-22T14:01:54",
"Modified": "2020-04-22T14:01:54",
"Volume": "/opt/home",
"id": 10001,
"version": "22a-20190925-230722"
},
{
"Creation": "2020-04-23T14:01:54",
"Modified": "2020-04-23T14:01:54",
"Volume": "/home/abcd",
"id": 10003,
"version": "21c-20190925-230722"
}
]
}
Now I need to loop thorough array and get id and volume values into 2 variables if version startswith 21a. No need to form another json
For educational purposes, here's a jq command that does both the things you want, but in 2 separate steps:
jq -r 'del(.Storages[] | select(.version | startswith("21a") | not))
.Storages[] | {id, version}'
The first part (del(.Storages[] | select(.version | startswith("21a") | not))) filters out the array elements that don't have a version starting with 21a. The second part (.Storages[] | {id, version}) drills and extracts the specific information you need.
You can use startswith builtin function such as
jq -r '.Storages[] | select(.version | startswith("21a")) | {id, Volume}'
Demo
Edit : Assuming the JSON embedded into a file(Storages.json), then you can assign the results into shell variables such as
$ readarray -t vars < <( jq -r '.Storages[] | select(.version|startswith("21a"))| .id, .Volume' Storages.json )
and display those variables as
$ declare -p vars
declare -a vars='([0]="10003" [1]="/home/abcd")'
Related
Hey I am using the next command
aws ec2 describe-volumes --filters Name=status,Values=available | jq '.Volumes[] | { State: .State, VolumeId: .VolumeId, Tags: .Tags}'
I get the next response:
{
"State": "available",
"VolumeId": "vol-03449dadd29f2067f",
"Tags": [
{
"Key": "sre",
"Value": "test"
}
]}
1.I want my response will be in 1 row regarding each volume.
2. I want to check if the volume has the tag "Name" if so I want the value, otherwise the tags are not interest me. How can I manipulate it ?
Use the -c option.
Use a query of the form select(any(Tags[]; CONDITION))
e.g. select(any(Tags[]; .Key == "Name"))
You can abbreviate expressions of the form {foo: .foo} to just {foo}
So, depending on your exact requirements, your invocation of jq could look like:
jq -c '.Volumes[] | {State, VolumeId, Tags} | select(any(.Tags[]; .Key == "Name"))'
I have an issue where I am trying to filter records with a CONTAINS, but it won't accept a variable that has spaces in it. I am including the JSON and the calls. I explain what works and the last one that does not work. I have looked High and Low but I can't make it work. I have seen and tried many (hundreds of ways taking into account the double quotes, escaped, not escaped, with, without, but no luck) can someone take a look and point me to something that might help.
JSON used to test
_metadatadashjson='{ "meta": { "provisionedExternalId": "" }, "dashboard": { "liveNow": false, "panels": [ { "collapsed": false, "title": "Gyrex Thread Count Gauges", "type": "row", "targets": [ { "expr": "jvm_threads_current{instance=\"192.1.50.22:8055\",job=\"prometheus_gyrex\"}", "refId": "B" } ] }, { "datasource": "Prometheus_16_Docker", "targets": [ { "exemplar": true, "expr": "jvm_threads_current{instance=\"10.32.0.4:8055\",job=\"prometheus_gyrex\"}" } ], "title": ".16 : 3279", "type": "gauge" }, { "description": "", "targets": [ { "expr": "jvm_threads_current{instance=\"10.32.0.7:8055\",job=\"prometheus_gyrex\"}", "refId": "B" } ], "title": ".16 : 3288", "type": "graph" }, { "description": "", "targets": [ { "expr": "jvm_threads_current{instance=\"192.168.2.16:3288\",job=\"prometheus_gyrex\"}", "refId": "C" } ], "title": ".16 : 3288", "type": "graph" } ], "version": 55 }}'
Set the string to search for in key "expr"
exprStrSearch="10.32.0.4:8055"
This works returns one record
echo "${_metadatadashjson}" | jq -r --arg EXPRSTRSEARCH "$exprStrSearch" '.dashboard.panels[] | select(.targets[].expr | contains($EXPRSTRSEARCH)) | .targets[].expr'
This works no problem returns two records.
echo "${_metadatadashjson}" | jq -r --arg EXPRSTRSEARCH "$exprStrSearch" '.dashboard.panels[] | select(.targets[].expr | contains("10.32.0.4:8055", "10.32.0.7:8055")) | .targets[].expr'
Change the value to include a space and another string
exprStrSearch="10.32.0.4:8055 10.32.0.7:8055"
Does not work.
echo "${_metadatadashjson}" | jq -r --arg EXPRSTRSEARCH "$exprStrSearch" '.dashboard.panels[] | select(.targets[].expr | contains($EXPRSTRSEARCH)) | .targets[].expr'
None of your data contains "10.32.0.4:8055 10.32.0.7:8055".
You could pass multiple strings to contains(), using a bash array:
strings=("10.32.0.4:8055" "10.32.0.7:8055")
echo "${_metadatadashjson}" |
jq -r --args '.dashboard.panels[] | select(.targets[].expr | contains($ARGS.positional[])) | .targets[].expr' "${strings[#]}"
But contains will evaluate to true for each match. Ie. if one expr contained both strings, it would be selected (and printed) twice.
With test, that won't happen. Here's how you can add the |s between multiple strings, and pass them in a single jq variable (as well as escape all the dots):
strings=("10.32.0.4:8055" "10.32.0.7:8055")
IFS=\|
echo "${_metadatadashjson}" |
jq -r --arg str "${strings[*]//./\\.}" '.dashboard.panels[] | select(.targets[].expr | test($str)) | .targets[].expr'
Both examples print this:
jvm_threads_current{instance="10.32.0.4:8055",job="prometheus_gyrex"}
jvm_threads_current{instance="10.32.0.7:8055",job="prometheus_gyrex"}
Update: I forgot to escape the dots for test. I edited the test example so that all the dots get escaped (with a single backslash). It's regex, so (unescaped) dots will match any character. The contains example matches the strings literally (not regex).
The problem is that the string with the space in it does not in fact occur in the given JSON. It's not too clear what you are trying to do but please note that contains is not symmetric:
"a" | contains("a b")
evaluates to false.
If you intended to write a boolean search criterion, you could use a boolean expression, or use jq's regular expression machinery, e.g.
test("10.32.0.4:8055|10.32.0.7:8055")
or probably even better:
test("\"(10[.]32[.]0[.]4:8055|10[.]32[.]0[.]7:8055)\"")
I've got this json-file:
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"23423565",
"23552352",
"08789089"
]
}
}
I need to iterate over every element of an list (upper_one), and replace it with other value.
I've tried this code:
#/bin/bash
for product in $(cat first-shop.json| jq -r '.shelves.upper_one[]')
do
cat first-shop.json| jq --arg id "$((1 + $RANDOM % 10))" --arg product "$product" -r '.shelves.upper_one[]|select(. == $product)|= $id'
done
But I got this kind of output:
1
23552352
08789089
23423565
10
08789089
23423565
23552352
7
Is it possible to iterate over list with jq, replace values with value from another function (like $id in the code), and print the whole final json with substituted values?
I need this kind of output:
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"1",
"10",
"7"
]
}
}
not just elements of "upper_one" list thrice.
You could try the following script :
#!/usr/bin/env bash
for product in $(jq -r '.shelves.upper_one[]' input.json)
do
id="$((1 + $RANDOM % 10))"
newIds+=("$id")
done
jq '.shelves.upper_one = $ARGS.positional' input.json --args "${newIds[#]}"
IMHO its better to use some scripting language and manipulate objects programmatically. If bash and jq is your only option - this do the job though not nice
$ jq '.shelves.upper_one[] |= (sub("23423565";"1") | sub("23552352";"10") | sub("08789089";"7"))' your.json
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"1",
"10",
"7"
]
}
}
consider conversion to numbers with | tonumber
I have this output variable
OUTPUT=$(echo $ZONE_LIST | jq -r '.response | .data[]')
The Output:
{
"accountId": "xyz",
"addDate": "2020-09-05T10:57:11Z",
"content": "\"MyContent\"",
"id": "MyID",
"priority": null
}
{
"accountId": "xyz",
"addDate": "2020-09-05T06:58:52Z",
"content": "\"MyContent\"",
"id": "MyID",
"priority": null
}
How can I create a loop for this two values?
MyLoop
echo "$content - $id"
done
I tried this, but then I get a loop through every single value
for k in $(echo $ZONE_LIST | jq -r '.response | .data[]'); do
echo $k
done
EDIT 1:
My complete JSON:
{
"errors": [],
"metadata": {
"transactionId": "",
},
"response": {
"data": [
{
"accountId": "xyz",
"addDate": "2020-09-05T10:57:11Z",
"content": "\"abcd\"",
"id": "myID1",
"lastChangeDate": "2020-09-05T10:57:11Z",
},
{
"accountId": "xyz",
"addDate": "2020-09-05T06:58:52Z",
"content": "\"abc\"",
"id": "myID2",
"lastChangeDate": "2020-09-05T07:08:15Z",
}
],
"limit": 10,
"page": 1,
"totalEntries": 2,
},
"status": "success",
"warnings": []
}
Now I need a loop for data, because I need it for a curl
The curl NOW:
curl -s -v -X POST --data '{
"deleteEntries": [
Data_from_json
]
}' https://URL_to_Update 2>/dev/null)
Now I want to create a new variable from my JSON data. My CURL should look like this at the end:
curl -s -v -X POST --data '{
"deleteEntries": [
{
"readID": "myID1",
"date": "2020-09-05T10:57:11Z", <--Value from addDate
"content": "abcd"
},
{
"readID": "myID2",
"date": "2020-09-05T06:58:52Z", <--Value from addDate
"content": "abc"
}
]
}' https://URL_to_Update 2>/dev/null)
Something like:
#!/usr/bin/env bash
while IFS=$'\37' read -r -d '' id content; do
echo "$id" "$content"
done < <(
jq -j '.response | .data[] | .id + "\u001f" + .content + "\u0000"' \
<<<"$ZONE_LIST"
)
jq -j: Forces a raw output from jq.
.id + "\u001f" + .content + "\u0000": Assemble fields delimited by ASCII FS (Hexadecimal 1f or Octal 37), and end record by a null character.
It then becomes easy and reliable to iterate over null delimited records by having read -d '' (null delimiter).
Fields id content are separated by ASCII FS, so just set the Internal Field Separator IFS environment variable to the corresponding octal IFS=$'37' before reading.
The first step is to realize you can turn the set of fields into an array like this using a technique like this:
jq '(.accountId + "," + .addDate)'
Now you can update your bash loop:
for k in $(echo $ZONE_LIST | jq -r '.response | .data[]' | jq '(.content + "," + .id)'); do
echo $k
done
There is probably a way to combine the two jq commands but I don't have your original json data for testing.
UPDATE - inside the loop you can parse the comma-delimited string into separate fields. This are more efficient ways to handle this task but I prefer simplicity.
ID=$(echo $k | cut -d',' -f1)
PRIORITY=$(echo $k | cut -d',' -f2)
echo "ID($ID) PRIORITY($PRIORITY)"
Try this.
for k in $(echo $ZONE_LIST | jq -rc '.response | .data[]'); do
echo $k|jq '.content + " - " + .id' -r
done
im encountering problems using curl within ruby. I want to run a ruby script and it should store a JSON type object like this
{
"remote": [
{"name": "cast-1",
"id": 1212
},
{"name": "cast-1",
"id": 1214
},
{"name": "home-11",
"id": 3212
},
{"name": "cast-3",
"id": 3212
},
{"name": "cast-3",
"id": 3213
},
{"name": "cast-4",
"id": 4211
}
]
}
in a text file. I mapped and grepped the curl input to get the desired entries.
Now when i run the script, my txt file pops out empty.
example of what i tried
#ruby
`curl -m 10 -s -H '[...]' "https://example_with_json_file.com" | jq -r .remote | group_by(.name) | map({ (.[0].name): map(.id) }) | "\(.[])" | grep '3\|4\|11' > file.txt`
foo = []
File.open(file.txt) do |file|
foo = file.readlines
end
[...]
running the above input in my temrinal works. I tried using Kernel.system order system within my ruby example. Nothing worked.
How can i pipe this output into a txt file
{"cast-3":[3212,3213]}
{"cast-4":[4211]}
{"home-11":[3212]}
Or in general, is it possible to run the curl input in a single line, that is executable using ruby?
Always it's better to do stuff in one language. Like:
response = HTTParty.get('https://path/to/json')
JSON.parse(response)
...
If you want mix it with Bash, try:
bar = `curl -m 10 -s -H '[...]' "https://example_with_json_file.com" | jq -r .remote | group_by(.name) | map({ (.[0].name): map(.id) }) | "\(.[])" | grep '3\|4\|11'`
foo = bar.split(/\n/)
File.write('/path/to/file', bar)