I have a yaml file say sample.yaml with the following structure
items:
- version: v1
data:
file1.json: |-
{
"key1": val1,
"key2": val2
}
- version: v2
data:
file2.json: |-
{
"key3": val3,
"key4": val4
}
In order to achieve the following output by splitting the data into separate files, what commands/operators should I be using in yq4? The documentation is rather confusing.
f1.json:
{
"key1": val1,
"key2": val2
}
f2.json:
{
"key3": val3,
"key4": val4
}
With yq3, I was able to run
files=$(yq read sample.yaml "items[*].data.*" -pp)
for file in $files; do
yq read sample.yaml "$file" > f1.json
done
Any pointers would be greatly appreciated. Thanks!
Using mikefarah/yq v4.14.1+ with from_json and the -oj option to convert to JSON, and the -s option to split up into files
yq -oj -s '"f" + ($index + 1)' '.items[].data[] | from_json' sample.yaml
will produce f1.json:
{
"key1": "val1",
"key2": "val2"
}
and f2.json:
{
"key3": "val3",
"key4": "val4"
}
If you want to use the filenames from the keys instead, you need to employ to_entries to have access to .key and .value. This will produce the files file1.json and file2.json:
yq -oj -s 'parent | .key | sub("\.json$","")' \
'(.items[].data | to_entries)[].value | from_json' sample.yaml
Related
I am trying to update an value for a key based on a select filter. The list contains dictionaries with same key names, but different values. Based on value, I am filtering to select a dictionary and then update the sibling key in it.
The value update works, but the dictionaries move out of list as expected because of .[], but how do I add them back to the list. Or, how can I do it without using .[]?
Input List:
[
{
"key1": "a",
"key2": "b"
},
{
"key1": "c",
"key2": "d"
},
{
"key1": "d",
"key2": "e"
}
]
Command I am running:
jq --arg temp "f" '.[] | select( .key1 == "c").key2 |= $temp' test.json
output:
{
"key1": "a",
"key2": "b"
}
{
"key1": "c",
"key2": "f"
}
{
"key1": "d",
"key2": "e"
}
The objects are not part of list now.
Expected output:
[
{
"key1": "a",
"key2": "b"
},
{
"key1": "c",
"key2": "f"
},
{
"key1": "d",
"key2": "e"
}
]
How can we add the objects back to a list, or do it in-place.
Use map() to keep the original structure:
jq --arg temp "f" 'map(select( .key1 == "c").key2 |= $temp)' test.json
Online demo
You could also just use another pair of parentheses around the left hand side of the update assignment in order to retain the context:
Either way works:
jq --arg temp "f" '(.[] | select(.key1 == "c").key2) |= $temp' test.json
Demo
jq --arg temp "f" '(.[] | select(.key1 == "c")).key2 |= $temp' test.json
Demo
I have a json file which i have obtained using curl command and it looks as below.
{
"Storages": [
{
"Creation": "2020-04-21T14:01:54",
"Modified": "2020-04-21T14:01:54",
"Volume": "/dev/null",
"id": 10000,
"version": "20190925-230722"
},
{
"Creation": "2020-04-22T14:01:54",
"Modified": "2020-04-22T14:01:54",
"Volume": "/opt/home",
"id": 10001,
"version": "22a-20190925-230722"
},
{
"Creation": "2020-04-23T14:01:54",
"Modified": "2020-04-23T14:01:54",
"Volume": "/home/abcd",
"id": 10003,
"version": "21c-20190925-230722"
}
]
}
Now I need to loop thorough array and get id and volume values into 2 variables if version startswith 21a. No need to form another json
For educational purposes, here's a jq command that does both the things you want, but in 2 separate steps:
jq -r 'del(.Storages[] | select(.version | startswith("21a") | not))
.Storages[] | {id, version}'
The first part (del(.Storages[] | select(.version | startswith("21a") | not))) filters out the array elements that don't have a version starting with 21a. The second part (.Storages[] | {id, version}) drills and extracts the specific information you need.
You can use startswith builtin function such as
jq -r '.Storages[] | select(.version | startswith("21a")) | {id, Volume}'
Demo
Edit : Assuming the JSON embedded into a file(Storages.json), then you can assign the results into shell variables such as
$ readarray -t vars < <( jq -r '.Storages[] | select(.version|startswith("21a"))| .id, .Volume' Storages.json )
and display those variables as
$ declare -p vars
declare -a vars='([0]="10003" [1]="/home/abcd")'
I have this json which I get from an API.
{
"result": {
"key1": "val1",
"key2": "val2",
"key3": "val3"
}
}
I have the following shell script which creates headers for table in a text file. I want to extract key values from the above result object and put in the same text file where keys should go under KEYS and values under VALUES in the table. I am new to jq and shell and struggling to achieve this.
echo "%table"
echo -e "KEYS\tVALUES" > outputfile.txt
KEYVALS=$(curl -uuser:password
"http://localhost:8080/customapi")
# here I want to split the key values using jq and write to the outputfile.txt
cat outputfile.txt
Outcome I am expecting is:
KEYS VALUES
key1 val1
key2 val2
key3 val3
How can I achieve this?
The key is to convert .result to an array of key/value pairs using to_entries, then outputing a set of strings (created using string interpolation) in raw mode.
% cat tmp.json
{
"result": {
"key1": "val1",
"key2": "val2",
"key3": "val3"
}
}
% jq -r '{"KEYS": "VALUES"} + .result | to_entries[] | "\(.key)\t\(.value)"' tmp.json
KEYS VALUES
key1 val1
key2 val2
key3 val3
I added the header to the input before conversion to the key/value list.
by adding the column call at the end the alignment will work for longer values as well ...
Note the usage of the # char as token separator ... of course if your data contains it this will not work ...
aws cognito-idp list-user-pools --max-results 20 | \
jq -r '.UserPools[]|to_entries[]|select (.key == "Name")|("\(.key):#\(.value)")'| column -t -s'#'
output
Name: corp_stg_user_pool
Name: corp_dev_user_pool
I have the following jq code snippet:
https://jqplay.org/s/QzOttRHoz1
I want to loop each element from the result array using bash such as the pseudo code shows:
#!/bin/bash
foreach result
print "My name is {name}, I'm {age} years old"
print "--"
The result would be:
My name is A, I'm 1 years old.
---
My name is B, I'm 2 years old.
---
My name is C, I'm 3 years old.
---
Of course this is a trivial example just to clarify that my goal is to manipulate each array from the jq result individually.
Any suggestions on how to write the pseudo code into valid bash statements?
Saving the json:
{
"Names": [
{ "Name": "A", "Age": "1" },
{ "Name": "B", "Age": "2" },
{ "Name": "C", "Age": "3" }
]
}
as /tmp/input.txt I can run:
</tmp/input.txt jq --raw-output 'foreach .Names[] as $name ([];[];$name | .Name, .Age )' \
| while read -r name && read -r age; do
printf "My name is %s, I'm %d years old.\n" "$name" "$age";
printf -- "--\n";
done
The --raw-output with | .Name, .Age just prints two lines per .Names array member, one with name and another with age. Then I read two lines at a time with while read && read and use that to loop through them.
If you rather have:
["A","1"]
["B","2"]
["C","3"]
that's sad, the best would be to write a full parser that would take strings like "\"" into account. Anyway then you can:
</tmp/input2.txt sed 's/^\[//;s/\]$//;' \
| while IFS=, read name age; do
name=${name%\"};
name=${name#\"};
age=${age%\"};
age=${age#\"};
printf "My name is %s, I'm %d years old.\n" "$name" "$age";
printf -- "--\n";
done
The first sed removed the leading and enclosing [ and ] in each line. Then I read two strings separated by , (so vars like "a,b","c,d" will be read incorrectly). Then these two strings are stripped of leading and enclosing ". Then the usuall printf is used to output the result.
I have a written a simple script to achieve what you need:
My Json file test.json which is similar to your snippet:
{
"Names": [
{ "Name": "A", "Age": "1" },
{ "Name": "B", "Age": "2" },
{ "Name": "C", "Age": "3" }
]
}
My script:
#!/bin/bash
for i in $(cat test.json | jq -r '.Names[] | #base64'); do
_jq() {
echo ${i} | base64 --decode | jq -r ${1}
}
echo "My Name is $(_jq '.Name'), I'm $(_jq '.Age') years old"
done
Note that foreach .Names[] as $name ([];[];$name | .Name, .Age )
can be simplified to:
.Names[] | ( .Name, .Age )
or even in this specific case to:
.Names[][]
or for that matter to:
.[][][]
The important point, however, is that foreach is not needed to achieve simple iteration.
I am totaly new to shell..... let me ut the proper use case.
Use case:-
I have written two get method in my shell script, and when a user calls that script I will perform some operation for many id's using a for loop. like below
test_get1(){
value1=//performing some operation and storing it
value2=//performing some operation and storing it
//below line I am converting the o/p of value1 and value2 in json
value=$JQ_TOOL -n --arg key1 "$value1" --arg key2 "$value2" '{"key1":"\($value1)","key2":"\($value2)"}'
}
test_get2(){
arr=(1,2,3)
local arr_values=()
for value in arr
do
// Calling test_get1 for each iteraion of this loop, like below
val=$(test_get1 $value)
//below line will store the values in array
arr_values+=("$val")
done
}
When I am doing echo for the above arr_values, I am getting the below output
Output.
arr_values={
"key1":"value1",
"key2":"value2"
}
{
"key1":"value1",
"key2":"value2"
}
I want to convert the above value in json format like below.
json_value=[
{
"key1":"value1",
"key2":"value2"
},
{
"key1":"value1",
"key2":"value2"
}
]
I tried to do it with JQ, but unable to get the proper result.
Use the slurp option:
jq -s . in.json > out.json
in.json
{
"key1": "value1",
"key2": "value2"
}
{
"key1": "value1",
"key2": "value2"
}
out.json
[
{
"key1": "value1",
"key2": "value2"
}
]
[
{
"key1": "value1",
"key2": "value2"
}
]
1) Your existing "value=" line can be simplified to:
value=$(jq -n --arg key1 "$value1" --arg key2 "$value2" '\
{key1: $value1, key2: $value2}')
because --arg always interprets the provided value as a string, and because jq expressions need not follow all the rules of JSON.
2) From your script, arr_value is a bash array of JSON values. To convert it into a JSON array, you should be able to use an incantation such as:
for r in "${a[#]}" ; do printf "%s" "$r" ; done | jq -s .
3) There is almost surely a much better way to achieve your ultimate goal. Perhaps it would help if you thought about calling jq just once.