Is there a way to instantiate key with JQ? - filter

Is there a way for jq to instantiate a key to output?
Given:
{
"foo": "bizz",
"bar": "buzz",
"bat": [
"somestring"
]
}
I would like to do something like jq '. | {foobar.foo, foobar.bar}'
and have it output:
{
"foobar": {
"foo": "bizz",
"bar": "buzz"
}
}
without having to make the initial foobar dictionary.


Another possibility to consider:
{foobar: del(.bat)}


You don't have to "create a dictionary" first. Simply define your object's structure (.| is a redundant no-op, it can be left out):
jq '{ foobar: { foo, bar } }'

Related

jq: iterate over every element of list and replace it with value

I've got this json-file:
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"23423565",
"23552352",
"08789089"
]
}
}
I need to iterate over every element of an list (upper_one), and replace it with other value.
I've tried this code:
#/bin/bash
for product in $(cat first-shop.json| jq -r '.shelves.upper_one[]')
do
cat first-shop.json| jq --arg id "$((1 + $RANDOM % 10))" --arg product "$product" -r '.shelves.upper_one[]|select(. == $product)|= $id'
done
But I got this kind of output:
1
23552352
08789089
23423565
10
08789089
23423565
23552352
7
Is it possible to iterate over list with jq, replace values with value from another function (like $id in the code), and print the whole final json with substituted values?
I need this kind of output:
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"1",
"10",
"7"
]
}
}
not just elements of "upper_one" list thrice.

You could try the following script :
#!/usr/bin/env bash
for product in $(jq -r '.shelves.upper_one[]' input.json)
do
id="$((1 + $RANDOM % 10))"
newIds+=("$id")
done
jq '.shelves.upper_one = $ARGS.positional' input.json --args "${newIds[#]}"

IMHO its better to use some scripting language and manipulate objects programmatically. If bash and jq is your only option - this do the job though not nice
$ jq '.shelves.upper_one[] |= (sub("23423565";"1") | sub("23552352";"10") | sub("08789089";"7"))' your.json
{
"name": "market",
"type": "grocery",
"shelves": {
"upper_one": [
"1",
"10",
"7"
]
}
}
consider conversion to numbers with | tonumber

jq how to get the last entry in an object

I'm working with jq 1.6 to get the last entry in an object. It should work like this:
data='{ "1": { "a": "1" }, "2": { "a": "2" }, "3": { "a": "3" } }'
result=`echo $data | jq 'myfilter'`
echo $result
{ "3": { "a": "3" } }
I tried these filters:
jq '. | last' # error: Cannot index object with number
How can I tell jq to quote the number?
jq '. | to_entries | last' # { "key": "3", "value": { "a": "3" } }
I guess I could munge this up by concatenating the key and value entries. Is there a simpler way?
The tutorial and the manual didn't help. No joy on SO either.

You can use the following :
jq 'to_entries | [last] | from_entries'
Try it here.
We can't use with_entries(last) because last returns a single element and from_entries requires an array, hence the [...] construct above.

Generate json file with formatting

I have a curl command which generates json output. I want to add a few characters in generated file to be able to process it further.
Command:
curl -sN --negotiate -u foo:bar "http://hostname/db/tbl_name/" >> db.json
This runs under a for loop which runs it for a db and tbl_name combination. Hence it ends up generating a number of json outputs(one for each table) concatenated together without any delimiter.
Output looks like :
{"columns":[{"name":"tbl_id","type":"varchar(50)"},{"name":"cret_timestmp","type":"timestamp"},{"name":"updt_timestmp","type":"timestamp"},{"name":"frst_nm","type":"varchar(50)"},{"name":"last_nm","type":"varchar(50)"},{"name":"acct_num","type":"varchar(15)"},{"name":"r_num","type":"varchar(15)"},{"name":"pid","type":"decimal(15,0)"},{"name":"ami_id","type":"varchar(30)"},{"name":"ssn","type":"varchar(9)"},{"name":"client_id","type":"varchar(30)"},{"name":"client_nm","type":"varchar(100)"},{"name":"info","type":"timestamp"},{"name":"rmx","type":"varchar(10)"},{"name":"id","type":"decimal(12,0)"},{"name":"ingest_timestamp","type":"string"},{"name":"incr_ingest_timestamp","type":"string"}],"database":"db_i","table":"db_tbl"}{"columns":[{"name":"key","type":"varchar(15)"},{"name":"foo_cd","type":"varchar(10)"},{"name":"foo_nm","type":"varchar(56)"},{"name":"tmc_regn_cd","type":"varchar(10)"},{"name":"tmc_mrkt_cd","type":"varchar(20)"},{"name":"mrkt_grp","type":"varchar(30)"},{"name":"ingest_timestamp","type":"string"},{"name":"incr_ingest_timestamp","type":"string"}],"database":"db_i","table":"ss_mv"}{"columns":[{"name":"bar_src_name","type":"string"},{"name":"bar_ent_name","type":"string"},{"name":"from_src","type":"string"},{"name":"reload","type":"string"},{"name":"column_mismatch","type":"string"},{"name":"xx_src_name","type":"string"},{"name":"xx_ent_name","type":"string"}],"database":"db_i","table":"test_table"}
Desired output is to start and end the output with []. Also I want to include "," between the end and beginning where column list starts.
So for ex: if the curl command runs against 3 tables as shown above, then the three generated jsons should be created like :
[{json1},{json2},{json3}]
Number 1,2,3 ...etc corresponds to different tables in curl command running in for loop against a particular db whose json should be created in one file but with desired format.
instead of what I'm currently getting :
{json1}{json2}{json3}
In the output pasted above, JSON 1 is :
{"columns":[{"name":"tbl_id","type":"varchar(50)"},{"name":"cret_timestmp","type":"timestamp"},{"name":"updt_timestmp","type":"timestamp"},{"name":"frst_nm","type":"varchar(50)"},{"name":"last_nm","type":"varchar(50)"},{"name":"acct_num","type":"varchar(15)"},{"name":"r_num","type":"varchar(15)"},{"name":"pid","type":"decimal(15,0)"},{"name":"ami_id","type":"varchar(30)"},{"name":"ssn","type":"varchar(9)"},{"name":"client_id","type":"varchar(30)"},{"name":"client_nm","type":"varchar(100)"},{"name":"info","type":"timestamp"},{"name":"rmx","type":"varchar(10)"},{"name":"id","type":"decimal(12,0)"},{"name":"ingest_timestamp","type":"string"},
{"name":"incr_ingest_timestamp","type":"string"}],"database":"db_i","table":"db_tbl"}
JSON 2 is :
{"columns":[{"name":"key","type":"varchar(15)"},{"name":"foo_cd","type":"varchar(10)"},{"name":"foo_nm","type":"varchar(56)"},{"name":"tmc_regn_cd","type":"varchar(10)"},{"name":"tmc_mrkt_cd","type":"varchar(20)"},{"name":"mrkt_grp","type":"varchar(30)"},{"name":"ingest_timestamp","type":"string"},{"name":"incr_ingest_timestamp","type":"string"}],"database":"db_i","table":"ss_mv"}
JSON 3 is :
{"columns":[{"name":"bar_src_name","type":"string"},{"name":"bar_ent_name","type":"string"},{"name":"from_src","type":"string"},{"name":"reload","type":"string"},{"name":"column_mismatch","type":"string"},{"name":"xx_src_name","type":"string"},{"name":"xx_ent_name","type":"string"}],"database":"db_i","table":"test_table"}
I hope the requirement is clear, thanks in advance, looking to achieve this via bash.

Use jq -s.
--slurp/-s: Instead of running the filter for each JSON object in the input, read the entire input stream into a large array
and run the filter just once.
Here's an example:
$ cat file.json
{ "key": "value1" }
{ "key": "value2" }
{ "key":
"value3"}{"key": "value4"}
$ jq -s < file.json
[
{
"key": "value1"
},
{
"key": "value2"
},
{
"key": "value3"
},
{
"key": "value4"
}
]

I'm not sure if I got it correctly, but I think you are looking for something like
echo "[$(cat *.json | paste -sd ',')]" > result.json
This works by creating a string that starts with [ and ends with ], and in the middle, there are the contents of the json files concatenated (cat) and separated by commas (with the help of paste). That string is echoed and written to a new file.

Presuming input in valid JSONL format (one JSON document per line of input), you can embed a Python script inside your bash script:
slurpjson_py='
import json, sys
json.dump([json.loads(line.strip()) for line in sys.stdin], sys.stdout, indent=4)
sys.stdout.write("\n")
'
slurpjson() { python -c "$slurpjson_py" "$#"; }
If called as:
slurpjson <<EOF
{ "first": "document", "starting": "here" }
{ "second": "document", "ending": "here" }
EOF
...output is correctly:
[
{
"starting": "here",
"first": "document"
},
{
"second": "document",
"ending": "here"
}
]

I managed to achieve this by running curl command and adding a "," with every line break using
sed 's/$/,/'
And then remove the last "," and added first and end [] using :
for i in *; do cat $i | sed '$ s/.$//' | awk '{print "["$0"]"}' > $json_dir/$i; done

Convert values from array in json object using bash shell

I am totaly new to shell..... let me ut the proper use case.
Use case:-
I have written two get method in my shell script, and when a user calls that script I will perform some operation for many id's using a for loop. like below
test_get1(){
value1=//performing some operation and storing it
value2=//performing some operation and storing it
//below line I am converting the o/p of value1 and value2 in json
value=$JQ_TOOL -n --arg key1 "$value1" --arg key2 "$value2" '{"key1":"\($value1)","key2":"\($value2)"}'
}
test_get2(){
arr=(1,2,3)
local arr_values=()
for value in arr
do
// Calling test_get1 for each iteraion of this loop, like below
val=$(test_get1 $value)
//below line will store the values in array
arr_values+=("$val")
done
}
When I am doing echo for the above arr_values, I am getting the below output
Output.
arr_values={
"key1":"value1",
"key2":"value2"
}
{
"key1":"value1",
"key2":"value2"
}
I want to convert the above value in json format like below.
json_value=[
{
"key1":"value1",
"key2":"value2"
},
{
"key1":"value1",
"key2":"value2"
}
]
I tried to do it with JQ, but unable to get the proper result.

Use the slurp option:
jq -s . in.json > out.json
in.json
{
"key1": "value1",
"key2": "value2"
}
{
"key1": "value1",
"key2": "value2"
}
out.json
[
{
"key1": "value1",
"key2": "value2"
}
]
[
{
"key1": "value1",
"key2": "value2"
}
]

1) Your existing "value=" line can be simplified to:
value=$(jq -n --arg key1 "$value1" --arg key2 "$value2" '\
{key1: $value1, key2: $value2}')
because --arg always interprets the provided value as a string, and because jq expressions need not follow all the rules of JSON.
2) From your script, arr_value is a bash array of JSON values. To convert it into a JSON array, you should be able to use an incantation such as:
for r in "${a[#]}" ; do printf "%s" "$r" ; done | jq -s .
3) There is almost surely a much better way to achieve your ultimate goal. Perhaps it would help if you thought about calling jq just once.

piping and redirecting weird result

Considering 2 json files:
fileA.json:
{
"foo": "hey",
"bar": "ola"
}
fileB.json:
{
"foo": "hoy"
}
, executing:
% cat fileA.json fileB.json | json
returns
{
"foo": "hoy",
"bar": "ola"
}
Ok
--
Now why when redirecting stdout to fileB.json I get:
% cat fileA.json fileB.json | json > fileB.json
I get:
{
"foo": "hey",
"bar": "ola"
}
Ie: fileA.json ???
PS: json utility is here: http://trentm.com/json

The shell sets up output redirection, > fileB.json, so it opens and truncates fileB.json before cat has started reading from it. This causes cat to read an empty file. (It might even end up reading partially written output data.)
Never read from and write to the same file in a pipeline. Try something like > fileC.json instead.

Resources