Why can't jq's slurp handle a combination of here-strings and other files? - bash

I want to merge some json in a file with some json generated at runtime. jq seems to have no difficulty if all the files passed to it are here-strings, or files in the system. But if I try to mix the file types it seems the here-strings are ignored, see snippet below:
Two normal files:
bash-4.2# echo '{"key":0}' > zero
bash-4.2# echo '{"key":1}' > one
bash-4.2# jq --slurp add zero one
{
"key": 1
}
Normal file and here-string (only normal file appears in result!):
bash-4.2# jq --slurp add zero <<< '{"key":1}'
{
"key": 0
}
Here-string first, then normal file (only normal file appears in result!):
bash-4.2# jq --slurp add <<< '{"key":0,"anotherkey":2}' one
{
"key": 1
}
Single here-string (works fine):
bash-4.2# jq --slurp add <<< '{"key":0}'
{
"key": 0
}
Two here-strings (works fine): EDIT: Output is misleading, something else is going on here.
bash-4.2# jq --slurp add <<< '{"key":0}' <<< '{"key":1}'
{
"key": 1
}
My suspicion is that jq works just fine and I am ignorant of how bash resolves the here-strings. But, how would I debug this to improve my understanding?
Note: A very easy workaround would be to evaluate my runtime json and produce a file, then merge the two files as above. I really want to know why the bold examples above don't produce what I would expect.

After reading through the comments this is my understanding:
<<< is evaluated by the shell first and redirects stdin. If jq receives no positional arguments after the filter, it reads from stdin. Therefore all these statements are equivalent:
echo "{}" | jq --slurp add
<<< {} jq --slurp add
jq <<< {} --slurp add
jq --slurp <<< {} add
jq --slurp add <<< {}
If jq does receive positional arguments after the filter, it interprets them as filenames. It adheres to the convention of treating - as stdin.
bash-4.2# echo '{"one":1,"two":1}' > first
bash-4.2# echo '{"three":3}' > third
bash-4.2# jq --slurp add first - third <<< '{"two":2}'
{
"one": 1,
"two": 2,
"three": 3
}

The here-string construct simply redirects standard input. You will separately need to tell jq to read standard input if you call it in a way where it receives file name arguments. The de facto standard way to do that is to specify - as the input (pseudo-) filename.
I believe one of your test cases didn't actually work, and just looked like it did because the input data was constructed so as to be a no-op.

One idea would be to use process substitution which, in essence, provides jq with a (temp) file descriptor it can work with.
Using awk to demonstrate the file descriptor idea:
$ awk '{print FILENAME}' <(echo 'abc')
/dev/fd/63
Demonstrating with a few of your examples:
$ jq --slurp add zero <(echo '{"key":1}')
{
"key": 1
}
$ jq --slurp add zero <(echo '{"keyx":1}')
{
"key": 0,
"keyx": 1
}
$ jq --slurp add <(echo '{"key":0,"anotherkey":2}') one
{
"key": 1,
"anotherkey": 2
}
$ jq --slurp add <(echo '{"key":0}') <(echo '{"key":1}')
{
"key": 1
}
$ jq --slurp add <(echo '{"key":0}') <(echo '{"keyx":1}')
{
"key": 0,
"keyx": 1
}

Related

bash loop error : Get JSON Object by property with jq / bash

I would like to get the values from Json file. Which is working.
JsonFileToTest:
{
"permissions": [
{
"emailid": "test1#test.com",
"rights": "read"
},
{
"emailid": "test2#test.com",
"rights": "read"
}
]
}
readPermissions=($(jq -r '.permissions' JsonFileToTest))
# The command below works perfectly, But when I Put it in a loop, It does not.
#echo ${readPermissions[#]} | jq 'values[].emailid'
for vals in ${readPermissions[#]}
do
# I would like o extract the email id of the user. The loop is not working atm.
echo ${vals[#]} | jq 'values[].emailid'
done
what am I missing here?
thanks
If you really want to do it this way, that might look like:
readarray -t permissions < <(jq -c '.permissions[]' JsonFileToTest)
for permissionSet in "${permissions[#]}"; do
jq -r '.emailid' <<<"$permissionSet"
done
Note that we're telling jq to print one line per item (with -c), and using readarray -t to read each line into an array element (unlike the array=( $(...command...) ) antipattern, which splits not just on newlines but on other whitespace as well, and expands globs in the process).
But there's no reason whatsoever to do any of that. You'll get the exact same result simply running:
jq -r '.permissions[].emailid' JsonFileToTest

jq with multiple inputs from different sources

How can we mix different input sources when using jq ?
For a specific usecase, I'd like to add some data from a file into a feed that was pipe in stdout.
$ echo '[{"a": 1}]' > /tmp/a1
$ echo '[{"a": 2}]' > /tmp/a2
$ jq --slurp '.[0] + .[1]' /tmp/a1 /tmp/a2
[
{
"a": 1
},
{
"a": 2
}
]
$ cat /tmp/a1 | jq --slurp '.[0] + .[1]' /tmp/a2 # Expecting the same result
[
{
"a": 2
}
]
As you can see, the last command didn't interpret the piped data.
Right now, I'm forced to save the output from the first operation into a temporary file, so that I can do the jq merging operation, before sending it back to the network. Having a single stream would be much more efficient
I'd like to add some data from a file into a feed that was pipe in stdout.
There are various ways to do this, depending on the shell and also the version of jq you are using.
Assuming your jq supports the --argfile option, you might find that quite congenial:
cat /tmp/a1 | jq --argfile a2 /tmp/a2 '. + $a2'
Here is another variation that suggests some of the other possibilities:
jq -n --argfile a1 <(cat /tmp/a1) --argfile a2 <(cat /tmp/a2) '$a1 + $a2'
More interestingly:
(cat /tmp/a1 ; cat /tmp/a2) | jq '. + input'
You might also wish to consider using the --slurpfile option instead of --argfile, but note that --slurpfile always "slurps" the file.
And finally an approach that should work for every version of jq:
jq -s '.[0] + .[1]' <(cat /tmp/a1) /tmp/a2
In general, though, it's best to avoid the -s option.
A note on slurping
If you compare the outputs produced by:
echo '1 2' |
jq -s --debug-dump-disasm --debug-trace '.[0], .[1]'
and
echo '1 2' |
jq --debug-dump-disasm --debug-trace '., input'
you'll notice the former has to PUSHK_UNDER to store the entire array [1,2],
whereas the latter program just reads the two inputs separately.
In the first program, the memory for the array cannot be freed until after
all the pointers into it have been processed, whereas in the second program,
the memory for . can be freed after the first RET.
You could do this, where cat forwards its stdin followed by a2:
<GENERATE a1> | cat - /tmp/a2 | jq --slurp '.[0] + .[1]'
Or this, which is a compound statement passing the results of two separate commands into a pipe:
{ <GENERATE a1> ; cat /tmp/a2; } | jq --slurp '.[0] + .[1]'
Take care to have spaces beside the curly braces and to have a semi-colon before the final one.
Completing peak's answer, you actually don't need redirections:
jq -n --argfile a1 /tmp/a1 --argfile a2 /tmp/a2 '$a1 + $a2'

using jq to assign multiple output variables

I am trying to use jq to parse information from the TVDB api. I need to pull a couple of fields and assign the values to variables that I can continue to use in my bash script. I know I can easily assign the output to one variable through bash with variable="$(command)" but I need the output to produce multiple variables and I don't want to make to use multiple commands.
I read this documentation:
https://stedolan.github.io/jq/manual/v1.5/#Advancedfeatures
but I don't know if this relevant to what I am trying to do.
jq '.data' produces the following output:
[
{
"absoluteNumber": 51,
"airedEpisodeNumber": 6,
"airedSeason": 4,
"airedSeasonID": 680431,
"dvdEpisodeNumber": 6,
"dvdSeason": 4,
"episodeName": "We Will Rise",
"firstAired": "2017-03-15",
"id": 5939660,
"language": {
"episodeName": "en",
"overview": "en"
},
"lastUpdated": 1490769062,
"overview": "Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team."
}
]
I tried jq '.data | {episodeName:$name}' and jq '.data | .episodeName as $name' just to try and get one working. I don't understand the documentation or even if it's what I'm looking for. Is there a way to do what I am trying to do?
You can use separate variables with read :
read var1 var2 var3 < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : $var1"
echo "name : $var2"
echo "full_name : $var3"
Using array :
read -a arr < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : ${arr[0]}"
echo "name : ${arr[1]}"
echo "full_name : ${arr[2]}"
Also you can split jq output with some character :
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')
Or use an array like :
set -f; IFS='|' data=($(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')); set +f
absoluteNumber, airedEpisodeNumber, episodeName & overview are respectively ${data[0]}, ${data[1]}, ${data[2]}, ${data[3]}. set -f and set +f are used to respectively disable & enable globbing.
For the jq part, all your required fields are mapped and delimited with a '|' character with join("|")
If your are using jq < 1.5, you'll have to convert Number to String with tostring for each Number fields eg:
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber|tostring, .airedEpisodeNumber|tostring, .episodeName, .overview] |
join("|")) | join("\n")')
jq always produces a stream of zero or more values. For example, to produce the two values corresponding to "episodeName" and "id"' you could write:
.data[] | ( .episodeName, .id )
For your purposes, it might be helpful to use the -c command-line option, to ensure each JSON output value is presented on a single line. You might also want to use the -r command-line option, which removes the outermost quotation marks from each output value that is a JSON string.
For further variations, please see the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, e.g. the question:
Q: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
Experimental conversion of quoted OP input, (tv.dat), to a series of bash variables, (and an array). The jq code is mostly borrowed from here and there, but I don't know how to get jq to unroll an array within an array, so the sed code does that, (that's only good for one level, but so are bash arrays):
jq -r ".[] | to_entries | map(\"DAT_\(.key) \(.value|tostring)\") | .[]" tv.dat |
while read a b ; do echo "${a,,}='$b'" ; done |
sed -e '/{.*}/s/"\([^"]*\)":/[\1]=/g;y/{},/() /' -e "s/='(/=(/;s/)'$/)/"
Output:
dat_absolutenumber='51'
dat_airedepisodenumber='6'
dat_airedseason='4'
dat_airedseasonid='680431'
dat_dvdepisodenumber='6'
dat_dvdseason='4'
dat_episodename='We Will Rise'
dat_firstaired='2017-03-15'
dat_id='5939660'
dat_language=([episodeName]="en" [overview]="en")
dat_lastupdated='1490769062'
dat_overview='Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team.'

How to create for-loops with jq in bash

I'm trying to split a json file into various json files. The input (r1.json) looks like :
{
"results" : [
{
content 1
}
,
{
content 2
}
,
{
content n
}
]
}
I'd like the output to be n files : 1.json, 2.json, n.json. Respectively containing {content 1}, {content 2} and {content n}.
I tried :
for i in {0..24}; do cat r1.json | jq '.results[$i]' >> $i.json; done
But I have the following error: error: i is not defined
While the above answers are correct, note that interpolating shell variables in jq scripts is a terrible idea for all but the most trivial of scripts. On any of the solutions provided, replace the following:
jq ".results[$i]"
With the following:
jq --arg i "$i" '.results[$i | tonumber]'
Try
for i in {0..24}; do cat r1.json | jq ".results[$i]" >> $i.json; done
Note that shell variables can't be expanded inside of single-quotes.
IHTH
The single quotes are probably what is messing you up. Bash variables are not expanded in single quotes. You are passing a literal string .results[$i] to jq. Try double quotes instead:
for i in {0..24}; do
cat r1.json | jq ".results[$i]" >> $i.json
done

converting lines to json in bash

I would like to convert a list into JSON array. I'm looking at jq for this but the examples are mostly about parsing JSON (not creating it). It would be nice to know proper escaping will occur. My list is single line elements so the new line will probably be the best delimiter.
I was also trying to convert a bunch of lines into a JSON array, and was at a standstill until I realized that -s was the only way I could handle more than one line at a time in the jq expression, even if that meant I'd have to parse the newlines manually.
jq -R -s -c 'split("\n")' < just_lines.txt
-R to read raw input
-s to read all input as a single string
-c to not pretty print the output
Easy peasy.
Edit: I'm on jq ≥ 1.4, which is apparently when the split built-in was introduced.
--raw-input, then --slurp
Just summarizing what the others have said in a hopefully quicker to understand form:
cat /etc/hosts | jq --raw-input . | jq --slurp .
will return you:
[
"fe00::0 ip6-localnet",
"ff00::0 ip6-mcastprefix",
"ff02::1 ip6-allnodes",
"ff02::2 ip6-allrouters"
]
Explanation
--raw-input/-R:
Don´t parse the input as JSON. Instead, each line of text is passed
to the filter as a string. If combined with --slurp, then the
entire input is passed to the filter as a single long string.
--slurp/-s:
Instead of running the filter for each JSON object in the input,
read the entire input stream into a large array and run the filter
just once.
You can also use jq -R . to format each line as a JSON string and then jq -s (--slurp) to create an array for the input lines after parsing them as JSON:
$ printf %s\\n aa bb|jq -R .|jq -s .
[
"aa",
"bb"
]
The method in chbrown's answer adds an empty element to the end if the input ends with a linefeed, but you can use printf %s "$(cat)" to remove trailing linefeeds:
$ printf %s\\n aa bb|jq -R -s 'split("\n")'
[
"aa",
"bb",
""
]
$ printf %s\\n aa bb|printf %s "$(cat)"|jq -R -s 'split("\n")'
[
"aa",
"bb"
]
If the input lines don't contain ASCII control characters (which have to be escaped in strings in valid JSON), you can use sed:
$ printf %s\\n aa bb|sed 's/["\]/\\&/g;s/.*/"&"/;1s/^/[/;$s/$/]/;$!s/$/,/'
["aa",
"bb"]
Update: If your jq has inputs you can simply write:
jq -nR [inputs] /etc/hosts
to produce a JSON array of strings. This avoids having to read the text file as a whole.
I found in the man page for jq and through experimentation what seems to me to be a simpler answer.
$ cat test_file.txt | jq -Rsc '. / "\n" - [""]'
["aa","bb"]
The -R is to read without trying to parse json, the -s says to read all of the input as one string, and the -c is for one-line output - not necessary, but it's what I was looking for.
Then in the string I pass to jq, the '.' says take the input as it is. The '/ \n' says to divide the string (split it) on newlines. The '- [""]' says to remove from the resulting array any empty strings (resulting from an extra newline at the end).
It's one line and without any complicated constructs, using just simple built in jq features.

Resources