I'm trying to split a json file into various json files. The input (r1.json) looks like :
{
"results" : [
{
content 1
}
,
{
content 2
}
,
{
content n
}
]
}
I'd like the output to be n files : 1.json, 2.json, n.json. Respectively containing {content 1}, {content 2} and {content n}.
I tried :
for i in {0..24}; do cat r1.json | jq '.results[$i]' >> $i.json; done
But I have the following error: error: i is not defined
While the above answers are correct, note that interpolating shell variables in jq scripts is a terrible idea for all but the most trivial of scripts. On any of the solutions provided, replace the following:
jq ".results[$i]"
With the following:
jq --arg i "$i" '.results[$i | tonumber]'
Try
for i in {0..24}; do cat r1.json | jq ".results[$i]" >> $i.json; done
Note that shell variables can't be expanded inside of single-quotes.
IHTH
The single quotes are probably what is messing you up. Bash variables are not expanded in single quotes. You are passing a literal string .results[$i] to jq. Try double quotes instead:
for i in {0..24}; do
cat r1.json | jq ".results[$i]" >> $i.json
done
Related
I want to merge some json in a file with some json generated at runtime. jq seems to have no difficulty if all the files passed to it are here-strings, or files in the system. But if I try to mix the file types it seems the here-strings are ignored, see snippet below:
Two normal files:
bash-4.2# echo '{"key":0}' > zero
bash-4.2# echo '{"key":1}' > one
bash-4.2# jq --slurp add zero one
{
"key": 1
}
Normal file and here-string (only normal file appears in result!):
bash-4.2# jq --slurp add zero <<< '{"key":1}'
{
"key": 0
}
Here-string first, then normal file (only normal file appears in result!):
bash-4.2# jq --slurp add <<< '{"key":0,"anotherkey":2}' one
{
"key": 1
}
Single here-string (works fine):
bash-4.2# jq --slurp add <<< '{"key":0}'
{
"key": 0
}
Two here-strings (works fine): EDIT: Output is misleading, something else is going on here.
bash-4.2# jq --slurp add <<< '{"key":0}' <<< '{"key":1}'
{
"key": 1
}
My suspicion is that jq works just fine and I am ignorant of how bash resolves the here-strings. But, how would I debug this to improve my understanding?
Note: A very easy workaround would be to evaluate my runtime json and produce a file, then merge the two files as above. I really want to know why the bold examples above don't produce what I would expect.
After reading through the comments this is my understanding:
<<< is evaluated by the shell first and redirects stdin. If jq receives no positional arguments after the filter, it reads from stdin. Therefore all these statements are equivalent:
echo "{}" | jq --slurp add
<<< {} jq --slurp add
jq <<< {} --slurp add
jq --slurp <<< {} add
jq --slurp add <<< {}
If jq does receive positional arguments after the filter, it interprets them as filenames. It adheres to the convention of treating - as stdin.
bash-4.2# echo '{"one":1,"two":1}' > first
bash-4.2# echo '{"three":3}' > third
bash-4.2# jq --slurp add first - third <<< '{"two":2}'
{
"one": 1,
"two": 2,
"three": 3
}
The here-string construct simply redirects standard input. You will separately need to tell jq to read standard input if you call it in a way where it receives file name arguments. The de facto standard way to do that is to specify - as the input (pseudo-) filename.
I believe one of your test cases didn't actually work, and just looked like it did because the input data was constructed so as to be a no-op.
One idea would be to use process substitution which, in essence, provides jq with a (temp) file descriptor it can work with.
Using awk to demonstrate the file descriptor idea:
$ awk '{print FILENAME}' <(echo 'abc')
/dev/fd/63
Demonstrating with a few of your examples:
$ jq --slurp add zero <(echo '{"key":1}')
{
"key": 1
}
$ jq --slurp add zero <(echo '{"keyx":1}')
{
"key": 0,
"keyx": 1
}
$ jq --slurp add <(echo '{"key":0,"anotherkey":2}') one
{
"key": 1,
"anotherkey": 2
}
$ jq --slurp add <(echo '{"key":0}') <(echo '{"key":1}')
{
"key": 1
}
$ jq --slurp add <(echo '{"key":0}') <(echo '{"keyx":1}')
{
"key": 0,
"keyx": 1
}
I'm seeing a is not defined at <top-level> when calling jq like so:
jq ".Changes[0].ResourceRecordSet.Name = word-is-here.domain.com" someFile.json
The error repeats for each word separated by a dash in the second side of the replacement. The full error is like
jq: error: word/0 is not defined at <top-level>, line 1:
.Changes[0].ResourceRecordSet.Name = word-is-here.domain.com
I've tried escaping quotes in many different ways but that didn't help. (what I mean by this is doing "'"'" weird stuff, I'm still learning bash so I'm just trowing stuff at the wall until it sticks)
EDIT:
So I'm trying to run this in a bash script, and both side of the = signs are variables such as jq --arg value "$value" --arg key "$key" '$key = $value' "$path" (what I tried after a suggestion)
and got the error:
Invalid path expression with result ".Changes[0].ResourceRecor...
The json I'm using is as such:
{
"Changes": [
{
"Action": "do something",
"ResourceRecordSet": {
"Name": "some name here to replace",
...
}
}
]
}
jq '.Changes[0].ResourceRecordSet.Name = "word-is-here.domain.com"' file.json
Quote the string you are assigning. Or pass it to jq via an argument:
jq --arg foo 'words-here' '.Changes[0].ResourceRecordSet.Name = $foo' file.json
For passing the path to the key you want as an argument, a suggestion from https://github.com/stedolan/jq/issues/1493 might work:
jq --argjson path '["Changes",0,"ResourceRecordSet","Name"]' \
--arg val 'word-is-here.domain.com' \
'getpath($path) = $val' file.json
The problem (or at least the obvious problem) here is evidently the string: word-is-here.domain.com, since jq is interpreting the dash ("-") as an operation ("minus").
Unfortunately, since you haven't given us many clues, it's not completely clear what specifically needs to be changed, but a reasonable guess is that word-is-here.domain.com is intended as a fixed string. If so, you would have to present it as a JSON string. So in a bash or bash-like environment, you could write:
jq '.Changes[0].ResourceRecordSet.Name = "word-is-here.domain.com"' someFile.json
Specifying the LHS path via a shell variable
If the LHS path must be specified by a shell variable, it should if possible be passed in as a JSON array, e.g. using the --argjson command-line option; one can then use an expression of the form setpath($path; $value) to update the path.
If for some reason a solution allowing the LHS to be specified as a jq path is preferred, then shell string-interpolation could be used, though as with any such interpolation, this should be done with care.
I would like to get the values from Json file. Which is working.
JsonFileToTest:
{
"permissions": [
{
"emailid": "test1#test.com",
"rights": "read"
},
{
"emailid": "test2#test.com",
"rights": "read"
}
]
}
readPermissions=($(jq -r '.permissions' JsonFileToTest))
# The command below works perfectly, But when I Put it in a loop, It does not.
#echo ${readPermissions[#]} | jq 'values[].emailid'
for vals in ${readPermissions[#]}
do
# I would like o extract the email id of the user. The loop is not working atm.
echo ${vals[#]} | jq 'values[].emailid'
done
what am I missing here?
thanks
If you really want to do it this way, that might look like:
readarray -t permissions < <(jq -c '.permissions[]' JsonFileToTest)
for permissionSet in "${permissions[#]}"; do
jq -r '.emailid' <<<"$permissionSet"
done
Note that we're telling jq to print one line per item (with -c), and using readarray -t to read each line into an array element (unlike the array=( $(...command...) ) antipattern, which splits not just on newlines but on other whitespace as well, and expands globs in the process).
But there's no reason whatsoever to do any of that. You'll get the exact same result simply running:
jq -r '.permissions[].emailid' JsonFileToTest
test.sh is not replacing test.json parameter values ($input1 and $input2). result.json has same ParameterValue "$input1/solution/$input2.result"
[
{
"ParameterKey": "Project",
"ParameterValue": [ "$input1/solution/$input2.result" ]
}
]
test.sh
#!/bin/bash
input1="test1"
input2="test2"
echo $input1
echo $input2
cat test.json | jq 'map(if .ParameterKey == "Project" then . + {"ParameterValue" : "$input1/solution/$input2.result" } else . end )' > result.json
shell variables in jq scripts should be interpolated or passed as arguments via --arg name value:
jq --arg inp1 "$input1" --arg inp2 "$input2" \
'map(if .ParameterKey == "Project"
then . + {"ParameterValue" : ($inp1 + "/solution/" + $inp2 + ".result") }
else . end)' test.json
The output:
[
{
"ParameterKey": "Project",
"ParameterValue": "test1/solution/test2.result"
}
]
In your jq program, you have quoted "$input1/solution/$input2.result", and therefore it is a JSON string literal, whereas you evidently want string interpolation; you also need to distinguish between the shell variables ($input1 and $input2) on the one hand, and the corresponding jq dollar-variables (which may or may not have the same name) on the other.
Since your shell variables are strings, you could pass them in using the --arg command-line option (e.g. --arg input1 "$input1" if you chose to name the variables in the same way).
You can read up on string interpolation in the jq manual (see https://stedolan.github.io/jq/manual, but note the links at the top for different versions of jq).
There are other ways to achieve the desired results too, but using string interpolation with same-named variables, you'd write:
"\($input1)/solution/\($input2).result"
Notice that the above string is NOT itself literally a JSON string. Only after string interpolation does it become so.
I have a JSON array conf=
[ { "fraudThreshold": 4, "fraudTTLSec": 60 }, { "fraudThreshold": 44, "fraudTTLSec": 60 } ]
I want to loop through its items. So I have done the following:
for configy in $(echo "${conf}" | jq -r ".[]"); do
echo configy=$configy
done
The results are:-
configy={
configy="fraudThreshold":
configy=4,
configy="fraudTTLSec":
and so on.
It is splitting the string using spaces and giving the results one by one.
Why is bash showing this weird behavior? Is there any solution to this?
Also, it is giving proper values when I do :
configy=$(echo $conf | jq .[-1])
echo configy=$configy
Result:
configy={ "fraudThreshold": 44, "fraudTTLSec": 60 }
In order to loop through the items in the JSON array using bash, you could write:
echo "${conf}" | jq -cr ".[]" |
while read -r configy
do
echo configy="$configy"
done
This yields:
configy={"fraudThreshold":4,"fraudTTLSec":60}
configy={"fraudThreshold":44,"fraudTTLSec":60}
However there is almost surely a better way to achieve your ultimate goal.
echo "${conf}" | jq -car '.[] | "configy=" + tojson'
produces:
configy={"fraudThreshold":4,"fraudTTLSec":60}
configy={"fraudThreshold":44,"fraudTTLSec":60}
for configy in $(echo "${conf}" | jq -r ".[]"); do
It is splitting the string using spaces and giving the results one by one. Why is bash showing this weird behavior?
This behavior is not weird at all. See the Bash Reference Manual: Word Splitting:
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
Is there any solution to this?
Mâtt Frëëman and peak presented working solutions; you can slightly optimize them by replacing echo "${conf}" | with <<<"$conf".