using jq to assign multiple output variables - bash

I am trying to use jq to parse information from the TVDB api. I need to pull a couple of fields and assign the values to variables that I can continue to use in my bash script. I know I can easily assign the output to one variable through bash with variable="$(command)" but I need the output to produce multiple variables and I don't want to make to use multiple commands.
I read this documentation:
https://stedolan.github.io/jq/manual/v1.5/#Advancedfeatures
but I don't know if this relevant to what I am trying to do.
jq '.data' produces the following output:
[
{
"absoluteNumber": 51,
"airedEpisodeNumber": 6,
"airedSeason": 4,
"airedSeasonID": 680431,
"dvdEpisodeNumber": 6,
"dvdSeason": 4,
"episodeName": "We Will Rise",
"firstAired": "2017-03-15",
"id": 5939660,
"language": {
"episodeName": "en",
"overview": "en"
},
"lastUpdated": 1490769062,
"overview": "Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team."
}
]
I tried jq '.data | {episodeName:$name}' and jq '.data | .episodeName as $name' just to try and get one working. I don't understand the documentation or even if it's what I'm looking for. Is there a way to do what I am trying to do?

You can use separate variables with read :
read var1 var2 var3 < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : $var1"
echo "name : $var2"
echo "full_name : $var3"
Using array :
read -a arr < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : ${arr[0]}"
echo "name : ${arr[1]}"
echo "full_name : ${arr[2]}"
Also you can split jq output with some character :
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')
Or use an array like :
set -f; IFS='|' data=($(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')); set +f
absoluteNumber, airedEpisodeNumber, episodeName & overview are respectively ${data[0]}, ${data[1]}, ${data[2]}, ${data[3]}. set -f and set +f are used to respectively disable & enable globbing.
For the jq part, all your required fields are mapped and delimited with a '|' character with join("|")
If your are using jq < 1.5, you'll have to convert Number to String with tostring for each Number fields eg:
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber|tostring, .airedEpisodeNumber|tostring, .episodeName, .overview] |
join("|")) | join("\n")')

jq always produces a stream of zero or more values. For example, to produce the two values corresponding to "episodeName" and "id"' you could write:
.data[] | ( .episodeName, .id )
For your purposes, it might be helpful to use the -c command-line option, to ensure each JSON output value is presented on a single line. You might also want to use the -r command-line option, which removes the outermost quotation marks from each output value that is a JSON string.
For further variations, please see the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, e.g. the question:
Q: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?

Experimental conversion of quoted OP input, (tv.dat), to a series of bash variables, (and an array). The jq code is mostly borrowed from here and there, but I don't know how to get jq to unroll an array within an array, so the sed code does that, (that's only good for one level, but so are bash arrays):
jq -r ".[] | to_entries | map(\"DAT_\(.key) \(.value|tostring)\") | .[]" tv.dat |
while read a b ; do echo "${a,,}='$b'" ; done |
sed -e '/{.*}/s/"\([^"]*\)":/[\1]=/g;y/{},/() /' -e "s/='(/=(/;s/)'$/)/"
Output:
dat_absolutenumber='51'
dat_airedepisodenumber='6'
dat_airedseason='4'
dat_airedseasonid='680431'
dat_dvdepisodenumber='6'
dat_dvdseason='4'
dat_episodename='We Will Rise'
dat_firstaired='2017-03-15'
dat_id='5939660'
dat_language=([episodeName]="en" [overview]="en")
dat_lastupdated='1490769062'
dat_overview='Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team.'

Related

Unable to loop through the JSON internal Array having spaces in values using Bash script JQ [duplicate]

Background
I want to be able to pass a json file to WP CLI, to iteratively create posts.
So I thought I could create a JSON file:
[
{
"post_type": "post",
"post_title": "Test",
"post_content": "[leaflet-map][leaflet-marker]",
"post_status": "publish"
},
{
"post_type": "post",
"post_title": "Number 2",
"post_content": "[leaflet-map fitbounds][leaflet-circle]",
"post_status": "publish"
}
]
and iterate the array with jq:
cat posts.json | jq --raw-output .[]
I want to be able to iterate these to execute a similar function:
wp post create \
--post_type=post \
--post_title='Test Map' \
--post_content='[leaflet-map] [leaflet-marker]' \
--post_status='publish'
Is there a way I can do this with jq, or similar?
The closest I've gotten so far is this:
> for i in $(cat posts.json | jq -c .[]); do echo $i; done
But this seems to take issue with the (valid) spaces in the strings. Output:
{"post_type":"post","post_title":"Test","post_content":"[leaflet-map][leaflet-marker]","post_status":"publish"}
{"post_type":"post","post_title":"Number
2","post_content":"[leaflet-map
fitbounds][leaflet-circle]","post_status":"publish"}
Am I way off with this approach, or can it be done?
Use a while to read entire lines, rather than iterating over the words resulting from the command substitution.
while IFS= read -r obj; do
...
done < <(jq -c '.[]' posts.json)
Maybe this would work for you:
Make a bash executable, maybe call it wpfunction.sh
#!/bin/bash
wp post create \
--post_type="$1"\
--post_title="$2" \
--post_content="$3" \
--post_status="$4"
Then run jq on your posts.json and pipe it into xargs
jq -M -c '.[] | [.post_type, .post_title, .post_content, .post_status][]' \
posts.json | xargs -n4 ./wpfunction`
I am experimenting to see how this would handle post_content that contained quotes...
First generate an array of the arguments you wish to pass then convert to a shell compatible form using #sh. Then you could pass to xargs to invoke the command.
$ jq -r '.[] | ["post", "create", (to_entries[] | "--\(.key)=\(.value|tojson)")] | #sh' input.json | xargs wp

Converting JSON response to key value pair using jq

So, I am getting a response from an API that I am calling in a shell script in the following form
[{"id":100000004,"name":"Customs Clearance Requested"},{"id":100000005,"name":"Customs Cleared"},{"id":100000006,"name":"Cargo Loaded to Vessel"}]
I want to create a map out of it that will help me lookup the id's from a name and use it in the shell script. So something like map["Customs Clearance Requested"] would give me 100000004 which I can use further. Can this be done using jq? I am pretty new to shell scripting and jq and got stuck with above thing
json='[{"id":100000004,"name":"Customs Clearance Requested"},{"id":100000005,"name":"Customs Cleared"},{"id":100000006,"name":"Cargo Loaded to Vessel"}]'
declare -A map
while IFS= read -r -d '' name && IFS= read -r -d '' value; do
map[$name]=$value
done < <(jq -j '.[] | "\(.name)\u0000\(.id)\u0000"' <<<"$json")
declare -p map # demo purposes: print the map we created as output
...emits as output:
declare -A map=(["Cargo Loaded to Vessel"]="100000006" ["Customs Clearance Requested"]="100000004" ["Customs Cleared"]="100000005" )
...which you can query exactly as requested:
$ echo "${map['Cargo Loaded to Vessel']}"
100000006
You could use the select function, e.g.:
data='[{"id":100000004,"name":"Customs Clearance Requested"},{"id":100000005,"name":"Customs Cleared"},{"id":100000006,"name":"Cargo Loaded to Vessel"}]'
jq 'map(select(.["name"] == "Customs Clearance Requested"))' <<< $data
It will get all elements which name equals "Customs Clearance Requested", e.g.:
[
{
"id": 100000004,
"name": "Customs Clearance Requested"
}
]
If you want to get the id field:
jq 'map(select(.["name"] == "Customs Clearance Requested")["id"])' <<< $data
This will output:
[
100000004
]
Please note that it will return an array and not a single element because the search does not know how many results will be found.
If you want to generalize this in a shell function, you could write:
function get_id_from_name
{
# $1=name to search for
local filter=$(printf 'map(select(.["name"] == "%s")["id"])' "$1")
jq "$filter"
}
Then call it like that:
get_id_from_name "Customs Clearance Requested" <<< $data
If your data is stored in a file, you could call it this way:
get_id_from_name "Customs Clearance Requested" < /path/to/file.json
The following is very similar to #CharlesDuffy's excellent answer but does not assume that the .name and .id values are NUL-free (i.e., do not have any "\u0000" characters):
declare -A map
while read -r name
do
name=$(sed -e 's/^"//' -e 's/"$//' <<< "$name")
read -r id
map[$name]="$id"
done < <(echo "$json" | jq -c '.[]|.name,.id')
The point is that the -j option is like -r (i.e., produces "raw output"), whereas the -c option produces JSON.
This means that if you don't want the .id values as JSON strings, then the above won't be a solution; also, if the .name values contain double-quotes, then you might want to deal with the occurrences of \".

Looping through json array not working - jq

I have a JSON array conf=
[ { "fraudThreshold": 4, "fraudTTLSec": 60 }, { "fraudThreshold": 44, "fraudTTLSec": 60 } ]
I want to loop through its items. So I have done the following:
for configy in $(echo "${conf}" | jq -r ".[]"); do
echo configy=$configy
done
The results are:-
configy={
configy="fraudThreshold":
configy=4,
configy="fraudTTLSec":
and so on.
It is splitting the string using spaces and giving the results one by one.
Why is bash showing this weird behavior? Is there any solution to this?
Also, it is giving proper values when I do :
configy=$(echo $conf | jq .[-1])
echo configy=$configy
Result:
configy={ "fraudThreshold": 44, "fraudTTLSec": 60 }
In order to loop through the items in the JSON array using bash, you could write:
echo "${conf}" | jq -cr ".[]" |
while read -r configy
do
echo configy="$configy"
done
This yields:
configy={"fraudThreshold":4,"fraudTTLSec":60}
configy={"fraudThreshold":44,"fraudTTLSec":60}
However there is almost surely a better way to achieve your ultimate goal.
echo "${conf}" | jq -car '.[] | "configy=" + tojson'
produces:
configy={"fraudThreshold":4,"fraudTTLSec":60}
configy={"fraudThreshold":44,"fraudTTLSec":60}
for configy in $(echo "${conf}" | jq -r ".[]"); do
It is splitting the string using spaces and giving the results one by one. Why is bash showing this weird behavior?
This behavior is not weird at all. See the Bash Reference Manual: Word Splitting:
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
Is there any solution to this?
Mâtt Frëëman and peak presented working solutions; you can slightly optimize them by replacing echo "${conf}" | with <<<"$conf".

JQ get key based on variable value

I'm trying to create a ohmyzsh function for Salesforce's DX CLI based on Wade Wegner's guide here. In order to get the value I want I need to expand how he is using JQ which I've never heard of before. I get the premise for this use case but I'm struggling with one abstraction point (within the aliasConfig json). Here's my script so far
get_sfdx_defaultusername() {
config="$(cat .sfdx/sfdx-config.json 2> /dev/null)";
globalConfig="$(cat ~/.sfdx/sfdx-config.json)";
aliasConfig="$(cat ~/.sfdx/alias.json)";
defaultusername="$(echo ${config} | jq -r .defaultusername)"
defaultusernamealias="NEED HELP HERE"
globaldefaultusername="$(echo ${globalConfig} | jq -r .defaultusername)"
if [ ! $defaultusernamealias = "null" ]
then
echoString=$echoString$defaultusernamealias"$txtylw (alias)"
elif [ ! $defaultusername = "null" ]
then
echoString=$echoString$defaultusername"$txtylw (local)"
else
echoString=$echoString$globaldefaultusername"$txtylw (global)"
fi
echo $echoString"\n"
}
The alias.json looks like this:
{
"orgs": {
"HubOrg": "myemail#domain.com",
"my-scrath-org": "test-jdj1iflkor4k#mydomain.net"
}
}
Using the ${defaultusername} I know the value in this case to be "test-jdj1iflkor4k#mydomain.net", therefore I need it to set the value of defaultusernamealias to "my-scrath-org"
NOTE: The closest answer I found was this, but unfortunately I still couldn't get what I needed with it.
Congratulations on figuring out how to use to_entries.
One small suggestion is to avoid using shell interpolation to "construct" the jq program. A much better way to achieve the desired goal is to pass in the relevant values on the command-line. In your case, the following would be appropriate:
$ jq --arg username "$defaultusername" '
.orgs | to_entries[] | select(.value == $username ).key'
Another small point is to avoid using echo to send JSON to STDIN. There are several possibilities, including these patterns:
if you are using bash: jq .... <<< "$JSON"
use printf "%s" "$JSON" | jq ...
jq -n --argjson JSON "$JSON" '$JSON | ...'
In your case, the last of these alternatives would look like this:
$ jq --arg username "$defaultusername" --argjson JSON "$aliasConfig" '
$JSON
| .orgs | to_entries[] | select(.value == $username ).key'
I think I got it figured out here:
get_sfdx_defaultusername() {
config="$(cat .sfdx/sfdx-config.json 2> /dev/null)";
globalConfig="$(cat ~/.sfdx/sfdx-config.json)";
aliasConfig="$(cat ~/.sfdx/alias.json)";
defaultusername="$(echo ${config} | jq -r .defaultusername)"
defaultusernamealias="$(echo ${aliasConfig} | jq -r '.orgs | to_entries[] | select(.value =="'$defaultusername'").key' )"
globaldefaultusername="$(echo ${globalConfig} | jq -r .defaultusername)"
if [ ! $defaultusernamealias = "null" ]
then
echoString=$echoString$defaultusernamealias"$txtylw (alias)"
elif [ ! $defaultusername = "null" ]
then
echoString=$echoString$defaultusername"$txtylw (local)"
else
echoString=$echoString$globaldefaultusername"$txtylw (global)"
fi
echo $echoString"\n"
}
This allows me to show my current defaultusername org like so:
In case anyone is interested in using this or contributing to it, I published a github repo here

jq and bash: object construction with --arg is not working

Given the following input:
J='{"a":1,"b":10,"c":100}
{"a":2,"b":20,"c":200}
{"a":3,"b":30,"c":300}'
The command
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{a,b}'
produces
{"a":1,"b":10}
{"a":2,"b":20}
{"a":3,"b":30}
but this command produces unexpected results:
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{$P1}'
{"P1":"a,b"}
{"P1":"a,b"}
{"P1":"a,b"}
How does one get jq to treat an arg string literally?
Using tostring gives an error
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{$P1|tostring}'
jq: error: syntax error, unexpected '|', expecting '}' (Unix shell quoting
issues?) at <top-level>, line 1:
.[]|{$SELECT|tostring}
jq: 1 compile error
SELECT needs to be a variable and not hardcoded in the script.
SELECT needs to be a variable and not hardcoded in the script.
Assuming you want to avoid the risks of "code injection" and that you want the shell variable SELECT to be a simple string such as "a,b", then consider this reduce-free solution along the lines you were attempting:
J='{"a":1,"b":10,"c":100}'
SELECT='a,b'
echo "$J" |
jq -c --arg P1 "$SELECT" '
. as $in | $P1 | split(",") | map( {(.): $in[.]} ) | add'
Output:
{"a":1,"b":10}
If you really want your data to be parsed as syntax...
This is not an appropriate use case for --arg. Instead, substitute into the code:
select='a,b'; jq -c -s '.[]|{'"$select"'}' <<<"$j"
Note that this has all the usual caveats of code injection: If the input is uncontrolled, the output (or other behavior of the script, particularly if jq gains more capable I/O features in the future) should be considered likewise.
If you want to split the literal string into a list of keys...
Here, we take your select_str (of the form a,b), and generate a map: {'a': 'a', 'b': 'b'}; then, we can break each data item into entries, select only the items in the map, and there's our output.
jq --arg select_str "$select" '
($select_str
| split(",")
| reduce .[] as $item ({}; .[$item]=$item)) as $select_map
| with_entries(select($select_map[.key]))' <<<"$j"

Resources