Serializing and de-serializing an associative array in bash using jq - bash

I have some bash code that serializes and de-serializes a single dimensional associative array in bash using jq. It does what I want for now, but I have two issues.
The first issue is, this code feels really klunky. Especially the serialization part. Is there a better way to do this? Either with jq or some other way?
The second issue is, I can deserialize nested data (.e.g, with {"data":{...}}) but I can't figure out how to wrap the output in the same nested structure. How can I recreate the original structure?
Edit: Clarification. What I want to be able to do is, with the commented json, json='{"data": {"one": "1", "two": "2", "three": "3"}}' in the example code and have the final result of
json='{"data": {"four": "4", "one": "100", "two": "2"}} dumped.
I can read in the 'data' structure and assign the key/values correctly, but I'm not having any luck in figuring out how to embed the {"four": ...} construct into the "data": {...} object.
Edit 2: The answer to my second issue, in combination with peak's answer is the following:
for key in "${!aaname[#]}"; do
printf '%s\n%s\n' "$key" "${aaname[$key]}"
done | jq -SRn '.data = ([inputs | {(.): input}] | add)'
The code is:
#!/bin/bash
#json='{"data": {"one": "1", "two": "2", "three": "3"}}'
json='{"one": "1", "two": "2", "three": "3"}'
#------------------------------------------------------------------------------
# De-serialize data
declare -A aaname
while IFS='=' read -r key value; do
aaname["$key"]="$value"
done < <(echo "$json" | jq -r '. | to_entries | .[] | .key + "=" + .value ')
#done < <(echo "$json" | jq -r '.data | to_entries | .[] | .key + "=" + .value ')
#------------------------------------------------------------------------------
# Manipulate data
# Change existing value ...
aaname['one']='100'
# Add element ...
aaname['four']='4'
# Remove element ...
unset aaname['three']
#------------------------------------------------------------------------------
# Serialize data
# Why can't I use ${#aaname[#]} in ((...))?
total="${#aaname[#]}"
count=0
{
printf '['
for key in "${!aaname[#]}"; do
printf '{"key": "%s", "value": "%s"}' "$key" "${aaname[$key]}"
((++count < total)) && printf ','
done
printf ']'
#}
#} | jq -S ' . | "data{" + from_entries + "}"'
} | jq -S ' . | from_entries'
# gives
#
#{
# "four": "4",
# "one": "100",
# "two": "2"
#}

It would be less klunky, and perhaps a bit more robust, if instead of:
jq -r '. | to_entries | .[] | .key + "=" + .value ')
you had:
jq -r 'to_entries[] | "\(.key)=\(.value)"'
And similarly you could replace the for loop used to create the JSON object with something like:
for key in "${!aaname[#]}"; do
printf "%s\n" "$key"
printf "%s\n" "${aaname[$key]}"
done | jq -Rn '[inputs | { (.): input}] | add'
Regarding the second issue, I'm afraid your question isn't so clear to me.
What format are you expecting for the non-JSON representation?
How generic a serialization/deserialization solution are you expecting?
In this connection, you might like to look at the output of jq --stream . <<< "$json"
for various JSON texts.

Related

jq keys as concenated string

I have a source file that looks like
{
"admin-user" : {
"index.md": "Index",
"user_profil.md": "User Profil"
}
}
By help of bash and jq I like to concat a string of the second level keys index.md and user_profile.md. Further I like replace .md with .html (key admin-user is unknown an can change)
This is where I hang:
KEYS=$(cat test.json | jq -r '.[]|keys')
concat_string=""
for i in $KEYS
do
md_name=${i/.md/.html}
concat_string="$concat_string$md_name"
done
echo $concat_string
Result:
["index.html","user_profil.html"]
So the result is an array. How can I concat a string with blanks between strings?
All of it can be done from within jq:
map(keys_unsorted[] | sub("\\.md$"; ".html"))
Demo
Alternatively, you can use to_entries to access each .key:
map(to_entries[].key | sub("\\.md$"; ".html"))
Demo
Both will give you an array of strings
["index.html","user_profil.html"]
which you can then -still in jq- concatenate using join and a glue character:
jq -r 'map(keys_unsorted[] | sub("\\.md$"; ".html")) | join(" ")' test.json
Demo
or using the alternative approach:
jq -r 'map(to_entries[].key | sub("\\.md$"; ".html")) | join(" ")' test.json
Demo
Note: Using keys instead of keys_unsorted (as you did in your attempt) will additionally sort the keys.

Shell Script error: "For loop" is not throwing the expected output

I have json file which extract the color value from the file. For some reason, it only fetch only one block of code & for the rest it throws error.
snippet
#!/bin/bash
clear
echo "Add the figma json file path"
read path
figma_json="$(echo -e "${path}" | tr -d '[:space:]')"
echo "*****************************************"
color_values=$(cat $figma_json | jq -r '.color')
color_keys=$(cat $figma_json | jq -r '.color | keys' |sed 's,^ *,,; s, *$,,'| tr -s ' ' | tr ' ' '_')
echo $color_keys
for c_key in $color_keys
do
echo "key string: $c_key"
echo $color_values | jq ".$c_key.value"
echo "*********************************************"
done
Output
trimmed string: "gray1",
{
"description": "",
"type": "color",
"value": "#333333ff",
"extensions": {
"org.lukasoppermann.figmaDesignTokens": {
"styleId": "S:0b49d19e868ec919fac01ec377bb989174094d7e,",
"exportKey": "color"
}
}
}
null
*********************************************
trimmed string: "gray2" //Expected output
"#333333ff"
*********************************************
If we look at the second output it prints the hex value of gray2 which is the expected output
Please use the follow link to get the json file
link
It's quite unclear what you are aiming at, but here's one way how you would read from a JSON file using just one call to jq, and most probably without the need to employ sed or tr. The selection as well as the formatting can easily be adjusted to your liking.
jq -r '.color | to_entries[] | "\(.key): \(.value.value)"' "$figma_json"
gray1: #333333ff
gray2: #4f4f4fff
gray3: #828282ff
gray4: #bdbdbdff
gray5: #e0e0e0ff
gray6: #f2f2f2ff
red: #eb5757ff
orange: #f2994aff
yellow: #f2c94cff
green1: #219653ff
green2: #27ae60ff
green3: #6fcf97ff
blue1: #2f80edff
blue2: #2d9cdbff
blue3: #56ccf2ff
purple1: #9b51e0ff
purple2: #bb6bd9ff
Demo

JQ statement to build Json from csv

I have a CSV file that I want to convert to a JSON file with the quotes from the CSV removed using JQ in a shell script.
Here is the CSV named input.csv:
1,"SC1","Leeds"
2,"SC2","Barnsley"
Here is the JQ extract:
jq --slurp --raw-input --raw-output \
'split("\n") | .[1:] | map(split(",")) |
map({
"ListElementCode": .[0],
"ListElement": "\(.[1]) \(.[2])
})' \
input.csv > output.json
this writes to output.json:
[
{
"ListElementCode": "1",
"ListElement": "\"SC1\" \"Leeds\""
},
{
"ListElementCode": "2",
"ListElement": "\"SC2\" \"Barnsley\""
}
]
Any idea how I can remove the quotes around the 2 text values that get put into the ListElement part?
To solve only the most immediate problem, one could write a function that strips quotes if-and-when they exist:
jq -n --raw-input --raw-output '
def stripQuotes: capture("^\"(?<content>.*)\"$").content // .;
[inputs | split(",") | map(stripQuotes) |
{
"ListElementCode": .[0],
"ListElement": "\(.[1]) \(.[2])"
}]
' <in.csv >out.json
That said, to really handle CSV correctly, you can't just split(","), but need to split only on commas that aren't inside quotes (and need to recognize doubled-up quotes as the escaped form of a single quote). Really, I'd use Python instead of jq for this job -- and of this writing, the jq cookbook agrees that native jq code is only suited for "trivially simple" CSV files.
As mentioned, a Ruby answer:
ruby -rjson -rcsv -e '
data = CSV.foreach(ARGV.shift)
.map do |row|
{
ListElementCode: row.first,
ListElement: row.drop(1).join(" ")
}
end
puts JSON.pretty_generate(data)
' input.csv
[
{
"ListElementCode": "1",
"ListElement": "SC1 Leeds"
},
{
"ListElementCode": "2",
"ListElement": "SC2 Barnsley"
}
]
Using a proper CSV/JSON parser in perl:
#!/usr/bin/env perl
use strict; use warnings;
use JSON::XS;
use Text::CSV qw/csv/;
# input.csv:
#1,"SC1","Leeds"
#2,"SC2","Barnsley"
my $vars = [csv in => 'input.csv'];
#use Data::Dumper;
#print Dumper $vars; # display the data structure
my $o = [ ];
foreach my $a (#{ $vars->[0] }) {
push #{ $o }, {
ListElementCode => $a->[0],
ListElement => $a->[1] . " " . $a->[2]
};
}
my $coder = JSON::XS->new->ascii->pretty->allow_nonref;
print $coder->encode($o);
Output
[
{
"ListElement" : "SC1 Leeds",
"ListElementCode" : "1"
},
{
"ListElement" : "SC2 Barnsley",
"ListElementCode" : "2"
}
]
Here's an uncomplicated and efficient way to solve this particular problem:
jq -n --raw-input --raw-output '
[inputs
| split(",")
| { "ListElementCode": .[0],
"ListElement": "\(.[1]|fromjson) \(.[2]|fromjson)"
} ]' input.csv
Incidentally, there are many robust command-line CSV-to-JSON tools, amongst which I would include:
any-json (https://www.npmjs.com/package/any-json)
csv2json (https://github.com/fadado/CSV)

JQ get key based on variable value

I'm trying to create a ohmyzsh function for Salesforce's DX CLI based on Wade Wegner's guide here. In order to get the value I want I need to expand how he is using JQ which I've never heard of before. I get the premise for this use case but I'm struggling with one abstraction point (within the aliasConfig json). Here's my script so far
get_sfdx_defaultusername() {
config="$(cat .sfdx/sfdx-config.json 2> /dev/null)";
globalConfig="$(cat ~/.sfdx/sfdx-config.json)";
aliasConfig="$(cat ~/.sfdx/alias.json)";
defaultusername="$(echo ${config} | jq -r .defaultusername)"
defaultusernamealias="NEED HELP HERE"
globaldefaultusername="$(echo ${globalConfig} | jq -r .defaultusername)"
if [ ! $defaultusernamealias = "null" ]
then
echoString=$echoString$defaultusernamealias"$txtylw (alias)"
elif [ ! $defaultusername = "null" ]
then
echoString=$echoString$defaultusername"$txtylw (local)"
else
echoString=$echoString$globaldefaultusername"$txtylw (global)"
fi
echo $echoString"\n"
}
The alias.json looks like this:
{
"orgs": {
"HubOrg": "myemail#domain.com",
"my-scrath-org": "test-jdj1iflkor4k#mydomain.net"
}
}
Using the ${defaultusername} I know the value in this case to be "test-jdj1iflkor4k#mydomain.net", therefore I need it to set the value of defaultusernamealias to "my-scrath-org"
NOTE: The closest answer I found was this, but unfortunately I still couldn't get what I needed with it.
Congratulations on figuring out how to use to_entries.
One small suggestion is to avoid using shell interpolation to "construct" the jq program. A much better way to achieve the desired goal is to pass in the relevant values on the command-line. In your case, the following would be appropriate:
$ jq --arg username "$defaultusername" '
.orgs | to_entries[] | select(.value == $username ).key'
Another small point is to avoid using echo to send JSON to STDIN. There are several possibilities, including these patterns:
if you are using bash: jq .... <<< "$JSON"
use printf "%s" "$JSON" | jq ...
jq -n --argjson JSON "$JSON" '$JSON | ...'
In your case, the last of these alternatives would look like this:
$ jq --arg username "$defaultusername" --argjson JSON "$aliasConfig" '
$JSON
| .orgs | to_entries[] | select(.value == $username ).key'
I think I got it figured out here:
get_sfdx_defaultusername() {
config="$(cat .sfdx/sfdx-config.json 2> /dev/null)";
globalConfig="$(cat ~/.sfdx/sfdx-config.json)";
aliasConfig="$(cat ~/.sfdx/alias.json)";
defaultusername="$(echo ${config} | jq -r .defaultusername)"
defaultusernamealias="$(echo ${aliasConfig} | jq -r '.orgs | to_entries[] | select(.value =="'$defaultusername'").key' )"
globaldefaultusername="$(echo ${globalConfig} | jq -r .defaultusername)"
if [ ! $defaultusernamealias = "null" ]
then
echoString=$echoString$defaultusernamealias"$txtylw (alias)"
elif [ ! $defaultusername = "null" ]
then
echoString=$echoString$defaultusername"$txtylw (local)"
else
echoString=$echoString$globaldefaultusername"$txtylw (global)"
fi
echo $echoString"\n"
}
This allows me to show my current defaultusername org like so:
In case anyone is interested in using this or contributing to it, I published a github repo here

using jq to assign multiple output variables

I am trying to use jq to parse information from the TVDB api. I need to pull a couple of fields and assign the values to variables that I can continue to use in my bash script. I know I can easily assign the output to one variable through bash with variable="$(command)" but I need the output to produce multiple variables and I don't want to make to use multiple commands.
I read this documentation:
https://stedolan.github.io/jq/manual/v1.5/#Advancedfeatures
but I don't know if this relevant to what I am trying to do.
jq '.data' produces the following output:
[
{
"absoluteNumber": 51,
"airedEpisodeNumber": 6,
"airedSeason": 4,
"airedSeasonID": 680431,
"dvdEpisodeNumber": 6,
"dvdSeason": 4,
"episodeName": "We Will Rise",
"firstAired": "2017-03-15",
"id": 5939660,
"language": {
"episodeName": "en",
"overview": "en"
},
"lastUpdated": 1490769062,
"overview": "Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team."
}
]
I tried jq '.data | {episodeName:$name}' and jq '.data | .episodeName as $name' just to try and get one working. I don't understand the documentation or even if it's what I'm looking for. Is there a way to do what I am trying to do?
You can use separate variables with read :
read var1 var2 var3 < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : $var1"
echo "name : $var2"
echo "full_name : $var3"
Using array :
read -a arr < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : ${arr[0]}"
echo "name : ${arr[1]}"
echo "full_name : ${arr[2]}"
Also you can split jq output with some character :
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')
Or use an array like :
set -f; IFS='|' data=($(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')); set +f
absoluteNumber, airedEpisodeNumber, episodeName & overview are respectively ${data[0]}, ${data[1]}, ${data[2]}, ${data[3]}. set -f and set +f are used to respectively disable & enable globbing.
For the jq part, all your required fields are mapped and delimited with a '|' character with join("|")
If your are using jq < 1.5, you'll have to convert Number to String with tostring for each Number fields eg:
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber|tostring, .airedEpisodeNumber|tostring, .episodeName, .overview] |
join("|")) | join("\n")')
jq always produces a stream of zero or more values. For example, to produce the two values corresponding to "episodeName" and "id"' you could write:
.data[] | ( .episodeName, .id )
For your purposes, it might be helpful to use the -c command-line option, to ensure each JSON output value is presented on a single line. You might also want to use the -r command-line option, which removes the outermost quotation marks from each output value that is a JSON string.
For further variations, please see the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, e.g. the question:
Q: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
Experimental conversion of quoted OP input, (tv.dat), to a series of bash variables, (and an array). The jq code is mostly borrowed from here and there, but I don't know how to get jq to unroll an array within an array, so the sed code does that, (that's only good for one level, but so are bash arrays):
jq -r ".[] | to_entries | map(\"DAT_\(.key) \(.value|tostring)\") | .[]" tv.dat |
while read a b ; do echo "${a,,}='$b'" ; done |
sed -e '/{.*}/s/"\([^"]*\)":/[\1]=/g;y/{},/() /' -e "s/='(/=(/;s/)'$/)/"
Output:
dat_absolutenumber='51'
dat_airedepisodenumber='6'
dat_airedseason='4'
dat_airedseasonid='680431'
dat_dvdepisodenumber='6'
dat_dvdseason='4'
dat_episodename='We Will Rise'
dat_firstaired='2017-03-15'
dat_id='5939660'
dat_language=([episodeName]="en" [overview]="en")
dat_lastupdated='1490769062'
dat_overview='Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team.'

Resources