shell script: Returning wrong output - bash

In the given script, the nested key is not getting appended with the value. I could not figure out where the script is going wrong.
#!/bin/bash
echo "Add the figma json file path"
read path
figma_json="$(echo -e "${path}" | tr -d '[:space:]')"
echo $(cat $figma_json | jq -r '.color | to_entries[] | "\(.key):\(.value| if .value == null then .[] | .value else .value end)"')
Sample input:
{
"color": {
"white": {
"description": "this is just plain white color",
"type": "color",
"value": "#ffffffff",
"extensions": {
"org.lukasoppermann.figmaDesignTokens": {
"styleId": "S:40940df38088633aa746892469dd674de8b147eb,",
"exportKey": "color"
}
}
},
"gray": {
"50": {
"description": "",
"type": "color",
"value": "#fafafaff",
"extensions": {
"org.lukasoppermann.figmaDesignTokens": {
"styleId": "S:748a0078c39ca645fbcb4b2a5585e5b0d84e5fd7,",
"exportKey": "color"
}
}
}
}
}
}
Actual output:
white:#ffffffff gray:#fafafaff
Excepted output:
white:#ffffffff gray:50:#fafafaff
Full input file

Here's a solution using tostream instead of to_entries to facilitate simultaneous access to the full path and its value:
jq -r '
.color | tostream | select(.[0][-1] == "value" and has(1)) | .[0][:-1]+.[1:] | join(":")
' "$figma_json"
white:#ffffffff
gray:50:#fafafaff
Demo

An approach attempting to demonstrate bash best-practices:
#!/bin/bash
figma_json=$1 # prefer command-line arguments to prompts
[[ $figma_json ]] || {
read -r -p 'Figma JSON file path: ' path # read -r means backslashes not munged
figma_json=${path//[[:space:]]/} # parameter expansion is more efficient than tr
}
jq -r '
def recurse_for_value($prefix):
to_entries[]
| .key as $current_key
| .value?.value? as $immediate_value
| if $immediate_value == null then
.value | recurse_for_value(
if $prefix != "" then
$prefix + ":" + $current_key
else
$current_key
end
)
else
if $prefix == "" then
"\($current_key):\($immediate_value)"
else
"\($prefix):\($current_key):\($immediate_value)"
end
end
;
.color |
recurse_for_value("")
' "$figma_json"

Related

Loop json Output in shell script

I have this output variable
OUTPUT=$(echo $ZONE_LIST | jq -r '.response | .data[]')
The Output:
{
"accountId": "xyz",
"addDate": "2020-09-05T10:57:11Z",
"content": "\"MyContent\"",
"id": "MyID",
"priority": null
}
{
"accountId": "xyz",
"addDate": "2020-09-05T06:58:52Z",
"content": "\"MyContent\"",
"id": "MyID",
"priority": null
}
How can I create a loop for this two values?
MyLoop
echo "$content - $id"
done
I tried this, but then I get a loop through every single value
for k in $(echo $ZONE_LIST | jq -r '.response | .data[]'); do
echo $k
done
EDIT 1:
My complete JSON:
{
"errors": [],
"metadata": {
"transactionId": "",
},
"response": {
"data": [
{
"accountId": "xyz",
"addDate": "2020-09-05T10:57:11Z",
"content": "\"abcd\"",
"id": "myID1",
"lastChangeDate": "2020-09-05T10:57:11Z",
},
{
"accountId": "xyz",
"addDate": "2020-09-05T06:58:52Z",
"content": "\"abc\"",
"id": "myID2",
"lastChangeDate": "2020-09-05T07:08:15Z",
}
],
"limit": 10,
"page": 1,
"totalEntries": 2,
},
"status": "success",
"warnings": []
}
Now I need a loop for data, because I need it for a curl
The curl NOW:
curl -s -v -X POST --data '{
"deleteEntries": [
Data_from_json
]
}' https://URL_to_Update 2>/dev/null)
Now I want to create a new variable from my JSON data. My CURL should look like this at the end:
curl -s -v -X POST --data '{
"deleteEntries": [
{
"readID": "myID1",
"date": "2020-09-05T10:57:11Z", <--Value from addDate
"content": "abcd"
},
{
"readID": "myID2",
"date": "2020-09-05T06:58:52Z", <--Value from addDate
"content": "abc"
}
]
}' https://URL_to_Update 2>/dev/null)
Something like:
#!/usr/bin/env bash
while IFS=$'\37' read -r -d '' id content; do
echo "$id" "$content"
done < <(
jq -j '.response | .data[] | .id + "\u001f" + .content + "\u0000"' \
<<<"$ZONE_LIST"
)
jq -j: Forces a raw output from jq.
.id + "\u001f" + .content + "\u0000": Assemble fields delimited by ASCII FS (Hexadecimal 1f or Octal 37), and end record by a null character.
It then becomes easy and reliable to iterate over null delimited records by having read -d '' (null delimiter).
Fields id content are separated by ASCII FS, so just set the Internal Field Separator IFS environment variable to the corresponding octal IFS=$'37' before reading.
The first step is to realize you can turn the set of fields into an array like this using a technique like this:
jq '(.accountId + "," + .addDate)'
Now you can update your bash loop:
for k in $(echo $ZONE_LIST | jq -r '.response | .data[]' | jq '(.content + "," + .id)'); do
echo $k
done
There is probably a way to combine the two jq commands but I don't have your original json data for testing.
UPDATE - inside the loop you can parse the comma-delimited string into separate fields. This are more efficient ways to handle this task but I prefer simplicity.
ID=$(echo $k | cut -d',' -f1)
PRIORITY=$(echo $k | cut -d',' -f2)
echo "ID($ID) PRIORITY($PRIORITY)"
Try this.
for k in $(echo $ZONE_LIST | jq -rc '.response | .data[]'); do
echo $k|jq '.content + " - " + .id' -r
done

JQ statement to build Json from csv

I have a CSV file that I want to convert to a JSON file with the quotes from the CSV removed using JQ in a shell script.
Here is the CSV named input.csv:
1,"SC1","Leeds"
2,"SC2","Barnsley"
Here is the JQ extract:
jq --slurp --raw-input --raw-output \
'split("\n") | .[1:] | map(split(",")) |
map({
"ListElementCode": .[0],
"ListElement": "\(.[1]) \(.[2])
})' \
input.csv > output.json
this writes to output.json:
[
{
"ListElementCode": "1",
"ListElement": "\"SC1\" \"Leeds\""
},
{
"ListElementCode": "2",
"ListElement": "\"SC2\" \"Barnsley\""
}
]
Any idea how I can remove the quotes around the 2 text values that get put into the ListElement part?
To solve only the most immediate problem, one could write a function that strips quotes if-and-when they exist:
jq -n --raw-input --raw-output '
def stripQuotes: capture("^\"(?<content>.*)\"$").content // .;
[inputs | split(",") | map(stripQuotes) |
{
"ListElementCode": .[0],
"ListElement": "\(.[1]) \(.[2])"
}]
' <in.csv >out.json
That said, to really handle CSV correctly, you can't just split(","), but need to split only on commas that aren't inside quotes (and need to recognize doubled-up quotes as the escaped form of a single quote). Really, I'd use Python instead of jq for this job -- and of this writing, the jq cookbook agrees that native jq code is only suited for "trivially simple" CSV files.
As mentioned, a Ruby answer:
ruby -rjson -rcsv -e '
data = CSV.foreach(ARGV.shift)
.map do |row|
{
ListElementCode: row.first,
ListElement: row.drop(1).join(" ")
}
end
puts JSON.pretty_generate(data)
' input.csv
[
{
"ListElementCode": "1",
"ListElement": "SC1 Leeds"
},
{
"ListElementCode": "2",
"ListElement": "SC2 Barnsley"
}
]
Using a proper CSV/JSON parser in perl:
#!/usr/bin/env perl
use strict; use warnings;
use JSON::XS;
use Text::CSV qw/csv/;
# input.csv:
#1,"SC1","Leeds"
#2,"SC2","Barnsley"
my $vars = [csv in => 'input.csv'];
#use Data::Dumper;
#print Dumper $vars; # display the data structure
my $o = [ ];
foreach my $a (#{ $vars->[0] }) {
push #{ $o }, {
ListElementCode => $a->[0],
ListElement => $a->[1] . " " . $a->[2]
};
}
my $coder = JSON::XS->new->ascii->pretty->allow_nonref;
print $coder->encode($o);
Output
[
{
"ListElement" : "SC1 Leeds",
"ListElementCode" : "1"
},
{
"ListElement" : "SC2 Barnsley",
"ListElementCode" : "2"
}
]
Here's an uncomplicated and efficient way to solve this particular problem:
jq -n --raw-input --raw-output '
[inputs
| split(",")
| { "ListElementCode": .[0],
"ListElement": "\(.[1]|fromjson) \(.[2]|fromjson)"
} ]' input.csv
Incidentally, there are many robust command-line CSV-to-JSON tools, amongst which I would include:
any-json (https://www.npmjs.com/package/any-json)
csv2json (https://github.com/fadado/CSV)

jq how to get the last entry in an object

I'm working with jq 1.6 to get the last entry in an object. It should work like this:
data='{ "1": { "a": "1" }, "2": { "a": "2" }, "3": { "a": "3" } }'
result=`echo $data | jq 'myfilter'`
echo $result
{ "3": { "a": "3" } }
I tried these filters:
jq '. | last' # error: Cannot index object with number
How can I tell jq to quote the number?
jq '. | to_entries | last' # { "key": "3", "value": { "a": "3" } }
I guess I could munge this up by concatenating the key and value entries. Is there a simpler way?
The tutorial and the manual didn't help. No joy on SO either.
You can use the following :
jq 'to_entries | [last] | from_entries'
Try it here.
We can't use with_entries(last) because last returns a single element and from_entries requires an array, hence the [...] construct above.

Pass an array variable to jq in bash

Is it possible to pass and use in jq a variable of type array?
jq --arg ips "${IPs[0]}" '.nodes.app.ip = $ips[0] | .nodes.data.ip = $ips[1]' nodes.json
The general case solution is to pass that array in on stdin with NUL delimiters:
IPs=( 1.2.3.4 5.6.7.8 )
original_doc='{"nodes": { "app": {}, "data": {} }}'
jq -Rn --argjson original_doc "$original_doc" '
input | split("\u0000") as $ips
| $original_doc
| .nodes.app.ip = $ips[0]
| .nodes.data.ip = $ips[1]
' < <(printf '%s\0' "${IPs[#]}")
...emits as output:
{
"nodes": {
"app": {
"ip": "1.2.3.4"
},
"data": {
"ip": "5.6.7.8"
}
}
}
This is overkill for an array of IP addresses, but it works in the general case, even for actively-hostile arrays (ones with literal quotes, literal newlines, and other data that's intentionally hard-to-parse).
If you want to keep stdin clean, you can use a second copy of jq to convert your array to JSON:
IPs=( 1.2.3.4 5.6.7.8 )
IPs_json=$(jq -Rns 'input | split("\u0000")' < <(printf '%s\0' "${IPs[#]}"))
jq --argjson ips "$IPs_json" '
.nodes.app.ip = $ips[0]
| .nodes.data.ip = $ips[1]
' <<<'{"nodes": { "app": {}, "data": {} }}'

How to hande a curl response fail in Bash script

How can I handle request fails in this example of bash curl requests. I.e. if all servers are responde with JSON all is okay and I have JSON file at end of a cycle. But if one of this servers not responde with JSON or not responde at all I do have nothing in "/data.json" file, even all other servers are working perfectly. How can I catch a server fail and skip it?
#!/bin/bash
CONFIG=config.json
jsondata=data.json
i=1
echo "{ \"success\": \"OK\", \"servers\": {" > $jsondata
jq -r '.servers|keys[]' $CONFIG | while read key ; do
if [ "$i" -ne "1" ]; then
echo "," >> $jsondata
fi
echo "\"server$i\": {" >> $jsondata
RESPONSE=$(curl -s $HTTP://$IP:$PORT/api)
DATA1=$(echo $RESPONSE | jq '.data1')
DATA2=$(echo $RESPONSE | jq '.data2')
echo " \"data1\": $DATA1", >> $jsondata
echo " \"data2\": $DATA2", >> $jsondata
echo "}" >> $jsondata
((i++))
done
echo "}," >> $jsondata
First, let's assume we have a function to make a single API call.
do_api () {
key=$1
curl $HTTP://$IP:$PORT/api # Presumably, key is needed here somewhere
}
I'll also assume that the intended output is a JSON object that has at least two fields, data1 and data2. Now, we can write a simple pipeline with three stages:
Read keys from $CONFIG
Make an API call for each key
Generate the desired output from the combined output of all the API calls.
It's not too complicated:
jq -r '.servers | keys[]' | # stage 1
while read key; do do_api "$key"; done | # stage 2
jq -s 'to_entries |
map({key: "server(\.key+1)",
value: {data1: .value.data1,
data2: .value.data2}
}) |
{success: "OK", servers: from_entries}' # stage 3
do_api should output nothing for a particular key of curl fails, but you can modify it to produce some sort of default data if you wish:
do_api () {
key=$1
curl --fail ... || jq -n '{data1: null, data2: null}'
}
jq -s 'to_entries' takes an input like
{ "data1": ..., "data2": ... }
{ "data1": ..., "data2": ... }
(which is what we expect from curl), and produces as output
[ { "key": 0, "value": { "data1": ..., "data2": ... } },
{ "key": 1, "value": { "data1": ..., "data2": ... } }
]
The map(...) filter takes the preceding array and produces the keys and values we want to add to the servers object in the final result, which is created by a call to from_entries.
Here is a full example. tmp.json contains the simulated output of do_api, complete with an extra field data3 that will be filtered from the final output.
$ cat tmp.json
{
"data1": "foo",
"data2": "bar",
"data3": "baz"
}
{
"data1": "hello",
"data2": "world",
"data3": "bye"
}
$ jq -s 'to_entries | map({key: "server\(.key+1)", value: {data1: .value.data1, data2: .value.data2}}) | {success: "OK", servers: from_entries}' tmp.json
{
"success": "OK",
"servers": {
"server1": {
"data1": "foo",
"data2": "bar"
},
"server2": {
"data1": "hello",
"data2": "world"
}
}
}

Resources