I'm using Bash to extract json keys by 'jsawk'. I converted a some.xlsx to some.json by node xlsx . I have some keys like this:
"Scheduled Release"
"Standard Input Parameters\n(Dev)"
The second one contains '\n' because the cell in the .xlsx has 2 lines.
The problem is that I cannot extract the value, using this:
$ cat some.json | jsawk -n 'out(this.["Scheduled Release"])'
or
$ cat some.json | jsawk -n 'out(this.["Standard Input Parameters\n(Dev)"])'
No problem for the keys without white spaces.
My some.json contains an array of objects like this:
{
"Service": "A service",
"Resource": "Some Users",
"Engineer": "A name",
"Description": "A description",
"Scheduled Release" : "1.0",
"Method": "A method",
"Endpoint": "An enpoint",
"Standard Input Parameters\n(Dev)": "Some inputs"
}
How can I do this with Bash?
Now I can extract my expected key using this:
$cat some.json | jsawk -n 'out(this["Standard Input Parameters\n(Dev)"])'
Related
I'm writing a bash script for hetzner's cloud API, and I need to store the server's ID to a text file. After the command using it will output the below,
{
"server": {
"id": 12345678,
"name": "servertest-101",
"status": "initializing",
"created": "2020-09-18T09:22:21+00:00",
This is just a snippet, but that's from the first line of the response.
How can I extract and store that value?
The api returns in json format: You've not given much information but use jq to parse it:
$ cat myinput.json
{
"server": {
"id": 12345678,
"name": "servertest-101",
"status": "initializing",
"created": "2020-09-18T09:22:21+00:00"
}
}
$ jq -r .server.id myinput.json
12345678
redirect to a file:
$ jq -r .server.id myinput.json > myoutputfile
$ cat myoutputfile
12345678
You can pipe output of your command to process it further as this:
cat yourjson.json | grep -m 1 -E -o '\"id\": [0-9]+' | cut -d" " -f 2 > yourtextfile.txt
First, get your json content, then send it through the grep command that extracts only part "id": 1234567 using regular expression. Then pipe this result to cut command that splits it by a space and selects the second part, which is your value. Lastly, you redirect result of the job to the desired text file.
If you are sure that your value is going to always be the first number in the input, you can just simply select it by grep:
cat yourjson.json | grep -m 1 -E -o '[0-9]+' > output.txt
I have a json object with below element,
rsrecords="{
"ResourceRecords": [
{
"Value": "\"heritage=external-dns,external-dns/owner=us-east-1:sandbox,external-dns/resource=service/api""
}
],
"Type": "TXT",
"Name": "\\052.apiconsumer.alpha.sandbox.test.net.",
"TTL": 300
}"
And in my bash script,I have below code snippet,
jq -r '.[] | .Name ,.ResourceRecords[0].Value' <<< "$rsrecords" | \
while read -r name; read -r value; do
echo $name
Output is printed as,
\052.apiconsumer.alpha.sandbox.test.net.
But I am expecting it to print as \\052.apiconsumer.alpha.sandbox.test.net., which is , as it is "Name" from the json object..
How can this be done?
Before getting to the heart of the matter, please note that
the sample data as given is a bit of a mishmash, so I'll assume you meant something like:
rsrecords='
{
"ResourceRecords": [
{
"Value": "heritage=external-dns,external-dns/owner=us-east-1:sandbox,external-dns/resource=service/api"
}
],
"Type": "TXT",
"Name": "\\052.apiconsumer.alpha.sandbox.test.net.",
"TTL": 300
}
'
Your jq query does not match the above JSON, so I'll assume you intended the query to be simply:
.Name, .ResourceRecords[0].Value
In any case, with the above JSON, the bash commands:
jq -r '.Name, .ResourceRecords[0].Value' <<< "$rsrecords" |
while read -r name; read -r value; do
echo "$name"
done
yields:
\052.apiconsumer.alpha.sandbox.test.net.
This is correct, because the JSON string "\\X" is an encoding of the raw string: \X
If you want to see the JSON string, then invoke jq without the -r option. If you want to invoke jq with the -r option and want to see two backslashes, you will have to encode them as four backslashes in your JSON.
In the below script, I am not able to successfully call the "repovar" variable in the jq command.
cat quayrepo.txt | while read line
do
export repovar="$line"
jq -r --arg repovar "$repovar" '.data.Layer| .Features[] | "\(.Name), \(.Version), $repovar"' severity.json > volume.csv
done
The script uses a text file to loop through the repo names
quayrepo.txt---> file has the list of names in this case the file has a value of "Reponame1"
sample input severity.json file:
{
"status": "scanned",
"data": {
"Layer": {
"IndexedByVersion": 3,
"Features": [
{
"Name": "elfutils",
"Version": "0.168-1",
"Vulnerabilities": [
{
"NamespaceName": "debian:9",
"Severity": "Medium",
"Name": "CVE-2016-2779"
}
]
}
]
}
}
}
desired output:
elfutils, 0.168-1, Medium, Reponame1
Required output: I need to retrieve the value of my environment variable as the last column in my output csv file
You need to surround $repovar with parenthesis, as the other values
repovar='qweqe'; jq -r --arg repovar "$repovar" '.data.Layer| .Features[] | "\(.Name), \(.Version), \($repovar)"' tmp.json
Result:
elfutils, 0.168-1, qweqe
There's no need for the export.
#!/usr/bin/env bash
while read line
do
jq -r --arg repovar "$line" '.data.Layer.Features[] | .Name + ", " + .Version + ", " + $repovar' severity.json
done < quayrepo.txt > volume.csv
with quayrepo.txt as
Reponame1
and severity.json as
{
"status": "scanned",
"data": {
"Layer": {
"IndexedByVersion": 3,
"Features": [
{
"Name": "elfutils",
"Version": "0.168-1",
"Vulnerabilities": [
{
"NamespaceName": "debian:9",
"Severity": "Medium",
"Name": "CVE-2016-2779"
}
]
}
]
}
}
}
produces volume.csv containing
elfutils, 0.168-1, Reponame1
To #peak's point, changing > to >> in ...severity.json >> volume.csv will create a multi-line csv instead of just overwriting until the last line
You don't need a while read loop in bash at all; jq itself can loop over your input lines, even when they aren't JSON, letting you run jq only once, not once per line in quayrepo.txt.
jq -rR --slurpfile inJson severity.json <quayrepo.txt >volume.csv '
($inJson[0].data.Layer | .Features[]) as $features |
[$features.Name, $features.Version, .] |
#csv
'
jq -R specifies raw input, letting jq directly read lines from quayrepo.txt into .
jq --slurpfile varname filename.json reads filename.json into an array of JSON objects parsed from that file. If the file contains only one object, one needs to refer to $varname[0] to refer to it.
#csv converts an array to a CSV output line, correctly handling data with embedded quotes or other oddities that require special processing.
I have a test.txt file in this format
{
"user": "sthapa",
"ticket": "LIN-5867_3",
"start_date": "2018-03-16",
"end_date": "2018-03-16",
"demo_nos": [692],
"service_names": [
"service1",
"service2",
"service3",
"service4",
"service5",
"service6",
"service7",
"service8",
"service9"
]
}
I need to look for a tag called demo_nos and provide the count of it.
For example in the above file "demo_nos": [692] which means only one demo nos...similarly if it had "demo_nos": [692,300] then the count would be 2
so what shell script can i write to fetch and print the count?
The output should say the demo nos = 1 or 2 depending on the values inside the tag [].
i.e I have a variable in my shell script called market_nos which should give me it's count
The gold standard for manipulating JSON data from the command line is jq:
$ jq '.demo_nos | length' test.txt
1
.demo_nos returns the value associated with the demo_nos key in the object, and that array is piped to the length function which does the obvious.
I'm assuming you have python and the file is JSON :)
$ cat some.json
{
"user": "sthapa",
"ticket": "LIN-5867_3",
"start_date": "2018-03-16",
"end_date": "2018-03-16",
"demo_nos": [692],
"service_names": [
"service1",
"service2",
"service3",
"service4",
"service5",
"service6",
"service7",
"service8",
"service9"
]
}
$ python -c 'import sys,json; print(len(json.load(sys.stdin)["demo_nos"]))' < some.json
1
Not the most elegant solution but this should do it
cat test.txt | grep -o -P 'demo_nos.{0,200}' | cut -d'[' -f2 | cut -d']' -f1 | awk -F',' '{ print NF }'
Please note that this is a quick and dirty solution treating input as raw text, and not taking into account JSON structure. In exceptional cases were "demo_nos" string would also appear elsewhere in the file, the output from the command above might be incorrect.
i'm trying to build script that takes specific attribute value and store it in the array , this is the following JSON file:
[
{
"id": 1,
"name": "myna",
"description": "Simple Question",
"speaker": "USER",
},
{
"all_Id's": [
"11111"
],
"user": "me",
},
{
"id": 2,
"name": "mry",
"description": "Simple",
"speaker": "aaa",
}
]
as you see object in json file don't have the same attributes so i'm looking only on object has "name " attribute,the following script reads the Json file and return the values of attribute name only ,but i build something wrond as theERROR always on the "{" of the last object in file I don't know why , what i am i doing wrong?
the expected output is : [myna, mry]
#!/bin/bash
declare -a OB_I=()
declare counter1=0
jq -r '.name' file.json ; while read -r val ; do
if [[ ! $val ]]
then
OB_I[$counter]=$val ;
counter=$((counter+1));
fi
done;
$ printf '%s\n' "${OB_I[#]}"
The input of jq is a list, which doesn't have any keys, let alone one named name. You want
jq -r '.[].name'
instead.
Unrelated, you don't need the variable counter. You can simply append to your array with OB_I+=("$val").