Formating awk output to json using jq - bash

I'm using grep to search for specific text inside multiple files.
grep -n -r "text_to_match" *
The output is:
john:15
Hana:11
After that, am using awk to print whatever meets my criteria of the condition
grep -n -r "text_to_match" * | awk -F '[:]' '{print $1,$2}'
I want now to use jq to have the following json document
{
"data": [
{
"name": "john",
"age": 15
},
{
"name": "Hana",
"age": 11
}
]
}
I tried the following:
jq -R 'split(" ") | { file:.[0], start_line:.[1]}'
Output
{
"name": "john",
"age": 15
}
{
"name": "Hana",
"age": 11
}
Thank you

As jq can also do the splitting, you could start before doing it with awk, having
john:15
Hana:11
Then, with -R you can read in the raw lines. Using -n in combination with [inputs] makes it an array. You can split using the / operator, and wrap everything in an object of your choice.
jq -Rn '{data: [inputs / ":" | {name: .[0], age: .[1]}]}'
{
"data": [
{
"name": "john",
"age": "15"
},
{
"name": "Hana",
"age": "11"
}
]
}
Demo

Related

How to iterate through list of dictionaries and sort the output based on the key in Bash?

We are getting the output from an API call in the below list of dictionaries format,
[
{
"Key": "/builder-deployer/test2/services/serviceA/ip",
"Val": "10.1.2.1"
},
{
"Key": "/builder-deployer/test2/services/serviceA/port",
"Val": "2"
},
{
"Key": "/builder-deployer/test2/services/serviceA/url",
"Val": "serviceA.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceA/username",
"Val": "jenkins"
},
{
"Key": "/builder-deployer/test2/services/serviceB/ip",
"Val": "10.1.2.2"
},
{
"Key": "/builder-deployer/test2/services/serviceB/port",
"Val": "3"
},
{
"Key": "/builder-deployer/test2/services/serviceB/url",
"Val": "serviceB.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceB/username",
"Val": "jenkins"
},
{
"Key": "/builder-deployer/test2/services/serviceC/ip",
"Val": "10.1.2.2"
},
{
"Key": "/builder-deployer/test2/services/serviceC/port",
"Val": "4"
},
{
"Key": "/builder-deployer/test2/services/serviceC/url",
"Val": "serviceC.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceC/username",
"Val": "jenkins"
}
]
I have a requirement to fetch the unique username and IP combination in the form of username#ip from the list to install some dependencies.
i.e. in the above list of dictionaries, we have 2 unique username#ip combinations. So I need to install packages on jenkins#10.1.2.1, jenkins#10.1.2.2 machines respectively.
I have the below code in a python script that does the same but most of our code is in Bash, we want to convert this also to Bash. Any help would be very helpful
ip=[]
for username in outputs:
for ip in outputs:
if username['Key'].split('/')[-2] == ip['Key'].split('/')[-2] and username['Key'].split('/')[-1] in ['username'] and ip['Key'].split('/')[-1] in ['ip']:
ip.append(username['Val']+'#'+ip['Val'])
print(set(ip))
jq
$ FILE="file.json"
$ paste -d# \
<(jq -r '.[]|select(.Key|contains("/username")).Val' "$FILE") \
<(jq -r '.[]|select(.Key|contains("/ip")).Val' "$FILE") | \
sort -u
jenkins#10.1.2.1
jenkins#10.1.2.2
awk
awk -F\" '/\/ip/{getline;ip=$4}/username/{getline;$0=$4"#"ip;print|"sort -u"}' "$FILE"
jenkins#10.1.2.1
jenkins#10.1.2.2
The suggestion in the comments to leave your solution in python (per Gordon Davisson) would be my preference as well. But, if you really have to have a bash solution, the following is one option. Bear in mind it will be fragile if your API JSON format changes over time. It is also not terribly efficient as it relies on reading the source JSON multiple occasions. Further, both grep and sort are also utilized:
#!/bin/bash
input_json="${1:-/tmp/bar/t.json}"
while read -r line ; do
search_str=$(grep -oB 1 "${line}" "${input_json}" \
| awk 'NR==1 {gsub(/^"/,"", $2); \
gsub(/ip",$/,"username", $2); print $2}')
user=$(grep -oA 1 "${search_str}" "${input_json}" | \
awk 'NR==2 {gsub(/"/,"", $2); print $2}')
echo "${user}#${line}"
# get unique ip addresses from source json
done < <(grep -Eo "([0-9]{1,3}[\.]){3}[0-9]{1,3}" "${input_json}" | sort -u)
Sample data:
[
{
"Key": "/builder-deployer/test2/services/serviceA/ip",
"Val": "10.1.2.1"
},
{
"Key": "/builder-deployer/test2/services/serviceA/port",
"Val": "2"
},
{
"Key": "/builder-deployer/test2/services/serviceA/url",
"Val": "serviceA.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceA/username",
"Val": "jenkins"
},
{
"Key": "/builder-deployer/test2/services/serviceB/ip",
"Val": "10.1.2.2"
},
{
"Key": "/builder-deployer/test2/services/serviceB/port",
"Val": "3"
},
{
"Key": "/builder-deployer/test2/services/serviceB/url",
"Val": "serviceB.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceB/username",
"Val": "jenkins"
},
{
"Key": "/builder-deployer/test2/services/serviceC/ip",
"Val": "10.1.2.2"
},
{
"Key": "/builder-deployer/test2/services/serviceC/port",
"Val": "4"
},
{
"Key": "/builder-deployer/test2/services/serviceC/url",
"Val": "serviceC.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceC/username",
"Val": "jenkins"
}
]
Sample output:
$ ./t.sh
jenkins#10.1.2.1
jenkins#10.1.2.2
Use jq once, no need of temporary file
$ cat test.json | jq -r '.[]|select(.Key|test("(username|ip)$")).Val' \
| paste -d '#' - - | sort -u
10.1.2.1#jenkins
10.1.2.2#jenkins

Bash get value of matching substring in a long string

I'm working on a bash script and having hard time extracting value from a matching substring.
I cannot share curl command as it has sensitive information but updating i value(1)(updated to mimic real value) that I'm having problem with. jq gives parse error
curl -s -g "$line" | jq -c '.allBuilds[]' | while read i; do
job_name=$(echo "$i" | jq .fullDisplayName | tr -d '»' | tr -s " " | sed 's/ /,/g' | tr -d '"')
done
expected output
Hello,Java,World,master,#47
I get expected output on most of the i values but some error out
Below are sample i values.
1
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.StringParameterValue","name":"environment","value":"DE"},{"_class":"hudson.model.BooleanParameterValue","name":"update","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"black","value":false},{"_class":"hudson.model.StringParameterValue","name":"description","value":"DE"},{"_class":"hudson.model.StringParameterValue","name":"number","value":""},{"_class":"hudson.model.StringParameterValue","name":"config","value":"{ "E": "DE", "Exp": "1111", "Pr": "D", "Man": { "Se": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Se-B8SKMz", "OR": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/OR-kvJ2lJ", "AR": "arn:aws:secretsmanager:region:111111111111:secret:A/DE/SA/rds", "User": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/mUixbWY", "sales": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/saELY", "vau": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/vNRR7BO", "sc": "", "exAd": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/ExyBoYL", "exp": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/Exl67GE", "sec": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Secle06a", "Secu": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Seia" }, "s3": { "buckets": { "hello": { "name": "helloDEBA", "region": "region", "account": "111111111111" }, "mlt": { "name": "sacdhbd", "region": "region", "account": "111111111111" }, "devo": { "name":"devvvvv", "region": "region", "account": "5555555" } } }, "roles": { "lam": "arn:aws:iam::111111111111:role/lam", "lambd": "arn:aws:iam::111111111111:role/lambd", "la": "arn:aws:iam::111111111111:role/lam", "la": "arn:aws:iam::111111111111:role/la", "la": "arn:aws:iam::111111111111:role/la","lasds": "arn:aws:iam::111111111111:role/lafgg", "lafdg": "arn:aws:iam::111111111111:role/dfsdv", "acc": "arn:aws:iam::111111111111:role/acc" }, "vpc": { "subnets": { "private": { "1a": "subnet-111111", "1b": "subnet-22222", "1c": "subnet-33333" } }, "securityGroupIds": { "lambda": "sg-1111" }, "endpoints": { "e": "" }, "links": { "b": "" } }, "securi": { "level": "FAILURE", "s": true }, "log": "debug", "se": "hello.com", "sa": { "env": "--DE" }, "lam": { "sss": { "environment": { "variables": { "test": "hello.com", "PhoneNumber": "11111" } } } } }"},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"ch","value":false}]},{"_class":"jenkins.scm.A.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello » Java » World » master #25","id":"25","number":25,"timestamp":1575582153372}
2
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.BooleanParameterValue","name":"helo-world","value":false},{"_class":"hudson.model.StringParameterValue","name":"environment","value":"hello"},{"_class":"hudson.model.StringParameterValue","name":"config","value":""},{"_class":"hudson.model.StringParameterValue","name":"description","value":""},{"_class":"hudson.model.BooleanParameterValue","name":"hello","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"hello2","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.StringParameterValue","name":"hello3","value":""}]},{"_class":"jenkins.scm.api.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello»Java»World»master#21","id":"21","number":21,"timestamp":1574705384077}
I'm trying to extract fullDisplayName value and I have tried some options like
printf '%s\n' "$i" | grep fullDisplayName
printf '%s\n' gives output in new lines when tried in shell but in script its different behavior
{}
{}
{}]
fullDisplayName:Hello » Java » World » master #25
id:25
number:25
timestamp:1575582153372
It would appear that you should be using read -r.
It also appears that it would be much simpler if you focused on using curl and jq to extract the information, without any grep or tr invocation and without any shell looping. Assuming you can arrange for the output of curl to be valid JSON(*), a single invocation of jq along the following lines should do the job:
jq -c '.allBuilds[] | .fullDisplayName | gsub("»";"")'
(*) To check whether the output of curl is valid, you can pipe the output of your curl command into jq empty:
curl ... | jq empty

Parsing JSON with jq

I have to parse a JSON which includes an array of of class/name:
{
"_class": "model.ListView",
"jobs": [
{
"_class": "hudson.matrix.MatrixProject",
"name": "tests-different-node-full"
},
{
"_class": "hudson.matrix.MatrixProject",
"name": "tests-jms-activemq-full"
},
{
"_class": "hudson.matrix.MatrixProject",
"name": "tests-txpropag-jpa-full"
}
]
}
I need to retrieve the list of "name".
Looking at the examples I've found of jq, I have tried with:
cat jobs.json | jq '.[].name'
It fails with:
jq: error (at <stdin>:0): Cannot index string with string "name"
How should I reference the name element of the array?
Thanks
found it:
cat jobs.json | jq '.jobs[].name'

Replace text and insert more text

I need to replace some text in a file and add couple of lines before the end of file.
I have this:
{
"something": "option1",
"other": [
"value",
]
}
and I need this:
{
"something": "option2",
"other": [
"value",
],
"more": {
"stuff": "yey!"
}
}
I tried this scipt:
stuff=$1
value=",
\"more\": {
\"stuff\": \"$stuff\"
}"
sed -e "s/option1/option2" -e '$ \i$value' \
file1.json > file2.json
but I get:
{
"something": "option2",
"other": [
"value",
]
$value
}
How do I do this properly?
The only right way to manipulate json data is using JSON parsers/processors. Period!
Use jq processor, it'll make your "relationship" with JSON easy and comfortable:
Valid JSON file1.json:
{
"something": "option1",
"other": [
"value"
]
}
stuff="jq got you"
jq --arg stuff "$stuff" '.more = {stuff: $stuff}' file1.json
The output:
{
"something": "option1",
"other": [
"value"
],
"more": {
"stuff": "jq got you"
}
}
It's not the better way, but if you want with sed, you can try
With gnu sed
echo "$value" | sed ':A;/\n.*}/!{N;bA};h;s/.*//;N;$!{s/.*\n//;:B;N;$!{H;s/option1/option2/;s/\n.*//p;g;s/\(.*\)\(\n.*\n[^\n]*\)/\1/;x;s/\(.*\n\)\([^\n]*\)/\2/;bB}};H;g;s/\(.*\n\)\([^\n]*\)\(\n[^\n]*\)/\2/;G;s/\(.*\n\)\([^\n]*\n\)\([^\n]*\)/\1\3/;s/\([^\n]\)\(\n\)\(.*\)/\1\3/' /dev/stdin infile
Or with OpenBSD sed
cat scriptsed
sed ':A;/\n.*}/!{N;bA
}
h;s/.*//;N;$!{s/.*\n//;:B;N;$!{H;s/option1/option2/;s/\n.*//p;g;s/\(.*\)\(\n.*\n[^\n]*\)/\1/;x;s/\(.*\n\)\([^\n]*\)/\2/;bB
}
}
H;g;s/\(.*\n\)\([^\n]*\)\(\n[^\n]*\)/\2/;G;s/\(.*\n\)\([^\n]*\n\)\([^\n]*\)/\1\3/;s/\([^\n]\)\(\n\)\(.*\)/\1\3/'
And call it like that
(echo "$value";cat infile) | ./scriptsed

Replace an attribute or key in JSON using jq or sed

Have a big json like this
"envConfig": {
"environmentName": {
"versions": [
{
"name": "version1",
"value": "Dev"
},
{
"name": "version2",
"host": "qa"
}
],
"userRoles": [
{
"name": "Roles",
"entry": [
{
"name": "employees",
"value": "rwx"
},
{
"name": "customers",
"value": "rx"
}
]
}
]
}
},
I wanted to change the JSON attribute from "environmentName" to "prod". Below is the output i am expecting
"envConfig": {
"prod": {
"versions": [
...
],
"userRoles": [
...
]
}
}
Tried with sed command as below
sed "s/\('environmentName':\)/\1\"prod\"\,/g" version.json
Tried with jq as below but not working
cat version.json | jq ' with_entries(.value |= {"prod" : .environmentName} ) '
Any help here to replace the attribute/key of an json with desired value
You weren't too far off with the jq, how about this?
jq '.envConfig |= with_entries(.key |= sub("^environmentName$"; "prod"))'
Two differences: first off, we want to drill down to envConfig before doing a with_entries, and second, when we get there, the thing we want will be a key, not a value. In case there are any other keys besides environmentName they'll be preserved.
TL,TR
You can use the following command:
jq '(.envConfig |= (. + {"prod":.environmentName}|del(.environmentName)))' foo.json
Let's say you have the following json:
{
"foo": {
"hello" : "world"
}
}
You can rename the node foo to bar by first duplicating it and then remove the original node:
jq '. + {"bar":.foo}|del(.foo)' foo.json
Output:
{
"bar": {
"hello" : "world"
}
}
It get's a bit more complicated if you want to replace a child key somewhere in the tree. Let's say you have the following json:
{
"test": {
"foo": {
"hello": "world"
}
}
}
You can use the following jq command for that:
jq '(.test |= (. + {"bar":.foo}|del(.foo)))' foo.json
Note the additional parentheses and the use of the assignment operator |=.
Output:
{
"test": {
"bar": {
"hello": "world"
}
}
}
Using sed:
sed -i '/^ \"environmentName\":/ s/environmentName/prod/' <yourfile>
Keep in mind that -i will overwrite the file. You may want to make a backup first.

Resources