jq - generate list of dictionaries instead of list of strings - shell

The following command:
cat foo | jq -r '[.items[] | select(.metadata.random_field["foo/foo"] == "true") | .metadata.name]'
results in
[
"test_data"
]
I would like the following output to be:
[
{"name":"test", "data":"test_data"}
]
foo
{
"apiVersion": "v1",
"items":[
{
"metadata": {
"random_field": {
"foo/foo": "true"
},
"creationTimestamp": "2022-03-09T21:54:08Z",
"name": "test_data"
}
}
]
}
UPDATE: added test data so the point can be illustrated properly.

It's not entirely clear where the value for name in the output comes from in your question, but let's assume for a moment that it is the first part until the underscore of the name from the input. The the following produces your required output:
.items | map(
.metadata
| select(.random_field."foo/foo" == "true")
| { name: .name|split("_")|first, data: .name }
)
Output:
[
{
"name": "test",
"data": "test_data"
}
]

Related

How to select on existing (or non existing) key/value pair in sub structure

I have the following (simplified) JSON object:
{
"data": {
"items": [
{
"id": 2584,
"customProperties": [
{
"name": "jira.assetid",
"value": "1"
},
{
"name": "system.categories",
"value": "collector"
}
]
},
{
"id": 2603,
"customProperties": [
{
"name": "system.categories",
"value": "snmp"
}
]
},
{
"id": 2703,
"customProperties": [
{
"name": "jira.assetid",
"value": "5"
},
{
"name": "system.categories",
"value": "snmpTCPUDP,Netsnmp,snmpHR,snmp"
}
]
}
]
}
}
How can I filter all devices which have (or don't have) a "jira.assetid"?
My goal is to get 2 outputs:
No jira.assetid present: id, jira.assetid
Yes jira.assetid present: only id
The following gives me the id's of the devices with a jira.assetid, but how to get the assetid also on the output:
cat devices.txt | jq -r '.data.items[]
| select(.customProperties[].name == "jira.assetid")
|.id'
Here are at least two possible solutions, one first building a list of all id + assetid combinations, the other converts the custom properties to an object first.
Streaming solution:
.data.items[]
| {
id,
assetId: (.customProperties[] | select(.name == "jira.assetid").value)
}
Object solution:
.data.items[]
| {
id,
assetId: (.customProperties | from_entries."jira.assetid")
}
| select(.assetId)
Or using the special empty filter instead of selecting a second time:
.data.items[]
| {
id,
assetId: (.customProperties | from_entries."jira.assetid" // empty)
}
Output:
{
"id": 2584,
"assetId": "1"
}
{
"id": 2703,
"assetId": "5"
}
To get only the elements without an assetid:
.data.items[]
| select(any(.customProperties[]; .name == "jira.assetid") | not)
| { id }
or by building an object first, which I find slightly easier to grok:
.data.items[]
| select(.customProperties | from_entries | has("jira.assetid") | not)
| { id }
Output:
{
"id": 2603
}
To get only the ids for items without asset id:
.data.items[]
| select(.customProperties | from_entries | has("jira.assetid") | not)
| .id
Output:
2603

Bash get value of matching substring in a long string

I'm working on a bash script and having hard time extracting value from a matching substring.
I cannot share curl command as it has sensitive information but updating i value(1)(updated to mimic real value) that I'm having problem with. jq gives parse error
curl -s -g "$line" | jq -c '.allBuilds[]' | while read i; do
job_name=$(echo "$i" | jq .fullDisplayName | tr -d '»' | tr -s " " | sed 's/ /,/g' | tr -d '"')
done
expected output
Hello,Java,World,master,#47
I get expected output on most of the i values but some error out
Below are sample i values.
1
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.StringParameterValue","name":"environment","value":"DE"},{"_class":"hudson.model.BooleanParameterValue","name":"update","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"black","value":false},{"_class":"hudson.model.StringParameterValue","name":"description","value":"DE"},{"_class":"hudson.model.StringParameterValue","name":"number","value":""},{"_class":"hudson.model.StringParameterValue","name":"config","value":"{ "E": "DE", "Exp": "1111", "Pr": "D", "Man": { "Se": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Se-B8SKMz", "OR": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/OR-kvJ2lJ", "AR": "arn:aws:secretsmanager:region:111111111111:secret:A/DE/SA/rds", "User": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/mUixbWY", "sales": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/saELY", "vau": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/vNRR7BO", "sc": "", "exAd": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/ExyBoYL", "exp": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/Exl67GE", "sec": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Secle06a", "Secu": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Seia" }, "s3": { "buckets": { "hello": { "name": "helloDEBA", "region": "region", "account": "111111111111" }, "mlt": { "name": "sacdhbd", "region": "region", "account": "111111111111" }, "devo": { "name":"devvvvv", "region": "region", "account": "5555555" } } }, "roles": { "lam": "arn:aws:iam::111111111111:role/lam", "lambd": "arn:aws:iam::111111111111:role/lambd", "la": "arn:aws:iam::111111111111:role/lam", "la": "arn:aws:iam::111111111111:role/la", "la": "arn:aws:iam::111111111111:role/la","lasds": "arn:aws:iam::111111111111:role/lafgg", "lafdg": "arn:aws:iam::111111111111:role/dfsdv", "acc": "arn:aws:iam::111111111111:role/acc" }, "vpc": { "subnets": { "private": { "1a": "subnet-111111", "1b": "subnet-22222", "1c": "subnet-33333" } }, "securityGroupIds": { "lambda": "sg-1111" }, "endpoints": { "e": "" }, "links": { "b": "" } }, "securi": { "level": "FAILURE", "s": true }, "log": "debug", "se": "hello.com", "sa": { "env": "--DE" }, "lam": { "sss": { "environment": { "variables": { "test": "hello.com", "PhoneNumber": "11111" } } } } }"},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"ch","value":false}]},{"_class":"jenkins.scm.A.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello » Java » World » master #25","id":"25","number":25,"timestamp":1575582153372}
2
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.BooleanParameterValue","name":"helo-world","value":false},{"_class":"hudson.model.StringParameterValue","name":"environment","value":"hello"},{"_class":"hudson.model.StringParameterValue","name":"config","value":""},{"_class":"hudson.model.StringParameterValue","name":"description","value":""},{"_class":"hudson.model.BooleanParameterValue","name":"hello","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"hello2","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.StringParameterValue","name":"hello3","value":""}]},{"_class":"jenkins.scm.api.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello»Java»World»master#21","id":"21","number":21,"timestamp":1574705384077}
I'm trying to extract fullDisplayName value and I have tried some options like
printf '%s\n' "$i" | grep fullDisplayName
printf '%s\n' gives output in new lines when tried in shell but in script its different behavior
{}
{}
{}]
fullDisplayName:Hello » Java » World » master #25
id:25
number:25
timestamp:1575582153372
It would appear that you should be using read -r.
It also appears that it would be much simpler if you focused on using curl and jq to extract the information, without any grep or tr invocation and without any shell looping. Assuming you can arrange for the output of curl to be valid JSON(*), a single invocation of jq along the following lines should do the job:
jq -c '.allBuilds[] | .fullDisplayName | gsub("»";"")'
(*) To check whether the output of curl is valid, you can pipe the output of your curl command into jq empty:
curl ... | jq empty

Nested Filtering json file with jq statement

I have a json file with the following structure of each object inside
{
"id": 2400321267,
"data": {
"q": "quinoa black bean and shrimp r",
"r": "quinoa black bean and shrimps r",
"s": "3"
},
"job_id": 1413792,
"results": {
"judgments": [
{
"id": 5022700047,
"unit_state": "good",
"data": {
"rewrite_quality": "1"
},
}
],
}
},
{
"id": 2400321267,
"data": {
"q": "quinoa black bean and shrimp r",
"r": "quinoa black bean and shrimps r",
"s": "3"
},
"job_id": 1413792,
"results": {
"judgments": [
{
"id": 5022700047,
"unit_state": "good",
"data": {
"rewrite_quality": "2"
},
}
],
}
}
and I was trying to use the command jq '.[] | select(any(.Tags[]; .rewrite_quality == "1"))' | less to try to see if the output is correct but I don't see any output.
I want the output to have only entries with rewrite_quality == '1', in this case only the first entry.
Reading between the lines, it would appear that the following filter should achieve the stated goals:
.[]
| select( .results | any(.judgments[]; .data.rewrite_quality == "1"))
"Tags"
If the intent in using ".Tags" was to indicate that it does not matter what path leads to .rewrite_quality, then the filter to use would be:
.[]
| select( any(.. | objects | .rewrite_quality == "1"))
Alternative to using less
If you want a brief indication of whether there are any matches, you could use this filter, which has the added value of revealing how many objects satisfy the criterion:
map(select(any(.. | objects | .rewrite_quality == "1"))) | length

Delete key/value pair with jq when key is a known string

I'm trying to delete one specific key/value pair from a json file.
My json file is, for this example, params.json
[
{
"ParameterKey": "RTSMMinSize",
"ParameterValue": "1"
},
{
"ParameterKey": "RTSMReplicateDB",
"ParameterValue": "false"
},
{
"ParameterKey": "RTSMSnapshotID",
"ParameterValue": "snapID"
},
{
"ParameterKey": "RTSMEMAIL",
"ParameterValue": ""
}
]
I want to remove the RTSMSnapshotID key value pair entirely as part of my bash script. The file should look like this after:
[
{
"ParameterKey": "RTSMMinSize",
"ParameterValue": "1"
},
{
"ParameterKey": "RTSMReplicateDB",
"ParameterValue": "false"
},
{
"ParameterKey": "RTSMEMAIL",
"ParameterValue": ""
}
]
I thought this would be something as simple as
jq 'del(.RTSMSnapshotID)' params.json
but I'm getting
jq: error (at <filename>): Cannot index array with string "RTSMSnapshotID"
Clearly I don't understand how delete works. Any help?
del(.foo) expects there to be a top-level dictionary with a key named foo. That's not the case here; instead, you have a top-level list with ParameterKey having a series of values, with only one of which you want to remove the entire pair.
jq '[ .[] | select(.ParameterKey != "RTSMSnapshotID") ]'
...or...
jq 'map(select(.ParameterKey != "RTSMSnapshotID"))'

Replace an attribute or key in JSON using jq or sed

Have a big json like this
"envConfig": {
"environmentName": {
"versions": [
{
"name": "version1",
"value": "Dev"
},
{
"name": "version2",
"host": "qa"
}
],
"userRoles": [
{
"name": "Roles",
"entry": [
{
"name": "employees",
"value": "rwx"
},
{
"name": "customers",
"value": "rx"
}
]
}
]
}
},
I wanted to change the JSON attribute from "environmentName" to "prod". Below is the output i am expecting
"envConfig": {
"prod": {
"versions": [
...
],
"userRoles": [
...
]
}
}
Tried with sed command as below
sed "s/\('environmentName':\)/\1\"prod\"\,/g" version.json
Tried with jq as below but not working
cat version.json | jq ' with_entries(.value |= {"prod" : .environmentName} ) '
Any help here to replace the attribute/key of an json with desired value
You weren't too far off with the jq, how about this?
jq '.envConfig |= with_entries(.key |= sub("^environmentName$"; "prod"))'
Two differences: first off, we want to drill down to envConfig before doing a with_entries, and second, when we get there, the thing we want will be a key, not a value. In case there are any other keys besides environmentName they'll be preserved.
TL,TR
You can use the following command:
jq '(.envConfig |= (. + {"prod":.environmentName}|del(.environmentName)))' foo.json
Let's say you have the following json:
{
"foo": {
"hello" : "world"
}
}
You can rename the node foo to bar by first duplicating it and then remove the original node:
jq '. + {"bar":.foo}|del(.foo)' foo.json
Output:
{
"bar": {
"hello" : "world"
}
}
It get's a bit more complicated if you want to replace a child key somewhere in the tree. Let's say you have the following json:
{
"test": {
"foo": {
"hello": "world"
}
}
}
You can use the following jq command for that:
jq '(.test |= (. + {"bar":.foo}|del(.foo)))' foo.json
Note the additional parentheses and the use of the assignment operator |=.
Output:
{
"test": {
"bar": {
"hello": "world"
}
}
}
Using sed:
sed -i '/^ \"environmentName\":/ s/environmentName/prod/' <yourfile>
Keep in mind that -i will overwrite the file. You may want to make a backup first.

Resources