Linux Mint 20.2
Here report.empty.json
{
"total": 0,
"project": "",
"Severity": [],
"issues": []
}
I want to set value = 500 (int value) to "total" and "MY_PROJECT".
To do this I use tool "jq"
Here by bash script file:
#!/bin/bash
readonly PROJECT_KEY=MY_PROJECT
readonly PAGE_SIZE=500
jq --arg totalArg "$PAGE_SIZE" '.total = $totalArg' report.empty.json > report.json
jq --arg projectKey "${PROJECT_KEY}" '.project = $projectKey' report.empty.json > report.json
echo "Done"
But it set only key project. The key total is not changed.
Content of file report.json
{
"total": 0,
"project": "MY_PROJECT",
"Severity": [],
"issues": []
}
But I need to update BOTH KEYS.
The result must be:
{
"total": 500,
"project": "MY_PROJECT",
"Severity": [],
"issues": []
}
The second command reads from report.empty.json instead of the already-modified report.json.
You could chain the jq
jq --arg totalArg "$PAGE_SIZE" '.total = $totalArg' report.empty.json |
jq --arg projectKey "${PROJECT_KEY}" '.project = $projectKey' >report.json
But a better solution is to use just use one command.
jq --arg totalArg "$PAGE_SIZE" --arg projectKey "$PROJECT_KEY" '
.total = $totalArg | .project = $projectKey
' report.empty.json >report.json
My proposal for How to populate JSON values, using jq
Thinking about How to process arrays using jq, here is my modified version of your script. (Of course, you could keep empty.json out of script)...
#!/bin/bash
declare -r projectKey=MY_PROJECT
declare -ir pageSize=500
declare -a issueList=()
declare -i issueCnt=0
declare issueStr='' jqCmd='.project = $projArg | .total = $totArg | .issues=[ '
declare promptMessage='Enter issue (or [return] if none): '
while read -rp "$promptMessage" issue && [ "$issue" ];do
promptMessage='Enter next issue (or [return] if no more): '
issueCnt+=1
issueList+=(--arg is$issueCnt "$issue")
issueStr+="\$is$issueCnt, "
done
jqCmd+="${issueStr%, } ]"
jq --arg totArg "$pageSize" --arg projArg "$projectKey" \
"${issueList[#]}" "( $jqCmd )" <<-EoEmptyJson
{
"total": 0,
"project": "",
"Severity": [],
"issues": []
}
EoEmptyJson
Sample run (I want to add two issues):
./reportJson
Enter issue (or [return] if none): Foo
Enter next issue (or [return] if no more): Bar Baz
Enter next issue (or [return] if no more):
{
"total": "500",
"project": "MY_PROJECT",
"Severity": [],
"issues": [
"Foo",
"Bar Baz"
]
}
No answer (so far) accounts for the requirement that total be of type int. This can be accomplished by using --argjson instead of --arg. Here's my two cents:
jq --argjson total 500 --arg project "MY_PROJECT" '. + {$total, $project}' report.json
{
"total": 500,
"project": "MY_PROJECT",
"Severity": [],
"issues": []
}
Related
I have the following code in the command line script:
output_json=$(jq -n \
--argjson ID "${id}" \
--arg Title "${title}" \
--argjson like "\"${like}\"" \
'$ARGS.named')
I put the id, title and like variables into the jq. I get the following output:
[
{
"ID": 6,
"Title": "ABC",
"like": ""
},
{
"ID": 22,
"Title": "ABC",
"like": "Yes"
}
]
But, I am trying to get the output in the following format, i.e. with null:
[
{
"ID": 6,
"Title": "ABC",
"like": null
},
{
"ID": 22,
"Title": "ABC",
"like": "Yes"
}
]
I don't quite get it is it possible to do this in general, or is it a problem with my jq command?
And as far as I understood "like": "" is not the same as "like": null. I am also a little confused now, and do not really understand what is the correct choice to use.
By using --argjson you need to provide valid JSON-encoded argument, thus if you want to receive null the value needs to be literally null. Your solution, however, adds quotes around it, so it can never be evaluated to null. (Also, it will only be a valid JSON string if it follows the JSON encoding for special characters such as the quote characters itself).
If you want to have a JSON string in the regular case, and null in the case where it is empty, import the content of ${like} as string using --arg and without the extra quotes (just as you do with ${title}), then use some jq logic to turn the empty string into null. An if statement would do, for example:
like=
jq -n --arg like "${like}" '{like: (if $like == "" then null else $like end)}'
{
"like": null
}
I'm having a json file sample.json. Below is the snippet from the sample.json -
{
"AddOnModules": {
"Description": "add on modules",
"Type": "Array",
"AllowedValues": [
"a",
"b",
"c"
],
"value": []
}
}
I'm trying to provide value to AddOnModules through git ci variable (parameter-value) at runtime while running the pipeline. The following is the snippet of the pipeline -
stages:
- deploy
# Job to deploy for development
dev-deploy:
variables:
before_script:
- apk add jq
image: python:3.7.4-alpine3.9
script:
- tmp=$(mktemp)
- jq -r --arg add_on_modules "$add_on_modules" '.AddOnModules.value |= .+ [$add_on_modules] ' sample.json > "$tmp" && mv "$tmp" sample.json
- cat sample.json
stage: deploy
tags:
- docker
- linux
only:
variables:
- $stage =~ /^deploy$/ && $deployment_mode =~ /^dev$/
I'm giving the value of variable add_on_modules as "a","b" through git ci while running the pipeline. On performing cat sample.json, it's observed to be -
{
"AddOnModules": {
"Description": "add on modules",
"Type": "Array",
"AllowedValues": [
"a",
"b",
"c"
],
"value": [ "\"a\",\"b\""]
}
}
The extra double quotes are getting the prepended and appended while the existing ones are escaped.
I want output something like -
{
"AddOnModules": {
"Description": "add on modules",
"Type": "Array",
"AllowedValues": [
"a",
"b",
"c"
],
"value": ["a","b"]
}
}
Looks like I'm missing something with jq -
- jq -r --arg add_on_modules "$add_on_modules" '.AddOnModules.value |= .+ [$add_on_modules] ' sample.json > "$tmp" && mv "$tmp" sample.json
Tried with -r/--raw-output flag with jq but no success. Any suggestions on how to solve this?
This is how I'm running the pipeline -
Pipeline run
If $add_on_modules is ["a","b"]
If you can set add_on_modules to the textual representation of a JSON array, then you would use --argjson, like so:
add_on_modules='["a","b"]'
jq -r --argjson add_on_modules "$add_on_modules" '
.AddOnModules.value += $add_on_modules
' sample.json
If $add_on_modules is the string "a","b"
add_on_modules='"a","b"'
jq -r --arg add_on_modules "$add_on_modules" '
.AddOnModules.value += ($add_on_modules|split(",")|map(fromjson))
' sample.json
I am looking for a way to find the full key path for given value taken from the variable. My input comes from the elasticsearch query result.
For example I want a full path to the key value: 9i6O4ERWWB
They key value is always unique and what only changes is the example.com and template1 keys (I cannot predict what will be the name).
Once knowing the key path:
_source.example.com.template1 I want to increment the "counter" field and update the elasticsearch document.
My input JSON:
{
"_index": "domains",
"_type": "doc",
"_id": "c66443eb1e6a0850b03a91fdb967f4d1",
"_score": 2.4877305,
"_source": {
"user_id": "c66443eb1e6a0850b03a91fdb967f4d1",
"statistics": {
"test_count": 0,
"datasize": 0,
"example.com": {
"template1": {
"image_id": "iPpDWbaO3YTIEb0pBkW3.png",
"link_id": "4ybOOUJpaBpDaLxPkz1j.html",
"counter": 0,
"subdomain_id": "9i6O4ERWWB"
},
"template2": {
"image_id": "iPpDWasdas322sdaW3.png",
"link_id": "4ybOOd3425sdfsz1j.html",
"counter": 1,
"subdomain_id": "432432sdxWWB"
}
},
"example1.com": {
"template1": {
"image_id": "iPpDWdasdasdasdas3.png",
"link_id": "4ybOOUadsasdadsasd1j.html",
"subdomain_id": "9i6O4ERWWB"
}
}
}
}
}
What I have tried was:
<myfile jq -c 'paths | select(.[-1])
<myfile jq -c 'paths | select(.[-1] == "subdomain_id")'
but this prints all apart the key values:
["_index"]
["_type"]
["_id"]
["_score"]
["_source"]
["_source","user_id"]
["_source","statistics"]
["_source","statistics","test_count"]
["_source","statistics","datasize"]
["_source","statistics","example.com"]
["_source","statistics","example.com","template1"]
["_source","statistics","example.com","template1","image_id"]
["_source","statistics","example.com","template1","link_id"]
["_source","statistics","example.com","template1","subdomain_id"]
["_source","statistics","template2"]
["_source","statistics","template2","image_id"]
["_source","statistics","template2","link_id"]
["_source","statistics","template2","subdomain_id"]
["_source","statistics","example1.com"]
["_source","statistics","example1.com","template1"]
["_source","statistics","example1.com","template1","image_id"]
["_source","statistics","example1.com","template1","link_id"]
["_source","statistics","example1.com","template1","subdomain_id"]
My pseudocode I am trying to write:
seeked_key_value="432432sdxWWB"
jq -n --arg seeked_key_value "$seeked_key_value" \
'paths | select(.[-1].$seeked_key_value'
Expected result: ["_source","statistics","example.com","template1","subdomain_id":"432432sdxWWB"]
Is this doable with jq in bash?
It's best to avoid grep in cases like this. To meet the exact requirements in the present case, one could write:
jq -c 'paths(scalars) as $p
| [$p, getpath($p)]
| select(.[1] == "9i6O4ERWWB")' input.json
If one really needs grep-like functionality, you can always use jq's test/1.
You can 'extract' paths using the following:
jq -c 'paths(scalars) as $p | [$p, getpath($p)]' file.json | grep 432432sdxWWB
and response is:
[["_source","statistics","example.com","template2","subdomain_id"],"432432sdxWWB"]
Possibly you can improve jq query to get only single value but I hope it helps you in determining final version :)
[
{
"id": "1636ea48-28b7-783a-48dd-5e041f10d9e6",
"name": "Test_Component1",
"desiredVersions": [],
"children": false
},
{
"id": "1636f939-136f-4609-ab93-238b1af193fe",
"name": "Test_Component2",
"desiredVersions": [],
"children": false
}
]
I am writing command in Execute Shell window in Jenkins. I have this json in a variable. I want to extract both Id values so further processing in next set of command can be done.
Using jq:
$ echo "$var" | jq '.[].id'
"1636ea48-28b7-783a-48dd-5e041f10d9e6"
"1636f939-136f-4609-ab93-238b1af193fe"
is it a string? If so, you can use a regexp expression to extract the ID´s values. Like:
(\"id\"\:\W)\"(.+)(\"\,)
I am trying to simply output a list of all instance IDs that do not follow a particular tagging convention.
Tag is missing (Tag Keys: Environment or Finance)
Environment Tag value is not one of (prod, stg, test, dev)
Finance Tag value is not one of (GroupA , GroupB)
For (1) I can use the following:
aws ec2 describe-instances --output json --query 'Reservations[*].Instances[?!not_null(Tags[?Key==`Environment`].Value)] | [].InstanceId'
[
"i-12345678901234567",
"i-76543210987654321"
]
But I still need (2) and (3). What if the tag exists but is empty, or has a typo in the value?
"ec2 --query" functionality is limited and I've yet to find a way for it to get me (2) or (3), especially when it comes to inverting results.
I've gone back and forth trying to
modify the output from the CLI to make it easier to parse in JQ
VS
trying to wrangle the output in JQ
For (2) and (3). Here's a pair of outputs from the CLI that I've tried sending to JQ to parse with sample output for 2 instances:
CLI Sample Output [A] Tag.Value and Tag.Key need to be paired when searching, and then negating/inverting a set of searches...
aws ec2 describe-instances --output json --query 'Reservations[].Instances[].{ID:InstanceId, Tag: Tags[]}' | jq '.[]'
{
"Tag": [
{
"Value": "GroupA",
"Key": "Finance"
},
{
"Value": "stg",
"Key": "Environment"
},
{
"Value": "true",
"Key": "Backup"
},
{
"Value": "Another Server",
"Key": "Name"
}
],
"ID": "i-87654321"
}
{
"Tag": [
{
"Value": "GroupB",
"Key": "Finance"
},
{
"Value": "Server 1",
"Key": "Name"
},
{
"Value": "true",
"Key": "Backup"
},
{
"Value": "stg",
"Key": "Environment"
}
],
"ID": "i-12345678"
}
CLI Sample Output [B] Tag value being inside an array has been enough to trigger syntax errors when attempting things like "jq map" or "jq select"
aws ec2 describe-instances --output json --query 'Reservations[].Instances[].{ID:InstanceId, EnvTag: Tags[?Key==`Environment`].Value, FinTag: Tags[?Key==`Finance`].Value}' | jq '.[]'
{
"EnvTag": [
"stg"
],
"ID": "i-87654321",
"FinTag": [
"GroupA"
]
}
{
"EnvTag": [
"stg"
],
"ID": "i-12345678",
"FinTag": [
"GroupB"
]
}
I find most of the time, when I try to expand some solution from a simpler use case, I only ever end up with cryptic syntax errors due to some oddity in the structure of my incoming dataset.
Example Issue 1
Below is an example of how the inverting / negating fails. This is using CLI output B:
aws ec2 describe-instances --output json --query 'Reservations[].Instances[].{ID:InstanceId, EnvTag: Tags[?Key==`Environment`].Value, FinTag: Tags[?Key==`Finance`].Value}' | jq '.[]' | jq 'select(.EnvTag[] | contains ("prod", "dev") | not)'
I would expect the above to return everything except prod and dev. But it looks like the logic is inverted on each item as opposed to the set of contains:
"!A + !B" instead of "!(A or B)"
The resulting dataset returned is a list of everything, including dev and prod.
Example Issue 1.5
I can workaround the logic issue by chaining the contain excludes, but then I discover that "contains" won't work for me as it will pickup typos that still happen to contain the string in question:
aws ec2 describe-instances --output json --query 'Reservations[].Instances[].{ID:InstanceId, EnvTag: Tags[?Key==`Environment`].Value, FinTag: Tags[?Key==`Finance`].Value}' | jq '.[]' | jq 'select(.EnvTag[] | contains ("dev") | not) | select(.EnvTag[] | contains ("stg") | not) | select(.EnvTag[] | contains ("test") | not) | select(.EnvTag[] | contains ("prod") | not) | select (.EnvTag[] | contains ("foo") | not)'
prod != production
"prod" contains("prod") = true
"production" contains ("prod") = true <-- bad :(
I believe I've found a solution.
It can be greatly simplified. First, in this case, there is no need to invoke jq twice. jq '.[]' | jq ... is equivalent to jq '.[] | ...'
Second, the long pipeline of 'select' filters can be condensed, for example to:
select(.EnvTag[]
| (. != "dev" and . != "stg" and . != "prod" and . != "test" and . != "ops"))
or, if your jq has all/2, even more concisely to:
select( . as $in | all( ("dev", "stg", "prod", "test", "ops"); . != $in.EnvTag[]) )
I believe I've found a solution. It may not be optimal, but I've found a way to pipe-chain excludes of exact strings:
aws ec2 describe-instances --output json --query 'Reservations[].Instances[].{ID:InstanceId, EnvTag: Tags[?Key==`Environment`].Value, FinTag: Tags[?Key==`Finance`].Value}' | jq '.[]' | jq 'select(.EnvTag[] != "dev") | select (.EnvTag[] != "stg") | select (.EnvTag[] != "prod") | select (.EnvTag[] != "test") | select (.EnvTag[] != "ops") | .ID'
I verified this by changing an environment tag from "ops" to "oops".
Upon running this query, it returned the single instance with the oops tag.