jq does not show null output - bash

I have the following code in the command line script:
output_json=$(jq -n \
--argjson ID "${id}" \
--arg Title "${title}" \
--argjson like "\"${like}\"" \
'$ARGS.named')
I put the id, title and like variables into the jq. I get the following output:
[
{
"ID": 6,
"Title": "ABC",
"like": ""
},
{
"ID": 22,
"Title": "ABC",
"like": "Yes"
}
]
But, I am trying to get the output in the following format, i.e. with null:
[
{
"ID": 6,
"Title": "ABC",
"like": null
},
{
"ID": 22,
"Title": "ABC",
"like": "Yes"
}
]
I don't quite get it is it possible to do this in general, or is it a problem with my jq command?
And as far as I understood "like": "" is not the same as "like": null. I am also a little confused now, and do not really understand what is the correct choice to use.

By using --argjson you need to provide valid JSON-encoded argument, thus if you want to receive null the value needs to be literally null. Your solution, however, adds quotes around it, so it can never be evaluated to null. (Also, it will only be a valid JSON string if it follows the JSON encoding for special characters such as the quote characters itself).
If you want to have a JSON string in the regular case, and null in the case where it is empty, import the content of ${like} as string using --arg and without the extra quotes (just as you do with ${title}), then use some jq logic to turn the empty string into null. An if statement would do, for example:
like=
jq -n --arg like "${like}" '{like: (if $like == "" then null else $like end)}'
{
"like": null
}

Related

How to extract data from a JSON file into a variable

I have the following json format, basically it is a huge file with several of such entries.
[
{
"id": "kslhe6em",
"version": "R7.8.0.00_BNK",
"hostname": "abacus-ap-hf-test-001:8080",
"status": "RUNNING",
},
{
"id": "2bkaiupm",
"version": "R7.8.0.00_BNK",
"hostname": "abacus-ap-hotfix-001:8080",
"status": "RUNNING",
},
{
"id": "rz5savbi",
"version": "R7.8.0.00_BNK",
"hostname": "abacus-ap-hf-test-005:8080",
"status": "RUNNING",
},
]
I wanted to fetch all the hostname values that starts with "abacus-ap-hf-test" and without ":8080" into a variable and then wanted to use those values for further commands over a for loop something like below. But, am bit confused how can I extract such informaion.
HOSTAME="abacus-ap-hf-test-001 abacus-ap-hf-test-005"
for HOSTANAME in $HOSTNAME
do
sh ./trigger.sh
done
The first line command update to this:
HOSTAME=$(grep -oP 'hostname": "\K(abacus-ap-hf-test[\w\d-]+)' json.file)
or if you sure that the hostname end with :8080", try this:
HOSTAME=$(grep -oP '(?<="hostname": ")abacus-ap-hf-test[\w\d-]+(?=:8080")' json.file)
you will find that abacus-ap-hf-test[\w\d-]+ is the regex, and other strings are the head or the end of the regex content which for finding result accuracy.
Assuming you have valid JSON, you can get the hostname values using jq:
while read -r hname ; do printf "%s\n" "$hname" ; done < <(jq -r .[].hostname j.json)
Output:
abacus-ap-hf-test-001:8080
abacus-ap-hotfix-001:8080
abacus-ap-hf-test-005:8080

Replace string with Bash variable in jq command

I realize this is a simple question but I haven't been able to find the answer. Thank you to anyone who may be able to help me understand what I am doing wrong.
Goal: Search and replace a string in a specific key in a JSON file with a string in a Bash variable using jq.
For example, in the following JSON file:
"name": "Welcome - https://docs.mysite.com/",
would become
"name": "Welcome",
Input (file.json)
[
{
"url": "https://docs.mysite.com",
"name": "Welcome - https://docs.mysite.com/",
"Ocurrences": "679"
},
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "382"
}
]
Failing script (using variable)
siteUrl="docs.mysite.com"
jq --arg siteUrl "$siteUrl" '.[].name|= (gsub(" - https://$siteUrl/"; ""))' file.json > file1.json`
Desired output (file1.json)
[
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "679"
},
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "382"
}
]
I've tried various iterations on removing quotes, changing between ' and ", and adding and removing backslashes.
Successful script (not using variable)
jq '.[].name|= (gsub(" - https://docs.mysite.com/"; ""))' file.json > file1.json
More specifically, if it matters, I am processing an export of a website's usage data from Azure App Insights. Unfortunately, the same page may be assigned different names. I sum the Ocurrences of the two objects with the newly identical url later. If it is better to fix this in App Insights I am grateful for that insight, although I know Bash better than Kusto queries. I am grateful for any help or direction.
Almost. Variables are not automatically expanded within a string. You must interpolate them explicitly with \(…):
jq --arg siteUrl 'docs.mysite.com' '.[].name |= (gsub(" - https://\($siteUrl)/"; ""))' file.json
Alternatively, detect a suffix match and extract the prefix by slicing:
jq --arg siteUrl 'docs.mysite.com' '" - https://\($siteUrl)/" as $suffix | (.[].name | select(endswith($suffix))) |= .[:$suffix|-length]' file.json

Get the key values using jq from json

I am looking for a way to find the full key path for given value taken from the variable. My input comes from the elasticsearch query result.
For example I want a full path to the key value: 9i6O4ERWWB
They key value is always unique and what only changes is the example.com and template1 keys (I cannot predict what will be the name).
Once knowing the key path:
_source.example.com.template1 I want to increment the "counter" field and update the elasticsearch document.
My input JSON:
{
"_index": "domains",
"_type": "doc",
"_id": "c66443eb1e6a0850b03a91fdb967f4d1",
"_score": 2.4877305,
"_source": {
"user_id": "c66443eb1e6a0850b03a91fdb967f4d1",
"statistics": {
"test_count": 0,
"datasize": 0,
"example.com": {
"template1": {
"image_id": "iPpDWbaO3YTIEb0pBkW3.png",
"link_id": "4ybOOUJpaBpDaLxPkz1j.html",
"counter": 0,
"subdomain_id": "9i6O4ERWWB"
},
"template2": {
"image_id": "iPpDWasdas322sdaW3.png",
"link_id": "4ybOOd3425sdfsz1j.html",
"counter": 1,
"subdomain_id": "432432sdxWWB"
}
},
"example1.com": {
"template1": {
"image_id": "iPpDWdasdasdasdas3.png",
"link_id": "4ybOOUadsasdadsasd1j.html",
"subdomain_id": "9i6O4ERWWB"
}
}
}
}
}
What I have tried was:
<myfile jq -c 'paths | select(.[-1])
<myfile jq -c 'paths | select(.[-1] == "subdomain_id")'
but this prints all apart the key values:
["_index"]
["_type"]
["_id"]
["_score"]
["_source"]
["_source","user_id"]
["_source","statistics"]
["_source","statistics","test_count"]
["_source","statistics","datasize"]
["_source","statistics","example.com"]
["_source","statistics","example.com","template1"]
["_source","statistics","example.com","template1","image_id"]
["_source","statistics","example.com","template1","link_id"]
["_source","statistics","example.com","template1","subdomain_id"]
["_source","statistics","template2"]
["_source","statistics","template2","image_id"]
["_source","statistics","template2","link_id"]
["_source","statistics","template2","subdomain_id"]
["_source","statistics","example1.com"]
["_source","statistics","example1.com","template1"]
["_source","statistics","example1.com","template1","image_id"]
["_source","statistics","example1.com","template1","link_id"]
["_source","statistics","example1.com","template1","subdomain_id"]
My pseudocode I am trying to write:
seeked_key_value="432432sdxWWB"
jq -n --arg seeked_key_value "$seeked_key_value" \
'paths | select(.[-1].$seeked_key_value'
Expected result: ["_source","statistics","example.com","template1","subdomain_id":"432432sdxWWB"]
Is this doable with jq in bash?
It's best to avoid grep in cases like this. To meet the exact requirements in the present case, one could write:
jq -c 'paths(scalars) as $p
| [$p, getpath($p)]
| select(.[1] == "9i6O4ERWWB")' input.json
If one really needs grep-like functionality, you can always use jq's test/1.
You can 'extract' paths using the following:
jq -c 'paths(scalars) as $p | [$p, getpath($p)]' file.json | grep 432432sdxWWB
and response is:
[["_source","statistics","example.com","template2","subdomain_id"],"432432sdxWWB"]
Possibly you can improve jq query to get only single value but I hope it helps you in determining final version :)

Invalid numeric literal when passing a truncated JSON object to jq

I have a response from cURL which looks like this:
{"username": "bot", "verified": true, "locale": "en-US", "mfa_enabled": false, "bot": true, "id": "123", "flags": 0, "avatar": null, "discriminator": "3114", "email": null} 200
which is stored in variable called auth
Then I want to be able to loop that object doing this:
response=$(jq -c "." <<< "${auth::-3}")
Note that I remove the last 3 characters because those are the status code.
So technically it should work, but it returns: parse error: Invalid numeric literal at line 1, column 11
If I enter the raw JSON as a string, it works. But not like this. Why?
Consider:
response=$(jq -n --argjson auth "${auth% *}" '$auth')
...which will work correctly with versions of bash too old to correctly support ${auth::-3} (a 4.x-only feature), and which will also log enough details to track down any issue caused by the content passed to jq when run with bash -x yourscript.

json processing in .sh files

[
{
"id": "1636ea48-28b7-783a-48dd-5e041f10d9e6",
"name": "Test_Component1",
"desiredVersions": [],
"children": false
},
{
"id": "1636f939-136f-4609-ab93-238b1af193fe",
"name": "Test_Component2",
"desiredVersions": [],
"children": false
}
]
I am writing command in Execute Shell window in Jenkins. I have this json in a variable. I want to extract both Id values so further processing in next set of command can be done.
Using jq:
$ echo "$var" | jq '.[].id'
"1636ea48-28b7-783a-48dd-5e041f10d9e6"
"1636f939-136f-4609-ab93-238b1af193fe"
is it a string? If so, you can use a regexp expression to extract the ID´s values. Like:
(\"id\"\:\W)\"(.+)(\"\,)

Resources