creating a nested json output file using jq variables using - shell

I'm trying to create a json file via shellscript, and someone has mentioned jq, but I'm struggling a little bit to make it work:
The desire output is:
inboundurls{
"op": "add",
"path": "/support",
"apiSupports": [
{
"familyType": "EXAMPLE",
"healthCheckUris": "http://example.com"
}
],
"inboundurls": [
{
"healthCheckUris": "http://example.com"
}
]
}
Researching about I found a start point, but it's not working properly, I need some help, here is what I have:
script:
#!/bin/bash
apiSupports=$(jq -n --arg familyType EXAMPLE \
--arg healthCheckUris http://example.com \
'$ARGS.named'
)
final=$(jq -n --arg op "add" \
--arg path "/supportServices" \
--argjson apiSupports "[$apiSupports]" \
'$ARGS.named'
)
echo "$final"
the output of the script above:
{
"op": "add",
"path": "/supportServices",
"apiSupports": [
{
"familyType": "EXAMPLE",
"healthCheckUris": "http://example.com"
}
]
}
If anyone could help me I would be glad, or even suggesting Ideas, thank you in advance?

The following produces the valid JSON component of what is shown as the desired output:
jq -n --arg op "add" \
--arg path "/support" \
--arg familyType EXAMPLE \
--arg healthCheckUris http://example.com '
{$op, $path,
apiSupports: [ {$familyType, $healthCheckUris }],
inboundurls: [ {$healthCheckUris }]
}
'

Related

Download iCloud file from shared link using bash

I want to download a file from iCloud. I can share a link to a file. However the file is not directly linked in the urls, but the "real" download url can be retrieved:
#!/bin/bash
# given "https://www.icloud.com/iclouddrive/<ID>#<Filename>
ID="...."
URL=$(curl 'https://ckdatabasews.icloud.com/database/1/com.apple.cloudkit/production/public/records/resolve' \
--data-raw '{"shortGUIDs":[{"value":"$ID"}]}' --compressed \
jq -r '.results[0].rootRecord.fields.fileContent.value.downloadURL')
curl "$URL" -o myfile.ext
Sorce: https://gist.github.com/jpillora/702ded79330043e38e8202b5c73835e5
"fileContent" : {
"value" : {
...
"downloadURL" : "https://cvws.icloud-content.com/B/CYo..."
},
This is, however not working:
rl: (6) Could not resolve host: jq
curl: (3) nested brace in URL position 17:
{
"results" : [ {
"shortGUID" : {
"value" : "$ID",
"shouldFetchRootRecord" : true
},
"reason" : "shortGUID cannot be null or empty",
"serverErrorCode" : "BAD_REQUEST"
} ]
}
Any ideas, what I can do to make this work?
as #dan mentioned, jq is not a curl argument, its a separate command. hence you would need to | pipe it instead of \.
So the command would look something like this:
URL=$(curl 'https://ckdatabasews.icloud.com/database/1/com.apple.cloudkit/production/public/records/resolve' \ --data-raw '{"shortGUIDs":[{"value":"$ID"}]}' --compressed | jq -r '.results[0].rootRecord.fields.fileContent.value.downloadURL')
I solved it by installing jq and adding the ID directly instead of using $id. Just installing jq was not sufficient.
brew install jq
#!/bin/bash
# given "https://www.icloud.com/iclouddrive/<ID>#<Filename>
URL=$(curl 'https://ckdatabasews.icloud.com/database/1/com.apple.cloudkit/production/public/records/resolve' \
--data-raw '{"shortGUIDs":[{"value":"ID"}]}' --compressed | jq -r '.results[0].rootRecord.fields.fileContent.value.downloadURL')
Echo $url
curl "$URL" -o myfile.ext

jq: create array of object in json and insert the new object each time bash scripts executes [duplicate]

This question already has answers here:
Add new element to existing JSON array with jq
(3 answers)
Closed 3 years ago.
I want to create valid json using jq in bash.
each time when bash script will execute "Add new element to existing JSON array" and if file is empty create new file.
I am using following jq command to create my json (which is incomplete, please help me to complete it)
$jq -n -s '{service: $ARGS.named}' \
--arg transcationId $TRANSACTION_ID_METRIC '{"transcationId":"\($transcationId)"}' \
--arg name $REALPBPODDEFNAME '{"name ":"\($name )"}'\
--arg lintruntime $Cloudlintruntime '{"lintruntime":"\($lintruntime)"}' \
--arg status $EXITCODE '{"status":"\($status)"}' \
--arg buildtime $totaltime '{"buildtime":"\($buildtime)"}' >> Test.json
which is producing output like
{
"service": {
"transcationId": "12345",
"name": "sdsjkdjsk",
"lintruntime": "09",
"status": "0",
"buildtime": "9876"
}
}
{
"service": {
"transcationId": "123457",
"servicename": "sdsjkdjsk",
"lintruntime": "09",
"status": "0",
"buildtime": "9877"
}
}
but I don't want output in this format
json should be created first time like
what should be jq command for creating below jason
{
"ServiceData":{
"date":"30/1/2020",
"ServiceInfo":[
{
"transcationId":"20200129T130718Z",
"name":"MyService",
"lintruntime":"178",
"status":"0",
"buildtime":"3298"
}
]
}
}
and when I next time execute the bash script element should be added into the array like
what is the jq command for getting json in this format
{
"ServiceData":{
"date":"30/1/2020",
"ServiceInfo":[
{
"transcationId":"20200129T130718Z",
"name":"MyService",
"lintruntime":"16",
"status":"0",
"buildtime":"3256"
},
{
"transcationId":"20200129T130717Z",
"name":"MyService",
"lintruntime":"16",
"status":"0",
"buildtime":"3256"
}
]
}
}
also I want "date " , "service data" , "service info"
fields in my json which are missing in my current one
You don't give a separate filter to each --arg option; it just defines a variable which can be used in the single filter argument. You just want to add new object to your input. jq doesn't do in-place file editing, so you'll have to write to a temporary file and replace your original after the fact.
jq --arg transactionId "$TRANSACTION_ID_METRIC" \
--arg name "$REALPBPODDEFNAME" \
--arg lintruntime "$Cloudlintruntime" \
--arg status "$EXITCODE" \
--arg buildtime "$totaltime" \
'.ServiceData.ServiceInfo += [ {transactionID: $transactionId,
name: $name,
lintruntime: $lintruntime,
status: $status,
buildtime: $buildtime
}]' \
Test.json > tmp.json &&
mv tmp.json Test.json
Here's the same command, but using an array to store all the --arg options and a variable to store the filter so the command line is a little simpler. (You also don't need explicit line continuations inside an array definition.)
args=(
--arg transactionId "$TRANSACTION_ID_METRIC"
--arg name "$REALPBPODDEFNAME"
--arg lintruntime "$Cloudlintruntime"
--arg status "$EXITCODE"
--arg buildtime "$totaltime"
)
filter='.ServiceData.ServiceInfo += [
{
transactionID: $transactionId,
name: $name,
lintruntime: $lintruntime,
status: $status,
buildtime: $buildtime
}
]'
jq "${args[#]}" "$filter" Test.json > tmp.json && mv tmp.json Test.json

Editing GIST with cURL

#!/bin/bash
COMMIT=$(git log -1 --pretty=format:'{"subject": "%s", "name": "xxx", "date": "%cD"}')
curl -X PATCH -d'{"files": {"latest-commit": {"content": "$COMMIT"}}}' -u user:xxxx https://api.github.com/gists/xxx
This just shows $COMMIT in the Gist. I tried playing with ''' and stuff but cannot make this work.
Your $COMMIT variable is not expanded to its value, because it is enclosed in single-quotes.
About an actual implementation in Bash
The GitHub API require you send the file content as a string: https://developer.github.com/v3/gists/#input-1
When file content contains newlines, double quotes or other characters needing an escaping within a string, the most appropriate shell tool to fill-in and escape the content string is jq.
JavaScript provide a JSON.stringify() method, but here in the shell world, we use jq to process JSON data.
If you don't have jq available you can convert the content of the file, to a properly escaped JSON string with GNU sed this way:
# compose the GitHub API JSON data payload
# to update the latest-commit.json file in the $gist_id
# uses sed to properly fill-in and escape the content string
read -r -d '' json_data_payload <<EOF
{
"description": "Updated from GitHub API call in Bash",
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"content": "$(
sed ':a;N;$!ba;s/\n/\\n/g;s/\r/\\r/g;s/\t/\\t/g;s/"/\\"/g;' <<<"$latest_commit_json_content"
)"
}
}
}
EOF
This is how jq is used to fill the content string with proper escaping:
json_data_payload="$(
jq \
--arg content "$latest_commit_json_content" \
--compact-output \
'.files."latest-commit.json".content = $content' \
<<'EOF'
{
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"content": ""
}
}
}
EOF
)"
Detailed and tested ok implementation:
#!/usr/bin/env bash
# Set to the gist id to update
gist_id='4b85f310233a6b9d385643fa3a889d92'
# Uncomment and set to your GitHub API OAUTH token
github_oauth_token='###################'
# Or uncomment this and set to your GitHub username:password
#github_user="user:xxxx"
github_api='https://api.github.com'
gist_description='Gist update with API call from a Bash script'
filename='latest-commit.json'
get_file_content() {
# Populate variables from the git log of latest commit
# reading null delimited strings for safety on special characters
{
read -r -d '' subject
read -r -d '' author
read -r -d '' date
} < <(
# null delimited subject, author, date
git log -1 --format=$'%s%x00%aN%x00%cD%x00'
)
# Compose the latest commit JSON, and populate it with the latest commit
# variables, using jq to ensure proper encoding and formatting of the JSON
read -r -d '' jquery <<'EOF'
.subject = $subject |
.author = $author |
.date = $date
EOF
jq \
--null-input \
--arg subject "$subject" \
--arg author "$author" \
--arg date "$date" \
"$jquery"
}
# compose the GitHub API JSON data payload
# to update the latest-commit.json file in the $gist_id
# uses jq to properly fill-in and escape the content string
# and compact the output before transmission
get_gist_update_json() {
read -r -d '' jquery <<'EOF'
.description = $description |
.files[$filename] |= (
.filename = $filename |
.content = $content
)
EOF
jq \
--null-input \
--compact-output \
--arg description "$gist_description" \
--arg filename "$filename" \
--arg content "$(get_file_content)" \
"$jquery"
}
# prepare the curl call with options for the GitHub API request
github_api_request=(
curl # The command to send the request
--fail # Return shell error if request unsuccessful
--request PATCH # The request type
--header "Content-Type: application/json" # The MIME type of the request
--data "$(get_gist_update_json)" # The payload content of the request
)
if [ -n "${github_oauth_token:-}" ]; then
github_api_request+=(
# Authenticate the GitHub API with a OAUTH token
--header "Authorization: token $github_oauth_token"
)
elif [ -n "${github_user:-}" ]; then
github_api_request+=(
# Authenticate the GitHub API with an HTTP auth user:pass
--user "$github_user"
)
else
echo 'GitHub API require either an OAUTH token or a user:pass' >&2
exit 1
fi
github_api_request+=(
-- # End of curl options
"$github_api/gists/$gist_id" # The GitHub API url to address the request
)
# perform the GitHub API request call
if ! "${github_api_request[#]}"; then
echo "Failed execution of:" >&2
env printf '%q ' "${github_api_request[#]}" >&2
echo >&2
fi
Here is the generated curl call with my token redacted out:
curl --fail --request PATCH --header 'Content-Type: application/json' \
--data '{"description":"Hello World Examples","files":{"latest-commit.json":{"filename":"latest-commit.json","content":"{\n \"subject\": \"depricate Phosphor\",\n \"name\": \"Blood Asp\",\n \"date\": \"Wed, 12 Dec 2018 18:55:39 +0100\"\n}"}}}' \
--header 'Authorization: token xxxx-redacted-xxxx' \
-- \
https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92
And the JSON response it replied with:
"url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92",
"forks_url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92/forks",
"commits_url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92/commits",
"id": "4b85f310233a6b9d385643fa3a889d92",
"node_id": "MDQ6R2lzdDRiODVmMzEwMjMzYTZiOWQzODU2NDNmYTNhODg5ZDky",
"git_pull_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92.git",
"git_push_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92.git",
"html_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92",
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"type": "application/json",
"language": "JSON",
"raw_url": "https://gist.githubusercontent.com/leagris/4b85f310233a6b9d385643fa3a889d92/raw/7cb7f9d4a0170daf5083929858fb7eef706f8b59/latest-commit.json",
"size": 105,
"truncated": false,
"content": "{\n \"subject\": \"depricate Phosphor\",\n \"name\": \"Blood Asp\",\n \"date\": \"Wed, 12 Dec 2018 18:55:39 +0100\"\n}"
}
},
...

JQ query on JSON file

I am having below code in JSON file.
{
"comment": {
"vm-updates": [],
"site-ops-updates": [
{
"comment": {
"message": "You can start maintenance on this resource"
},
"hw-name": "Machine has got missing disks. "
}
]
},
"object_name": "4QXH862",
"has_problems": "yes",
"tags": ""
}
I want to separate "hw-name" from this JSON file using jq. I've tried below combinations, but nothing worked.
cat jsonfile | jq -r '.comment[].hw-name'
cat json_file.json | jq -r '.comment[].site-ops-updates[].hw-name'
Appreciated help from StackOverflow!!!
It should be:
▶ cat jsonfile | jq -r '.comment."site-ops-updates"[]."hw-name"'
Machine has got missing disks.
Or better still:
▶ jq -r '.comment."site-ops-updates"[]."hw-name"' jsonfile
Machine has got missing disks.
From the docs:
If the key contains special characters, you need to surround it with double quotes like this: ."foo$", or else .["foo$"].

Google Speech bash script with base64 : Unexpected token.\n

I use the following code : https://github.com/sararob/ml-talk-demos/blob/master/speech/request.sh
to fit my own bash script.
cat <<EOF > $JSONFILENAME
{
"config": {
"encoding":"LINEAR16",
"sampleRateHertz":8000,
"languageCode": "nl-NL",
"speechContexts": {
"phrases": ['']
},
"maxAlternatives": 1
},
"audio": {
"content":
}
}
EOF
base64 $1 -w 0 > $SOUNDFILE.base64
#MYBASE64=$(base64 $1 -w 0)
sed -i $JSONFILENAME -e "/\"content\":/r $SOUNDFILE.base64"
#sed -i $JSONFILENAME -e "/\"content\":/r $MYBASE64"
curl -s -X POST -H "Content-Type: application/json" --data-binary #${JSONFILENAME} https://speech.googleapis.com/v1/speech:recognize?key=$API_KEY
The base64 output is correctly filled in by the sed command, however there are also newlines added.
This is the Google API response :
{
"error": {
"code": 400,
"message": "Invalid JSON payload received. Unexpected token.\n\": {\n \"content\":\nUklGRqTIAgBXQVZFZm10\n ^",
"status": "INVALID_ARGUMENT"
}
}
How can I make sure the "content" in my JSON-object is a continuous string of base64 ?
You should avoid updating JSON data with sed.
If you have valid JSON data (i.e you have to fix the lines "phrases": [] and "content": "", you could use jq instead:
jq ".audio.content = \"$(base64 -w 0 "$1")\"" "$JSONFILENAME"
I don't recommend sed, but in this case where a large entry must be appended, you could try this:
echo \"$(base64 -w 0 "$1")\" > "$SOUNDFILE.base64"
sed -i "$JSONFILENAME" -e "/\"content\":/r $SOUNDFILE.base64"
The google error you receive is likely due to the fact that the string is not double quoted.

Resources