Google Speech bash script with base64 : Unexpected token.\n - bash

I use the following code : https://github.com/sararob/ml-talk-demos/blob/master/speech/request.sh
to fit my own bash script.
cat <<EOF > $JSONFILENAME
{
"config": {
"encoding":"LINEAR16",
"sampleRateHertz":8000,
"languageCode": "nl-NL",
"speechContexts": {
"phrases": ['']
},
"maxAlternatives": 1
},
"audio": {
"content":
}
}
EOF
base64 $1 -w 0 > $SOUNDFILE.base64
#MYBASE64=$(base64 $1 -w 0)
sed -i $JSONFILENAME -e "/\"content\":/r $SOUNDFILE.base64"
#sed -i $JSONFILENAME -e "/\"content\":/r $MYBASE64"
curl -s -X POST -H "Content-Type: application/json" --data-binary #${JSONFILENAME} https://speech.googleapis.com/v1/speech:recognize?key=$API_KEY
The base64 output is correctly filled in by the sed command, however there are also newlines added.
This is the Google API response :
{
"error": {
"code": 400,
"message": "Invalid JSON payload received. Unexpected token.\n\": {\n \"content\":\nUklGRqTIAgBXQVZFZm10\n ^",
"status": "INVALID_ARGUMENT"
}
}
How can I make sure the "content" in my JSON-object is a continuous string of base64 ?

You should avoid updating JSON data with sed.
If you have valid JSON data (i.e you have to fix the lines "phrases": [] and "content": "", you could use jq instead:
jq ".audio.content = \"$(base64 -w 0 "$1")\"" "$JSONFILENAME"
I don't recommend sed, but in this case where a large entry must be appended, you could try this:
echo \"$(base64 -w 0 "$1")\" > "$SOUNDFILE.base64"
sed -i "$JSONFILENAME" -e "/\"content\":/r $SOUNDFILE.base64"
The google error you receive is likely due to the fact that the string is not double quoted.

Related

variables in curl api requests

I have been playing around with a api and I am able to request data and parse the json response.
However, I am using variables in my requests and those are giving me issues.
If the variable contains no spaces then all works great, but if there are spaces then I get failures.
This is how I am doing it
search="string with space"
curl -s 'https://apiurl.com' \
> -d 'search "'$search'";' \
> -H 'user-key: xxxxxxxxxxx' \
> -H 'Accept: application/json'
[
{
"title": "Syntax Error",
"status": 400,
"cause": "Mismatched input, double check your input. Common cause is sending \\\" instead of \"."
}
]
Can anyone explain why this happens?
If I use the string with spaces in the request, not using the variable, then it works no problem.
Classic case of the QUOTATION-MARK-19 virus. :)
Just replace 'search "'$search'";' with 'search "'"$search"'";'
All you were missing was an extra double quote to allow for the variable expansion of $search
search="string with space"
curl -s 'https://apiurl.com' \
> -d 'search "'"$search"'";' \
> -H 'user-key: xxxxxxxxxxx' \
> -H 'Accept: application/json'
[
{
"title": "Syntax Error",
"status": 400,
"cause": "Mismatched input, double check your input. Common cause is sending \\\" instead of \"."
}
]

Editing GIST with cURL

#!/bin/bash
COMMIT=$(git log -1 --pretty=format:'{"subject": "%s", "name": "xxx", "date": "%cD"}')
curl -X PATCH -d'{"files": {"latest-commit": {"content": "$COMMIT"}}}' -u user:xxxx https://api.github.com/gists/xxx
This just shows $COMMIT in the Gist. I tried playing with ''' and stuff but cannot make this work.
Your $COMMIT variable is not expanded to its value, because it is enclosed in single-quotes.
About an actual implementation in Bash
The GitHub API require you send the file content as a string: https://developer.github.com/v3/gists/#input-1
When file content contains newlines, double quotes or other characters needing an escaping within a string, the most appropriate shell tool to fill-in and escape the content string is jq.
JavaScript provide a JSON.stringify() method, but here in the shell world, we use jq to process JSON data.
If you don't have jq available you can convert the content of the file, to a properly escaped JSON string with GNU sed this way:
# compose the GitHub API JSON data payload
# to update the latest-commit.json file in the $gist_id
# uses sed to properly fill-in and escape the content string
read -r -d '' json_data_payload <<EOF
{
"description": "Updated from GitHub API call in Bash",
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"content": "$(
sed ':a;N;$!ba;s/\n/\\n/g;s/\r/\\r/g;s/\t/\\t/g;s/"/\\"/g;' <<<"$latest_commit_json_content"
)"
}
}
}
EOF
This is how jq is used to fill the content string with proper escaping:
json_data_payload="$(
jq \
--arg content "$latest_commit_json_content" \
--compact-output \
'.files."latest-commit.json".content = $content' \
<<'EOF'
{
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"content": ""
}
}
}
EOF
)"
Detailed and tested ok implementation:
#!/usr/bin/env bash
# Set to the gist id to update
gist_id='4b85f310233a6b9d385643fa3a889d92'
# Uncomment and set to your GitHub API OAUTH token
github_oauth_token='###################'
# Or uncomment this and set to your GitHub username:password
#github_user="user:xxxx"
github_api='https://api.github.com'
gist_description='Gist update with API call from a Bash script'
filename='latest-commit.json'
get_file_content() {
# Populate variables from the git log of latest commit
# reading null delimited strings for safety on special characters
{
read -r -d '' subject
read -r -d '' author
read -r -d '' date
} < <(
# null delimited subject, author, date
git log -1 --format=$'%s%x00%aN%x00%cD%x00'
)
# Compose the latest commit JSON, and populate it with the latest commit
# variables, using jq to ensure proper encoding and formatting of the JSON
read -r -d '' jquery <<'EOF'
.subject = $subject |
.author = $author |
.date = $date
EOF
jq \
--null-input \
--arg subject "$subject" \
--arg author "$author" \
--arg date "$date" \
"$jquery"
}
# compose the GitHub API JSON data payload
# to update the latest-commit.json file in the $gist_id
# uses jq to properly fill-in and escape the content string
# and compact the output before transmission
get_gist_update_json() {
read -r -d '' jquery <<'EOF'
.description = $description |
.files[$filename] |= (
.filename = $filename |
.content = $content
)
EOF
jq \
--null-input \
--compact-output \
--arg description "$gist_description" \
--arg filename "$filename" \
--arg content "$(get_file_content)" \
"$jquery"
}
# prepare the curl call with options for the GitHub API request
github_api_request=(
curl # The command to send the request
--fail # Return shell error if request unsuccessful
--request PATCH # The request type
--header "Content-Type: application/json" # The MIME type of the request
--data "$(get_gist_update_json)" # The payload content of the request
)
if [ -n "${github_oauth_token:-}" ]; then
github_api_request+=(
# Authenticate the GitHub API with a OAUTH token
--header "Authorization: token $github_oauth_token"
)
elif [ -n "${github_user:-}" ]; then
github_api_request+=(
# Authenticate the GitHub API with an HTTP auth user:pass
--user "$github_user"
)
else
echo 'GitHub API require either an OAUTH token or a user:pass' >&2
exit 1
fi
github_api_request+=(
-- # End of curl options
"$github_api/gists/$gist_id" # The GitHub API url to address the request
)
# perform the GitHub API request call
if ! "${github_api_request[#]}"; then
echo "Failed execution of:" >&2
env printf '%q ' "${github_api_request[#]}" >&2
echo >&2
fi
Here is the generated curl call with my token redacted out:
curl --fail --request PATCH --header 'Content-Type: application/json' \
--data '{"description":"Hello World Examples","files":{"latest-commit.json":{"filename":"latest-commit.json","content":"{\n \"subject\": \"depricate Phosphor\",\n \"name\": \"Blood Asp\",\n \"date\": \"Wed, 12 Dec 2018 18:55:39 +0100\"\n}"}}}' \
--header 'Authorization: token xxxx-redacted-xxxx' \
-- \
https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92
And the JSON response it replied with:
"url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92",
"forks_url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92/forks",
"commits_url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92/commits",
"id": "4b85f310233a6b9d385643fa3a889d92",
"node_id": "MDQ6R2lzdDRiODVmMzEwMjMzYTZiOWQzODU2NDNmYTNhODg5ZDky",
"git_pull_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92.git",
"git_push_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92.git",
"html_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92",
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"type": "application/json",
"language": "JSON",
"raw_url": "https://gist.githubusercontent.com/leagris/4b85f310233a6b9d385643fa3a889d92/raw/7cb7f9d4a0170daf5083929858fb7eef706f8b59/latest-commit.json",
"size": 105,
"truncated": false,
"content": "{\n \"subject\": \"depricate Phosphor\",\n \"name\": \"Blood Asp\",\n \"date\": \"Wed, 12 Dec 2018 18:55:39 +0100\"\n}"
}
},
...

Shell script call API via curl and process response

I need to create a shell script that calls my login API via curl.
The script should be able to store and process the response from curl api call.
myscript.sh
#!/bin/bash
echo "Extract bearer token from curl calling login api"
echo
# Check cURL command if available (required), abort if does not exists
type curl >/dev/null 2>&1 || { echo >&2 "Required curl but it's not installed. Aborting."; exit 1; }
echo
PAYLOAD='{"email": "dummy-user#acme.com", "password": "secret"}'
curl -s --request POST -H "Content-Type:application/json" http://acme.com/api/authentications/login --data "${PAYLOAD}"
My problem in the given script is:
it does not get the response of curl calling the API.
From the response json, get only the token value.
Sample Login API response:
{
"user": {
"id": 123,
"token": "<GENERATED-TOKEN-HERE>",
"email": "dummy-user#acme.com",
"refreshToken": "<GENERATED-REFRESH-TOKEN>",
"uuid": "1239c226-8dd7-4edf-b948-df2f75508888"
},
"clientId": "abc12345",
"clientSecretKey": "thisisasecret"
}
I only need to get the value of token and store it in a variable... I will use token value in other curl api call as bearer token.
What do I need to change in my script to extract the token value from the response of curl api call?
Thanks!
Your curl statement has an error in it. You are executing it with the target URL as an header field:
curl --request POST -H "Content-Type:application/json" -H http://acme.com/api/authentications/login --data "${PAYLOAD}"
^
|
Remove this header flag
Also the silent -s flag helps when curl is executed from scripts:
-s, --silent
Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute.
Afterwards you could store the data in a variable and execute a regular expression on it to extract the token you need for further processing.
The complete script could look like the following:
#!/bin/bash
echo "Extract bearer token from curl calling login api"
echo
# Check cURL command if available (required), abort if does not exists
type curl >/dev/null 2>&1 || { echo >&2 "Required curl but it's not installed. Aborting."; exit 1; }
echo
PAYLOAD='{"email": "dummy-user#acme.com", "password": "secret"}'
RESPONSE=`curl -s --request POST -H "Content-Type:application/json" http://acme.com/api/authentications/login --data "${PAYLOAD}"`
TOKEN=`echo $RESPONSE | grep -Po '"token":(\W+)?"\K[a-zA-Z0-9._]+(?=")'`
echo "$TOKEN" # Use for further processsing
An alternate solution to parsing JSON with regex is jq :
echo '{ "user": { "id": 123, "token": "<GENERATED-TOKEN-HERE>", "email": "dummy-user#acme.com", "refreshToken": "<GENERATED-REFRESH-TOKEN>", "uuid": "1239c226-8dd7-4edf-b948-df2f75508888" }, "clientId": "abc12345", "clientSecretKey": "thisisasecret" }' | jq -r '.user.token'

JQ query on JSON file

I am having below code in JSON file.
{
"comment": {
"vm-updates": [],
"site-ops-updates": [
{
"comment": {
"message": "You can start maintenance on this resource"
},
"hw-name": "Machine has got missing disks. "
}
]
},
"object_name": "4QXH862",
"has_problems": "yes",
"tags": ""
}
I want to separate "hw-name" from this JSON file using jq. I've tried below combinations, but nothing worked.
cat jsonfile | jq -r '.comment[].hw-name'
cat json_file.json | jq -r '.comment[].site-ops-updates[].hw-name'
Appreciated help from StackOverflow!!!
It should be:
▶ cat jsonfile | jq -r '.comment."site-ops-updates"[]."hw-name"'
Machine has got missing disks.
Or better still:
▶ jq -r '.comment."site-ops-updates"[]."hw-name"' jsonfile
Machine has got missing disks.
From the docs:
If the key contains special characters, you need to surround it with double quotes like this: ."foo$", or else .["foo$"].

String addition on shell [duplicate]

I want to send a json request and embedd a variable in the post data.
I did a little research and I came up with the single quotes around the variable.
#!/bin/bash
FILENAME="/media/file.avi"
curl -i -X POST -H "Content-Type: application/json" —d '{"jsonrpc": "2.0", "method": "Player.Open", "params":{"item":{"file":"'$FILENAME'"}}}' http://192.167.0.13/jsonrpc
Unfortunately I get some errors:
curl: (6) Couldn't resolve host '—d'
curl: (3) [globbing] nested braces not supported at pos 54
HTTP/1.1 200 OK
Content-Length: 76
Content-Type: application/json
Date: Wed, 29 Jan 2014 19:16:56 GMT
{"error":{"code":-32700,"message":"Parse error."},"id":null,"jsonrpc":"2.0"}
Appearently there are some problems with the braces and the http answer states, that the command could not be executed. What's wrong with my code here?
Thanks!
This is my curl version:
curl 7.30.0 (mips-unknown-linux-gnu) libcurl/7.30.0 OpenSSL/0.9.8y
Protocols: file ftp ftps http https imap imaps pop3 pop3s rtsp smtp smtps tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL
Update: use the simpler
request_body=$(cat <<EOF
{
"jsonrpc": "2.0",
"method": "Player.Open",
"params": {
"item": {
"file": "$FILENAME"
}
}
}
EOF
)
rather than what I explain below. However, if it is an option, use jq to generate the JSON instead. This ensures that the value of $FILENAME is properly quoted.
request_body=$(jq -n --arg fname "$FILENAME" '
{
jsonrpc: "2.0",
method: "Player.Open",
params: {item: {file: $fname}}
}'
It would be simpler to define a variable with the contents of the request body first:
#!/bin/bash
header="Content-Type: application/json"
FILENAME="/media/file.avi"
request_body=$(< <(cat <<EOF
{
"jsonrpc": "2.0",
"method": "Player.Open",
"params": {
"item": {
"file": "$FILENAME"
}
}
}
EOF
))
curl -i -X POST -H "$header" -d "$request_body" http://192.167.0.13/jsonrpc
This definition might require an explanation to understand, but note two big benefits:
You eliminate a level of quoting
You can easily format the text for readability.
First, you have a simple command substitution that reads from a file:
$( < ... ) # bash improvement over $( cat ... )
Instead of a file name, though, you specify a process substitution, in which the output of a command is used as if it were the body of a file.
The command in the process substitution is simply cat, which reads from a here document. It is the here document that contains your request body.
My suggestion:
#!/bin/bash
FILENAME="/media/file 2.avi"
curl -i -X POST -H "Content-Type: application/json" -d '{"jsonrpc": "2.0", "method": "Player.Open", "params":{"item":{"file":"'"$FILENAME"'"}}}' http://192.167.0.13/jsonrpc
The differences are hyphen in -d (instead of a dash) and double quotes around $FILENAME.
Here is another way to insert data from a file into a JSON property.
This solution is based on a really cool command called jq.
Below is an example which prepares request JSON data, used to create a CoreOS droplet on Digital Ocean:
# Load the cloud config to variable
user_data=$(cat config/cloud-config)
# Prepare the request data
request_data='{
"name": "server name",
"region": "fra1",
"size": "512mb",
"image": "coreos-stable",
"backups": false,
"ipv6": true,
"user_data": "---this content will be replaced---",
"ssh_keys": [1234, 2345]
}'
# Insert data from file into the user_data property
request_data=$(echo $request_data | jq ". + {user_data: \"$user_data\"}")

Resources