Editing GIST with cURL - bash

#!/bin/bash
COMMIT=$(git log -1 --pretty=format:'{"subject": "%s", "name": "xxx", "date": "%cD"}')
curl -X PATCH -d'{"files": {"latest-commit": {"content": "$COMMIT"}}}' -u user:xxxx https://api.github.com/gists/xxx
This just shows $COMMIT in the Gist. I tried playing with ''' and stuff but cannot make this work.

Your $COMMIT variable is not expanded to its value, because it is enclosed in single-quotes.
About an actual implementation in Bash
The GitHub API require you send the file content as a string: https://developer.github.com/v3/gists/#input-1
When file content contains newlines, double quotes or other characters needing an escaping within a string, the most appropriate shell tool to fill-in and escape the content string is jq.
JavaScript provide a JSON.stringify() method, but here in the shell world, we use jq to process JSON data.
If you don't have jq available you can convert the content of the file, to a properly escaped JSON string with GNU sed this way:
# compose the GitHub API JSON data payload
# to update the latest-commit.json file in the $gist_id
# uses sed to properly fill-in and escape the content string
read -r -d '' json_data_payload <<EOF
{
"description": "Updated from GitHub API call in Bash",
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"content": "$(
sed ':a;N;$!ba;s/\n/\\n/g;s/\r/\\r/g;s/\t/\\t/g;s/"/\\"/g;' <<<"$latest_commit_json_content"
)"
}
}
}
EOF
This is how jq is used to fill the content string with proper escaping:
json_data_payload="$(
jq \
--arg content "$latest_commit_json_content" \
--compact-output \
'.files."latest-commit.json".content = $content' \
<<'EOF'
{
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"content": ""
}
}
}
EOF
)"
Detailed and tested ok implementation:
#!/usr/bin/env bash
# Set to the gist id to update
gist_id='4b85f310233a6b9d385643fa3a889d92'
# Uncomment and set to your GitHub API OAUTH token
github_oauth_token='###################'
# Or uncomment this and set to your GitHub username:password
#github_user="user:xxxx"
github_api='https://api.github.com'
gist_description='Gist update with API call from a Bash script'
filename='latest-commit.json'
get_file_content() {
# Populate variables from the git log of latest commit
# reading null delimited strings for safety on special characters
{
read -r -d '' subject
read -r -d '' author
read -r -d '' date
} < <(
# null delimited subject, author, date
git log -1 --format=$'%s%x00%aN%x00%cD%x00'
)
# Compose the latest commit JSON, and populate it with the latest commit
# variables, using jq to ensure proper encoding and formatting of the JSON
read -r -d '' jquery <<'EOF'
.subject = $subject |
.author = $author |
.date = $date
EOF
jq \
--null-input \
--arg subject "$subject" \
--arg author "$author" \
--arg date "$date" \
"$jquery"
}
# compose the GitHub API JSON data payload
# to update the latest-commit.json file in the $gist_id
# uses jq to properly fill-in and escape the content string
# and compact the output before transmission
get_gist_update_json() {
read -r -d '' jquery <<'EOF'
.description = $description |
.files[$filename] |= (
.filename = $filename |
.content = $content
)
EOF
jq \
--null-input \
--compact-output \
--arg description "$gist_description" \
--arg filename "$filename" \
--arg content "$(get_file_content)" \
"$jquery"
}
# prepare the curl call with options for the GitHub API request
github_api_request=(
curl # The command to send the request
--fail # Return shell error if request unsuccessful
--request PATCH # The request type
--header "Content-Type: application/json" # The MIME type of the request
--data "$(get_gist_update_json)" # The payload content of the request
)
if [ -n "${github_oauth_token:-}" ]; then
github_api_request+=(
# Authenticate the GitHub API with a OAUTH token
--header "Authorization: token $github_oauth_token"
)
elif [ -n "${github_user:-}" ]; then
github_api_request+=(
# Authenticate the GitHub API with an HTTP auth user:pass
--user "$github_user"
)
else
echo 'GitHub API require either an OAUTH token or a user:pass' >&2
exit 1
fi
github_api_request+=(
-- # End of curl options
"$github_api/gists/$gist_id" # The GitHub API url to address the request
)
# perform the GitHub API request call
if ! "${github_api_request[#]}"; then
echo "Failed execution of:" >&2
env printf '%q ' "${github_api_request[#]}" >&2
echo >&2
fi
Here is the generated curl call with my token redacted out:
curl --fail --request PATCH --header 'Content-Type: application/json' \
--data '{"description":"Hello World Examples","files":{"latest-commit.json":{"filename":"latest-commit.json","content":"{\n \"subject\": \"depricate Phosphor\",\n \"name\": \"Blood Asp\",\n \"date\": \"Wed, 12 Dec 2018 18:55:39 +0100\"\n}"}}}' \
--header 'Authorization: token xxxx-redacted-xxxx' \
-- \
https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92
And the JSON response it replied with:
"url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92",
"forks_url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92/forks",
"commits_url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92/commits",
"id": "4b85f310233a6b9d385643fa3a889d92",
"node_id": "MDQ6R2lzdDRiODVmMzEwMjMzYTZiOWQzODU2NDNmYTNhODg5ZDky",
"git_pull_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92.git",
"git_push_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92.git",
"html_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92",
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"type": "application/json",
"language": "JSON",
"raw_url": "https://gist.githubusercontent.com/leagris/4b85f310233a6b9d385643fa3a889d92/raw/7cb7f9d4a0170daf5083929858fb7eef706f8b59/latest-commit.json",
"size": 105,
"truncated": false,
"content": "{\n \"subject\": \"depricate Phosphor\",\n \"name\": \"Blood Asp\",\n \"date\": \"Wed, 12 Dec 2018 18:55:39 +0100\"\n}"
}
},
...

Related

Sending a complex html as the body of email from the command line via an SendGrid API

I need to send an HTML file as the body of an eamil to several customers. Our company will be using SendGrid for this, and I need to be able to send the email via API Curl Call.
The way that I'm doing it so far works for simple html or plain text:
curl -s --request POST \
--url https://api.sendgrid.com/v3/mail/send \
--header "Authorization: Bearer SECRET_API_KEY" \
--header 'Content-Type: application/json' \
--data '{"personalizations":[{"to":[{"email":"my1#email.com"},{"email":"my2#email.com"}]}],"from":{"email":"info#somewhere.com"},"subject":"Testing sending emails via SendgridAPI","content":[{"type":"text\/html","value":"Test API Email From ME"}]}'
Now this works just fine. The problem is when I want to replace 'Test API Email From ME' with the contents of a rather large, complex HTML file. This has all the usual cli nightmares such as a mix of ' and " and new lines everywhere. I need to sanatize the HTML in order to accomplish three things:
The final result needs to be a valid command line string
The --data switch argument needs to remain a valid JSON enconded string
The HTML should not break.
What I do is I create the actual string command and the execute it using a scripting language. So I can perform any operation that I want on the html before inserting it in the value field of the content field. So my question is: what are the string operations that I should perform on the html so that I can send the email using this methodology?
Using jq and bash
I'll do it with static data, you may improve upon it
Define a JSON template for the API:
IFS='' read -r -d '' json_template <<'EOF'
{
"personalizations": [
{
"to": [
{ "email": "my1#email.com" },
{ "email": "my2#email.com" }
]
}
],
"from": { "email": "info#somewhere.com" },
"subject": "Testing sending emails via SendgridAPI",
"content": [
{
"type": "text/html",
"value": "Test API Email From ME"
}
]
}
EOF
Define the HTML content:
IFS='' read -r -d '' html_email <<'EOF'
<!doctype html>
<html>
<head>
title>Simple Email</title>
</head>
<body>
Test API Email From ME
</body
</html>
EOF
Replace the email content in the JSON with the HTML
json_data=$(
jq -c -n \
--arg html "$html_email" \
--argjson template "$json_template" \
'$template | .content[0].value = $html'
)
Send the query
curl -s --request POST \
--url https://api.sendgrid.com/v3/mail/send \
--header "Authorization: Bearer SECRET_API_KEY" \
--header 'Content-Type: application/json' \
--data "$json_data"
Here is how you can compose a proper JSON data payload with jq so it can be sent to the API.
jq will ensure every values, recipients, from, subject and the html body will be respectively encoded to proper JSON data objects, arrays and strings before it is submitted as --data #- to curl:
I added comments everywhere, so it is very clear what is done at every step:
#!/usr/bin/env bash
recipients=(
'my1#email.com'
'my2#email.com'
)
from='info#somewhere.com'
subject='Testing sending emails via SendgridAPI'
# Streams null-delimited recipients array entries
printf '%s\0' "${recipients[#]}" |
# jq slurps the null-delimited recipients,
# read the raw html content into the jq $contentHTML variable
# and integrate it all as a proper JSON
jq --slurp --raw-input --rawfile contentHTML example.html \
--arg from "$from" \
--arg subject "$subject" \
'
# Fills the jq $recipient JSON array variable
# by splitting the null-delmited entries
# from the incoming stream
split( "\u0000") as $recipients |
{
"personalizations": [
{
# Uses the $recipients array that has been
# slurped from the input stream
"to": $recipients
}
],
"from": {
# Use the $from that has been passed as --arg
"email": $from
},
# Use the $subject that has been passed as --arg
"subject": $subject,
"content": [
{
"type": "text/html",
"value": $contentHTML
}
]
}
' |
# Get the resultant JSON piped into curl
# that will read the data from the standard input
# using --data #-
# rather than passing it as an argument, because
# the payload could exceed the maximum length of arguments
curl -s --request POST \
--url https://api.sendgrid.com/v3/mail/send \
--header "Authorization: Bearer SECRET_API_KEY" \
--header 'Content-Type: application/json' \
--data #-

provide method for check status code with pass callback function in bash script

I faced with problem with organization code. I have api which returned jwt token with some expire lifetime and several api which protected by jwt token, I need call this api's during some time and jwt token can be expired, how to protect from this case ? Need develop some approach until http_code !=401 -> execute curls, else -> relogin. How to organization bash script for correct works ? Maybe need move some logic(aobut check status code 401) to method, how to pass callback function with some curl(for some api) to method ?
This what I did, looking at this, you can analyze it and made conclusion how it should works:
http_code='401'
successful_status_code='200'
auth='false'
while [ "$auth" != "true" ]
do
# in that [if] cheking status code if it 401, int case hwen token expired, need to fetch new token, by again execute api/v2/login
if [ "$http_code" != "$successful_status_code" ]
then
# this curl execute api with username and password from file, then
#response with `token` saving in `result.json`, from this file
#anotther #api will be fetch token for get access to the protected api
http_code=$(curl --request POST -sL \
--url 'http://qbee.local/api/v2/login'\
--output './result.json' \
--header "Content-Type: application/json" \
-d "#login.json" \
--write-out "%{http_code}")
else
auth='true'
# fetched token which was saved after api/v2/login curl
tokenValue=$(jq -r '.token' './result.json')
echo 'token was' + $tokenValue
# For get access to the potected api need to add header
# 'Authorization: Bearer some_token', then apireturn some data,
#but in case when toke expired need to execute `api/v2/login`
# this api curl saved response in file, that's it
http_code=$(curl --request GET -sL \
--url 'http://qbee.local/api/v2/grouptree?_format=json'\
--output './grouptree_result.json' \
--header 'Authorization: Bearer '"$tokenValue")
echo "$(cat grouptree_result.json)"
fi
done
How I imagine it, some bash script should executed several api during by some time period, so need to provide some structure which should be catch case when toke will be expire and relogin again, saving new tokne in file.
#some scenario where several api's should be call
statuscode = executeApi(fetchGroupTree())
statuscode = executeApi(fetchSomeData())
...
//etc methods which call api
#method which provide approach catch 401 and relogin user, refresh
#toke and then again execute current api method
public function executeApi(function api())
{
statuscode = execute api()
if (statuscode == 401) {
execute auth()
statuscode = execute api()
}
return statuscode
}
# another some api which expect not expire token for provide access
public function fetchSomeData()
{
tokenValue=$(jq -r '.token' './result.json')
echo 'token was' + $tokenValue
// execute some curl api
}
# some api which expect not expire token for provide access
public function fetchGroupTree()
{
tokenValue=$(jq -r '.token' './result.json')
echo 'token was' + $tokenValue
curl --request GET -sL \
--url 'http://qbee.local/api/v2/grouptree?_format=json'\
--output './grouptree_result.json' \
--header 'Authorization: Bearer '"$tokenValue" \
echo "$(cat grouptree_result.json)"
}
# method for relogin and saving new token in file after old token was expired
public function auth()
{
http_code=$(curl --request POST -sL \
--url 'http://qbee.local/api/v2/login'\
--output './result.json' \
--header "Content-Type: application/json" \
-d "#login.json" \
--write-out "%{http_code}")
}
But this is my vision, I'm not expert on bash scripts how usually develop bash script for this goal, maybe some best practices ?
You typically write some wrappers:
# the tokenfile location
tokenf=$(mktemp)
trap 'rm "$tokenf"' EXIT
# just some abstractions
get_token() {
cat "$tokenf"
}
has_token() {
[[ -s "$tokenf" ]]
}
clear_token() {
: > "$tokenf"
}
# the generic curl wrapper way we take
# remember to add -S so that you see errors
_curl() {
curl -sSL "$#"
}
# execute authentication request and place the result in token file
do_auth() {
local cmd
cmd=(
_curl --request POST
--url 'http://qbee.local/api/v2/login'
--output "$tokenf"
--header "Content-Type: application/json"
-d "#login.json"
# TODO: implement checking error code
# --write-out "%{http_code}"
)
"${cmd[#]}"
}
# small wrapper so that loop looks nice
# first argument - filename for output
# other arguments passed to curl
# outputs HTTP exit code
_execute_api_callback() {
local outf cmd
outf="$1"
shift
cmd=(
_curl --request GET
--header 'Authorization: Bearer '"$(get_token)"
--output "$outf" --write-out "%{http_code}"
"$#"
)
"${cmd[#]}"
}
# run curl request in a loop requesting for authentication
execute_api() {
# execute in a subshell for EXIT trap to auto-cleanup
(
# The temporary location for curl output
tmpf=$(mktemp)
trap 'rm "$tmpf"' EXIT
while
# TODO: check other error codes
statuscode=$(_execute_api_callback "$tmpf" "$#")
((statuscode != 200)
# TODO: implement timeout
do
do_auth # TODO: check failure
done
cat "$tmpf"
)
}
if ! has_token; then
do_auth
fi
# is --url really that needed?
execute_api 'http://qbee.local/api/v2/grouptree?_format=json'
This is just a code I have written in 5 mins on this forum - I did not test it, I did not run it. Check your code with https://shellcheck.net , read https://mywiki.wooledge.org/BashFAQ/005 and check out https://mywiki.wooledge.org/BashGuide/Parameters .

Adding Mailchimp members through Bash from .csv file

I have got around 1000 contacts to import to Mailchimp. This is my company's old database, which we have exported from the CSM system, and we want every contact to confirm their subscription if they want to be on our subscription list.
When I try to import it through Mailchimp, I can't give the contact status pending.
So, I have managed how to do it with single contact through bash, but I will want to import the whole contact list.
I am not familiar with this scripting language that much, so can anybody advise me, is there a way to import the data from the CSV file and how can I do it?
Or maybe there is some other way to do it?
This is the code that is working for a single contact:
#!/bin/bash
set -euo pipefail
list_id="Add_LIST_ID"
user_email="Add_E_MAIL"
user_fname="Add_F_NAME"
user_lname="Add_L_NAME"
curl -sS --request POST \
--url "https://$API_SERVER.api.mailchimp.com/3.0/lists/$list_id/members" \
--user "key:$API_KEY" \
--header 'content-type: application/json' \
--data #- \
<<EOF | jq '.id'
{
"email_address": "$user_email",
"status": "pending",
"merge_fields": {
"FNAME": "$user_fname",
"LNAME": "$user_lname"
}
}
EOF
EDIT1
Okay, I have managed to load the data from csv file. The code is below.
while IFS=, read -r col1
do
{
#!/bin/bash
set -euo pipefail
list_id="LIST_ID"
echo "$col1"
curl -sS --request POST \
--url "https://$API_SERVER.api.mailchimp.com/3.0/lists/$list_id/members" \
--user "key:$API_KEY" \
--header 'content-type: application/json' \
--data #- \
<<EOF | jq '.id'
{
"email_address": "$(echo $col1)",
"status": "pending",
"merge_fields": {
"FNAME": "",
"LNAME": ""
}
}
EOF
}
done < mails.csv
I have put echo line after list_id to see if the data is imported correctly.
The code is working (no errors in the buildup), but I have managed to add a contact to the list only once (subscriber hash is the response). In other tries, I have got a "null" value in response. Does anybody know why?

Shell script call API via curl and process response

I need to create a shell script that calls my login API via curl.
The script should be able to store and process the response from curl api call.
myscript.sh
#!/bin/bash
echo "Extract bearer token from curl calling login api"
echo
# Check cURL command if available (required), abort if does not exists
type curl >/dev/null 2>&1 || { echo >&2 "Required curl but it's not installed. Aborting."; exit 1; }
echo
PAYLOAD='{"email": "dummy-user#acme.com", "password": "secret"}'
curl -s --request POST -H "Content-Type:application/json" http://acme.com/api/authentications/login --data "${PAYLOAD}"
My problem in the given script is:
it does not get the response of curl calling the API.
From the response json, get only the token value.
Sample Login API response:
{
"user": {
"id": 123,
"token": "<GENERATED-TOKEN-HERE>",
"email": "dummy-user#acme.com",
"refreshToken": "<GENERATED-REFRESH-TOKEN>",
"uuid": "1239c226-8dd7-4edf-b948-df2f75508888"
},
"clientId": "abc12345",
"clientSecretKey": "thisisasecret"
}
I only need to get the value of token and store it in a variable... I will use token value in other curl api call as bearer token.
What do I need to change in my script to extract the token value from the response of curl api call?
Thanks!
Your curl statement has an error in it. You are executing it with the target URL as an header field:
curl --request POST -H "Content-Type:application/json" -H http://acme.com/api/authentications/login --data "${PAYLOAD}"
^
|
Remove this header flag
Also the silent -s flag helps when curl is executed from scripts:
-s, --silent
Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute.
Afterwards you could store the data in a variable and execute a regular expression on it to extract the token you need for further processing.
The complete script could look like the following:
#!/bin/bash
echo "Extract bearer token from curl calling login api"
echo
# Check cURL command if available (required), abort if does not exists
type curl >/dev/null 2>&1 || { echo >&2 "Required curl but it's not installed. Aborting."; exit 1; }
echo
PAYLOAD='{"email": "dummy-user#acme.com", "password": "secret"}'
RESPONSE=`curl -s --request POST -H "Content-Type:application/json" http://acme.com/api/authentications/login --data "${PAYLOAD}"`
TOKEN=`echo $RESPONSE | grep -Po '"token":(\W+)?"\K[a-zA-Z0-9._]+(?=")'`
echo "$TOKEN" # Use for further processsing
An alternate solution to parsing JSON with regex is jq :
echo '{ "user": { "id": 123, "token": "<GENERATED-TOKEN-HERE>", "email": "dummy-user#acme.com", "refreshToken": "<GENERATED-REFRESH-TOKEN>", "uuid": "1239c226-8dd7-4edf-b948-df2f75508888" }, "clientId": "abc12345", "clientSecretKey": "thisisasecret" }' | jq -r '.user.token'

"Invalid credentials" while doing a curl POST

I have a curl request in below format
curl -v -H "Content-Type:application/json" -H "x-user-id:xxx" -H "x-api-key:yyy" --data '{"logs":"'"${TEST_OUTPUT}"'","pass":"true | false"}' https://razeedash.one.qqq.cloud.com/api/v1/clusters/zzz/api/test_results
This works fine while I do from my MAC terminal. But the same command throws
13:49:26 {
13:49:26 "status": "error",
13:49:26 "message": "Invalid credentials"
13:49:26 }
I saw this post but not sure how else would I send a json body without curly braces. I know that we can save it as a file.json and use the file as body.But for some reasons that cannot be implemented in my scenario
In general, you should avoid trying to build JSON using string interpolation. Use a tool like jq to handle any necessary quoting.
jq -n --argson o "$TEST_OUTPUT" '{logs: $o, pass: "true | false"}' |
curl -v -H "Content-Type:application/json" \
-H "x-user-id:xxx" \
-H "x-api-key:yyy" \
--data #- \
https://razeedash.one.qqq.cloud.com/api/v1/clusters/zzz/api/test_results
However, if you can manage to correctly generate your JSON as you are now, you can just replace the jq command with echo:
echo '{"logs": ...' | curl ...
The #- argument to --data says to read from standard input.

Resources