Posting to Twitter account using Windows .bat file code - bash

I'm trying to setup an automatic tweet script that runs after a git commit. I'm using Windows 7 have curl available in the command line.
I'm not sure how to set variables with the language windows scripts run and also am not positive about the oauth process.
I have the api key and secret, and also the consumer key and secret but I'm just not sure how to wrap it all together.
Here is a paper clipped mashup of code I'm trying to use as a foundation:
#!/bin/sh
# PATH modification needed for http_post and oauth_sign
export PATH=$PATH:/usr/local/bin
toplevel_path=`git rev-parse --show-toplevel`
toplevel_dir=`basename "$toplevel_path"`
branch=`git rev-parse --abbrev-ref HEAD`
subject=`git log --pretty=format:%s -n1`
hashtags="#code #$toplevel_dir"
tweet=$hashtags' ['$branch']: "'$subject'"'
# truncate tweets that are longer than 140 characters
if [ ${#tweet} -gt 140 ]
then
tweet_trunc=$(echo $tweet | cut -c1-137)
tweet=${tweet_trunc}...
fi
//set vars
consumer_key="mPijnvYpD0sHAY8r*******"
consumer_secret="OWuvuyQeYrT3ToJgyvNdR6baNuDldmTDF5IIJCI************"
access_token="2476143012-ld78CrgnNY3kUmD0QRdvIchXeDC13nO3********"
access_secret="3HTdOlf8jCVzPi5I9usV7rIbGFtM5f****************"
//build oauth
//post data
//example curl code found during research
curl --request 'POST' 'https://api.twitter.com/1.1/statuses/update.json' --header 'Authorization: OAuth oauth_consumer_key="mPijnvYpD0sHAY8r6fkox0KBj", oauth_nonce="OWuvuyQeYrT3ToJgyvNdR6baNuDldmTDF5IIJCIablQbyHA2PS", oauth_signature="Ba6IB8uH2SjtrK8a%2FgZnqCgvIKs%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1346207448", oauth_token="14814762-vvYtBOLX8hBAQ0i0f1k4wxrioG1jOk49MJrqn3myE", oauth_version="1.0"' --verbose -F "media[]=#mack.jpg" -F "status=Test from cURL" --header "Expect: "
Any help at all is appreciated.

Bro, on Windows you should use PowerShell now. .bat is lame!
$toplevel_path = git rev-parse --show-toplevel
$toplevel_dir = Split-Path $toplevel_path -Leaf
$branch = git rev-parse --abbrev-ref HEAD
$subject = git log --pretty=format:%s -n1
$hashtags = "#code #$toplevel_dir"
$tweet = '{0} [{1}]: "{2}"' -f $hashtags, $branch, $subject
if ($tweet.length -gt 140) {
$tweet = $tweet.substring(0,137)
}
$oauths =
'oauth_consumer_key="mPijnvYpD0sHAY8r6fkox0KBj"',
'oauth_nonce="OWuvuyQeYrT3ToJgyvNdR6baNuDldmTDF5IIJCIablQbyHA2PS"',
'oauth_signature="Ba6IB8uH2SjtrK8a%2FgZnqCgvIKs%3D"',
'oauth_signature_method="HMAC-SHA1"',
'oauth_timestamp="1346207448"',
'oauth_token="14814762-vvYtBOLX8hBAQ0i0f1k4wxrioG1jOk49MJrqn3myE"',
'oauth_version="1.0"'
$header = 'Authorization: OAuth {0}' -f ($oauths -join ',')
curl --verbose --request POST -F 'media[]=#mack.jpg' `
-F 'status=Test from cURL' --header 'Expect: ' `
--header $header https://api.twitter.com/1.1/statuses/update.json
command substitution is actually better than with Bash
PowerShell "printf" is cool
arrays are cool
the continuation character ` is too small though, hard to see

Related

provide method for check status code with pass callback function in bash script

I faced with problem with organization code. I have api which returned jwt token with some expire lifetime and several api which protected by jwt token, I need call this api's during some time and jwt token can be expired, how to protect from this case ? Need develop some approach until http_code !=401 -> execute curls, else -> relogin. How to organization bash script for correct works ? Maybe need move some logic(aobut check status code 401) to method, how to pass callback function with some curl(for some api) to method ?
This what I did, looking at this, you can analyze it and made conclusion how it should works:
http_code='401'
successful_status_code='200'
auth='false'
while [ "$auth" != "true" ]
do
# in that [if] cheking status code if it 401, int case hwen token expired, need to fetch new token, by again execute api/v2/login
if [ "$http_code" != "$successful_status_code" ]
then
# this curl execute api with username and password from file, then
#response with `token` saving in `result.json`, from this file
#anotther #api will be fetch token for get access to the protected api
http_code=$(curl --request POST -sL \
--url 'http://qbee.local/api/v2/login'\
--output './result.json' \
--header "Content-Type: application/json" \
-d "#login.json" \
--write-out "%{http_code}")
else
auth='true'
# fetched token which was saved after api/v2/login curl
tokenValue=$(jq -r '.token' './result.json')
echo 'token was' + $tokenValue
# For get access to the potected api need to add header
# 'Authorization: Bearer some_token', then apireturn some data,
#but in case when toke expired need to execute `api/v2/login`
# this api curl saved response in file, that's it
http_code=$(curl --request GET -sL \
--url 'http://qbee.local/api/v2/grouptree?_format=json'\
--output './grouptree_result.json' \
--header 'Authorization: Bearer '"$tokenValue")
echo "$(cat grouptree_result.json)"
fi
done
How I imagine it, some bash script should executed several api during by some time period, so need to provide some structure which should be catch case when toke will be expire and relogin again, saving new tokne in file.
#some scenario where several api's should be call
statuscode = executeApi(fetchGroupTree())
statuscode = executeApi(fetchSomeData())
...
//etc methods which call api
#method which provide approach catch 401 and relogin user, refresh
#toke and then again execute current api method
public function executeApi(function api())
{
statuscode = execute api()
if (statuscode == 401) {
execute auth()
statuscode = execute api()
}
return statuscode
}
# another some api which expect not expire token for provide access
public function fetchSomeData()
{
tokenValue=$(jq -r '.token' './result.json')
echo 'token was' + $tokenValue
// execute some curl api
}
# some api which expect not expire token for provide access
public function fetchGroupTree()
{
tokenValue=$(jq -r '.token' './result.json')
echo 'token was' + $tokenValue
curl --request GET -sL \
--url 'http://qbee.local/api/v2/grouptree?_format=json'\
--output './grouptree_result.json' \
--header 'Authorization: Bearer '"$tokenValue" \
echo "$(cat grouptree_result.json)"
}
# method for relogin and saving new token in file after old token was expired
public function auth()
{
http_code=$(curl --request POST -sL \
--url 'http://qbee.local/api/v2/login'\
--output './result.json' \
--header "Content-Type: application/json" \
-d "#login.json" \
--write-out "%{http_code}")
}
But this is my vision, I'm not expert on bash scripts how usually develop bash script for this goal, maybe some best practices ?
You typically write some wrappers:
# the tokenfile location
tokenf=$(mktemp)
trap 'rm "$tokenf"' EXIT
# just some abstractions
get_token() {
cat "$tokenf"
}
has_token() {
[[ -s "$tokenf" ]]
}
clear_token() {
: > "$tokenf"
}
# the generic curl wrapper way we take
# remember to add -S so that you see errors
_curl() {
curl -sSL "$#"
}
# execute authentication request and place the result in token file
do_auth() {
local cmd
cmd=(
_curl --request POST
--url 'http://qbee.local/api/v2/login'
--output "$tokenf"
--header "Content-Type: application/json"
-d "#login.json"
# TODO: implement checking error code
# --write-out "%{http_code}"
)
"${cmd[#]}"
}
# small wrapper so that loop looks nice
# first argument - filename for output
# other arguments passed to curl
# outputs HTTP exit code
_execute_api_callback() {
local outf cmd
outf="$1"
shift
cmd=(
_curl --request GET
--header 'Authorization: Bearer '"$(get_token)"
--output "$outf" --write-out "%{http_code}"
"$#"
)
"${cmd[#]}"
}
# run curl request in a loop requesting for authentication
execute_api() {
# execute in a subshell for EXIT trap to auto-cleanup
(
# The temporary location for curl output
tmpf=$(mktemp)
trap 'rm "$tmpf"' EXIT
while
# TODO: check other error codes
statuscode=$(_execute_api_callback "$tmpf" "$#")
((statuscode != 200)
# TODO: implement timeout
do
do_auth # TODO: check failure
done
cat "$tmpf"
)
}
if ! has_token; then
do_auth
fi
# is --url really that needed?
execute_api 'http://qbee.local/api/v2/grouptree?_format=json'
This is just a code I have written in 5 mins on this forum - I did not test it, I did not run it. Check your code with https://shellcheck.net , read https://mywiki.wooledge.org/BashFAQ/005 and check out https://mywiki.wooledge.org/BashGuide/Parameters .

How to upload an asset from Azure Release pipeline to Github Enterprise using Bash

Since the company I work for will not allow any non-Microsoft released packages from the Azure Marketplace, I need to create a bash script to be able to upload my built/published asset to the Github Enterprise repo. For some reason I always get redirected when I try to get the info of a release tag, be it a specific tag or latest, does not matter.
The following is my script:
set -e
xargs=$(which gxargs || which xargs)
# Validate settings.
[ "$TRACE" ] && set -x
CONFIG=$#
for line in $CONFIG; do
eval "$line"
done
# Define variables.
GH_API="https://git.[company].com/api/v3"
GH_REPO="$GH_API/repos/[owner]/$(Build.Repository.Name)"
GH_TAGS="$GH_REPO/releases/tags/$(Build.SourceBranchName)"
AUTH="Authorization: token $github_api_token"
WGET_ARGS="--content-disposition --auth-no-challenge --no-cookie"
CURL_ARGS="-LJO#"
tag="$(Build.SourceBranchName)"
filename="BaseRepoName_$(Build.SourceBranchName)_$(Build.BuildId).zip"
echo "tag is: $tag"
echo $AUTH
echo "Repo: $GH_REPO"
if [[ "$tag" == 'LATEST' ]]; then
GH_TAGS="$GH_REPO/releases/latest"
fi
echo "Tags url: $GH_TAGS"
echo "Validate token ..."
# Validate token.
curl -o /dev/null -sH "$AUTH" $GH_REPO || { echo "Error: Invalid repo, token or network issue!"; exit 1; }
echo "Get api endpoints"
apiresponse=$(curl -sH "$AUTH" "$GH_API")
echo "API: $apiresponse"
echo "Read asset tags: curl -sH "$AUTH" $GH_TAGS"
# Read asset tags.
response=$(curl -sH "$AUTH" $GH_TAGS)
echo "Response: $response"
## In case of success, we upload a file
upload=$(echo $succ | grep upload_url)
if [[ $? -eq 0 ]]; then
echo Release created.
else
echo Error creating release!
return
fi
echo "Get the upload url for the given tag"
upload=$(echo $upload | cut -d "\"" -f4 | cut -d "{" -f1)
upload="$upload?name=$filename"
# Upload asset
echo "Uploading asset... "
succ=$(curl -H "Authorization: token $perstok" \
-H "Content-Type: $(file -b --mime-type $filename)" \
--data-binary #$filename $upload)
The logs of the Bash task:
2020-03-13T05:53:35.4274045Z ##[section]Starting: Bash Script
2020-03-13T05:53:35.4654068Z ==============================================================================
2020-03-13T05:53:35.4654773Z Task : Bash
2020-03-13T05:53:35.4655254Z Description : Run a Bash script on macOS, Linux, or Windows
2020-03-13T05:53:35.4655535Z Version : 3.163.1
2020-03-13T05:53:35.4656320Z Author : Microsoft Corporation
2020-03-13T05:53:35.4656864Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/bash
2020-03-13T05:53:35.4657312Z ==============================================================================
2020-03-13T05:53:36.4699984Z Generating script.
2020-03-13T05:53:36.4703019Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc -c pwd
2020-03-13T05:53:36.8704279Z /d/a/_temp
2020-03-13T05:53:36.8755504Z
2020-03-13T05:53:36.8794400Z ========================== Starting Command Output ===========================
2020-03-13T05:53:36.8801355Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc /d/a/_temp/c7a40af4-c1f2-4127-a2cf-4aa11ac19e48.sh
2020-03-13T05:53:36.9965374Z which: no gxargs in (/mingw64/bin:/usr/bin:/c/Users/VssAdministrator/bin:/c/hostedtoolcache/windows/Python/3.6.8/x64:/c/hostedtoolcache/windows/Python/3.6.8/x64/Scripts:/c/Program Files/Mercurial:/c/ProgramData/kind:/c/vcpkg:/c/cf-cli:/c/Program Files (x86)/NSIS:/c/Program Files/Mercurial:/c/hostedtoolcache/windows/Boost/1.69.0:/c/Program Files/dotnet:/c/mysql-5.7.21-winx64/bin:/c/Program Files/Java/zulu-8-azure-jdk_8.40.0.25-8.0.222-win_x64/bin:/c/SeleniumWebDrivers/GeckoDriver:/c/Program Files (x86)/sbt/bin:/c/Rust/.cargo/bin:/c/hostedtoolcache/windows/Ruby/2.5.7/x64/bin:/c/Go1.12.7/bin:/bin:/c/hostedtoolcache/windows/Python/3.6.8/x64/Scripts:/c/hostedtoolcache/windows/Python/3.6.8/x64:/c/npm/prefix:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/c/Program Files/Microsoft MPI/Bin:/c/windows/system32:/c/windows:/c/windows/System32/Wbem:/c/windows/System32/WindowsPowerShell/v1.0:/c/ProgramData/Chocolatey/bin:/c/Program Files/Docker:/c/Program Files/PowerShell/7:/c/Program Files/dotnet:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/110/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/150/DTS/Binn:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/c/Program Files/nodejs:/c/Strawberry/c/bin:/c/Strawberry/perl/site/bin:/c/Strawberry/perl/bin:/cmd:/mingw64/bin:/usr/bin:/c/tools/php:/c/Program Files (x86)/sbt/bin:/c/Program Files (x86)/Subversion/bin:/c/SeleniumWebDrivers/ChromeDriver:/c/SeleniumWebDrivers/EdgeDriver:/c/ProgramData/chocolatey/lib/maven/apache-maven-3.6.3/bin:/c/Program Files/CMake/bin:/c/Program Files/OpenSSL/bin:/c/Users/VssAdministrator/.dotnet/tools:/c/Program Files (x86)/Microsoft SQ)
2020-03-13T05:53:37.0042653Z tag is: MBP_TestTag6
2020-03-13T05:53:37.0043260Z Authorization: token [Edited: secret]
2020-03-13T05:53:37.0043852Z Repo: https://git.[company].com/api/v3/repos/[owner]/[MyTestRepo]
2020-03-13T05:53:37.0044723Z Tags url: https://git.[company].com/api/v3/repos/[owner]/[MyTestRepo]/releases/tags/MBP_TestTag6
2020-03-13T05:53:37.0045218Z Validate token ...
2020-03-13T05:53:38.5703717Z Get api endpoints
2020-03-13T05:53:38.8038551Z API: {
2020-03-13T05:53:38.8050142Z "current_user_url": "https://git.[company].com/api/v3/user",
2020-03-13T05:53:38.8053206Z "current_user_authorizations_html_url": "https://git.[company].com/settings/connections/applications{/client_id}",
2020-03-13T05:53:38.8060689Z "authorizations_url": "https://git.[company].com/api/v3/authorizations",
2020-03-13T05:53:38.8064874Z "code_search_url": "https://git.[company].com/api/v3/search/code?q={query}{&page,per_page,sort,order}",
2020-03-13T05:53:38.8071647Z "commit_search_url": "https://git.[company].com/api/v3/search/commits?q={query}{&page,per_page,sort,order}",
2020-03-13T05:53:38.8073059Z "emails_url": "https://git.[company].com/api/v3/user/emails",
2020-03-13T05:53:38.8074445Z "emojis_url": "https://git.[company].com/api/v3/emojis",
2020-03-13T05:53:38.8075767Z "events_url": "https://git.[company].com/api/v3/events",
2020-03-13T05:53:38.8077032Z "feeds_url": "https://git.[company].com/api/v3/feeds",
2020-03-13T05:53:38.8078149Z "followers_url": "https://git.[company].com/api/v3/user/followers",
2020-03-13T05:53:38.8079345Z "following_url": "https://git.[company].com/api/v3/user/following{/target}",
2020-03-13T05:53:38.8080572Z "gists_url": "https://git.[company].com/api/v3/gists{/gist_id}",
2020-03-13T05:53:38.8081548Z "hub_url": "https://git.[company].com/api/v3/hub",
2020-03-13T05:53:38.8082418Z "issue_search_url": "https://git.[company].com/api/v3/search/issues?q={query}{&page,per_page,sort,order}",
2020-03-13T05:53:38.8083431Z "issues_url": "https://git.[company].com/api/v3/issues",
2020-03-13T05:53:38.8084430Z "keys_url": "https://git.[company].com/api/v3/user/keys",
2020-03-13T05:53:38.8086272Z "label_search_url": "https://git.[company].com/api/v3/search/labels?q={query}&repository_id={repository_id}{&page,per_page}",
2020-03-13T05:53:38.8091275Z "notifications_url": "https://git.[company].com/api/v3/notifications",
2020-03-13T05:53:38.8092166Z "organization_repositories_url": "https://git.[company].com/api/v3/orgs/{org}/repos{?type,page,per_page,sort}",
2020-03-13T05:53:38.8093095Z "organization_url": "https://git.[company].com/api/v3/orgs/{org}",
2020-03-13T05:53:38.8096487Z "public_gists_url": "https://git.[company].com/api/v3/gists/public",
2020-03-13T05:53:38.8097620Z "rate_limit_url": "https://git.[company].com/api/v3/rate_limit",
2020-03-13T05:53:38.8098584Z "repository_url": "https://git.[company].com/api/v3/repos/{owner}/{repo}",
2020-03-13T05:53:38.8100945Z "repository_search_url": "https://git.[company].com/api/v3/search/repositories?q={query}{&page,per_page,sort,order}",
2020-03-13T05:53:38.8102239Z "current_user_repositories_url": "https://git.[company].com/api/v3/user/repos{?type,page,per_page,sort}",
2020-03-13T05:53:38.8104230Z "starred_url": "https://git.[company].com/api/v3/user/starred{/owner}{/repo}",
2020-03-13T05:53:38.8104831Z "starred_gists_url": "https://git.[company].com/api/v3/gists/starred",
2020-03-13T05:53:38.8105328Z "team_url": "https://git.[company].com/api/v3/teams",
2020-03-13T05:53:38.8105772Z "user_url": "https://git.[company].com/api/v3/users/{user}",
2020-03-13T05:53:38.8106279Z "user_organizations_url": "https://git.[company].com/api/v3/user/orgs",
2020-03-13T05:53:38.8106871Z "user_repositories_url": "https://git.[company].com/api/v3/users/{user}/repos{?type,page,per_page,sort}",
2020-03-13T05:53:38.8107491Z "user_search_url": "https://git.[company].com/api/v3/search/users?q={query}{&page,per_page,sort,order}"
2020-03-13T05:53:38.8108556Z }
2020-03-13T05:53:38.8109344Z Read asset tags: curl -sH Authorization: token [Edited: Secret] https://git.[company].com/api/v3/repos/[owner]/[MyTestRepo]/releases/tags/MBP_TestTag6
2020-03-13T05:53:39.0378180Z Response: {
2020-03-13T05:53:39.0379040Z "message": "Not Found",
2020-03-13T05:53:39.0379599Z "documentation_url": "https://developer.github.com/enterprise/2.20/v3/repos/releases/#get-a-release-by-tag-name"
2020-03-13T05:53:39.0380100Z }
2020-03-13T05:53:39.2992296Z
2020-03-13T05:53:39.3060923Z ##[error]Bash exited with code '1'.
2020-03-13T05:53:39.3120104Z ##[section]Finishing: Bash Script
As you can see from the logs it always fails when calling the api to get the info of the supplied tag (or even latest tag).
Any idea why the system is trying to redirect me?
You should make changes on this line:
GH_API="https://git.[company].com/api/v3"
To
GH_API="https://api.github.com"
The problem that I experienced has to do with the way Git handles releases. In this case there was a Pre-Release (Release name/Tag: v0.0.1), which is what I got back when querying for all releases. As soon as I deleted this pre-release I got back nothing, but then realized that none of the other 'releases' were ever published.
After publishing a couple of releases I then finally got back a list of published releases. All other commands then worked as expected as well.
So, no sinister problems or bugs. Just some minor mistakes from a DevOps pipeline newbie.

Running Curl POST w/ JSON Payload Sequentially Against File

I am hitting a wall trying to build a script to save myself quite a good bit of time. I am working in a system in which I need to run a curl POST against a list of values. The list is about 400 lines long, so I am hoping to find a way of scripting this in Bash instead of running that call manually for each entry. Below are some details to help understand what I'm trying to accomplish:
If I were to be doing this task manually, each call would be formatted like the below:
curl -X POST --header "Content-Type: application/json" -v 'http://www.website.com:8081/cc/membership' -d #json_payload.json
This points to my JSON in the listed file which shows as the below:
{
"groupId": "12345678987654321",
"type": "serial",
"memberInfo": "apple"
}
If I run the above, the call works, and the expected operation occurs. The issue is that I need to run this against roughly 400 values for that "memberInfo" field in the JSON payload. I'm trying to identify a way to run a single bash script, which will run this curl command over and over, and update the JSON payload to use each row in a file as the below:
memberList.txt
apple
banana
peach
pear
orange
.
.
And then maybe insert a pointer in my JSON for the "memberInfo" field over to this file.
Any and all help/suggestions are greatly appreciated!
.
This will do as you intend. Its a little convoluted but you might polish it a bit.
#!/bin/bash
function getString(){
echo $1 | python3 -c '
import json
import sys
payload="""
{
"groupId": "12345678987654321",
"type": "serial",
"memberInfo": ""
}
"""
obj = json.loads(payload)
obj["memberInfo"] = sys.stdin.read().strip()
print(json.dumps(obj, indent = " "))
'
}
while read member
do
getString "$member" > json_payload.json
curl -X POST --header "Content-Type: application/json" -v 'http://www.website.com:8081/cc/membership' -d #json_payload.json
done <<< "$( cat fruits.txt )"
Hope it helps!
while read member; do
curl -X POST --header "Content-Type: application/json" -v 'http://www.website.com:8081/cc/membership' -d '{"groupId": "12345678987654321","type": "serial","memberInfo": "$member"}'
done <members.txt
This will work if you only care about the memberInfo field, another method could be writing your json line by line to payloads.txt file.
payloads.txt
{"groupId": "12345678987455432","type": "stereo","memberInfo": "apple"}
{"groupId": "34532453453453465","type": "serial","memberInfo": "banana"}
...
then use this as the script
while read payload; do
curl -X POST --header "Content-Type: application/json" -v 'http://www.website.com:8081/cc/membership' -d '$payload'
done <payloads.txt
here is a collection of bash scripting common uses I've had to use
https://github.com/felts94/advanced-bash/blob/master/bash_learn.sh

Editing GIST with cURL

#!/bin/bash
COMMIT=$(git log -1 --pretty=format:'{"subject": "%s", "name": "xxx", "date": "%cD"}')
curl -X PATCH -d'{"files": {"latest-commit": {"content": "$COMMIT"}}}' -u user:xxxx https://api.github.com/gists/xxx
This just shows $COMMIT in the Gist. I tried playing with ''' and stuff but cannot make this work.
Your $COMMIT variable is not expanded to its value, because it is enclosed in single-quotes.
About an actual implementation in Bash
The GitHub API require you send the file content as a string: https://developer.github.com/v3/gists/#input-1
When file content contains newlines, double quotes or other characters needing an escaping within a string, the most appropriate shell tool to fill-in and escape the content string is jq.
JavaScript provide a JSON.stringify() method, but here in the shell world, we use jq to process JSON data.
If you don't have jq available you can convert the content of the file, to a properly escaped JSON string with GNU sed this way:
# compose the GitHub API JSON data payload
# to update the latest-commit.json file in the $gist_id
# uses sed to properly fill-in and escape the content string
read -r -d '' json_data_payload <<EOF
{
"description": "Updated from GitHub API call in Bash",
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"content": "$(
sed ':a;N;$!ba;s/\n/\\n/g;s/\r/\\r/g;s/\t/\\t/g;s/"/\\"/g;' <<<"$latest_commit_json_content"
)"
}
}
}
EOF
This is how jq is used to fill the content string with proper escaping:
json_data_payload="$(
jq \
--arg content "$latest_commit_json_content" \
--compact-output \
'.files."latest-commit.json".content = $content' \
<<'EOF'
{
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"content": ""
}
}
}
EOF
)"
Detailed and tested ok implementation:
#!/usr/bin/env bash
# Set to the gist id to update
gist_id='4b85f310233a6b9d385643fa3a889d92'
# Uncomment and set to your GitHub API OAUTH token
github_oauth_token='###################'
# Or uncomment this and set to your GitHub username:password
#github_user="user:xxxx"
github_api='https://api.github.com'
gist_description='Gist update with API call from a Bash script'
filename='latest-commit.json'
get_file_content() {
# Populate variables from the git log of latest commit
# reading null delimited strings for safety on special characters
{
read -r -d '' subject
read -r -d '' author
read -r -d '' date
} < <(
# null delimited subject, author, date
git log -1 --format=$'%s%x00%aN%x00%cD%x00'
)
# Compose the latest commit JSON, and populate it with the latest commit
# variables, using jq to ensure proper encoding and formatting of the JSON
read -r -d '' jquery <<'EOF'
.subject = $subject |
.author = $author |
.date = $date
EOF
jq \
--null-input \
--arg subject "$subject" \
--arg author "$author" \
--arg date "$date" \
"$jquery"
}
# compose the GitHub API JSON data payload
# to update the latest-commit.json file in the $gist_id
# uses jq to properly fill-in and escape the content string
# and compact the output before transmission
get_gist_update_json() {
read -r -d '' jquery <<'EOF'
.description = $description |
.files[$filename] |= (
.filename = $filename |
.content = $content
)
EOF
jq \
--null-input \
--compact-output \
--arg description "$gist_description" \
--arg filename "$filename" \
--arg content "$(get_file_content)" \
"$jquery"
}
# prepare the curl call with options for the GitHub API request
github_api_request=(
curl # The command to send the request
--fail # Return shell error if request unsuccessful
--request PATCH # The request type
--header "Content-Type: application/json" # The MIME type of the request
--data "$(get_gist_update_json)" # The payload content of the request
)
if [ -n "${github_oauth_token:-}" ]; then
github_api_request+=(
# Authenticate the GitHub API with a OAUTH token
--header "Authorization: token $github_oauth_token"
)
elif [ -n "${github_user:-}" ]; then
github_api_request+=(
# Authenticate the GitHub API with an HTTP auth user:pass
--user "$github_user"
)
else
echo 'GitHub API require either an OAUTH token or a user:pass' >&2
exit 1
fi
github_api_request+=(
-- # End of curl options
"$github_api/gists/$gist_id" # The GitHub API url to address the request
)
# perform the GitHub API request call
if ! "${github_api_request[#]}"; then
echo "Failed execution of:" >&2
env printf '%q ' "${github_api_request[#]}" >&2
echo >&2
fi
Here is the generated curl call with my token redacted out:
curl --fail --request PATCH --header 'Content-Type: application/json' \
--data '{"description":"Hello World Examples","files":{"latest-commit.json":{"filename":"latest-commit.json","content":"{\n \"subject\": \"depricate Phosphor\",\n \"name\": \"Blood Asp\",\n \"date\": \"Wed, 12 Dec 2018 18:55:39 +0100\"\n}"}}}' \
--header 'Authorization: token xxxx-redacted-xxxx' \
-- \
https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92
And the JSON response it replied with:
"url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92",
"forks_url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92/forks",
"commits_url": "https://api.github.com/gists/4b85f310233a6b9d385643fa3a889d92/commits",
"id": "4b85f310233a6b9d385643fa3a889d92",
"node_id": "MDQ6R2lzdDRiODVmMzEwMjMzYTZiOWQzODU2NDNmYTNhODg5ZDky",
"git_pull_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92.git",
"git_push_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92.git",
"html_url": "https://gist.github.com/4b85f310233a6b9d385643fa3a889d92",
"files": {
"latest-commit.json": {
"filename": "latest-commit.json",
"type": "application/json",
"language": "JSON",
"raw_url": "https://gist.githubusercontent.com/leagris/4b85f310233a6b9d385643fa3a889d92/raw/7cb7f9d4a0170daf5083929858fb7eef706f8b59/latest-commit.json",
"size": 105,
"truncated": false,
"content": "{\n \"subject\": \"depricate Phosphor\",\n \"name\": \"Blood Asp\",\n \"date\": \"Wed, 12 Dec 2018 18:55:39 +0100\"\n}"
}
},
...

Github API: Get pull request for specific release tag

Is it possible to get a list of pull requests (or just the numbers) related to a release tag?
I have been looking at the Github API documentation all day and have tried different things but i can't see how i could make this work.
I can't see that the pull request information is available when i get a commit via the API, even if the pull request id & link is available here for example:
https://github.com/octokit/octokit.rb/commit/1d82792d7d16457206418850a3ed0a0230defc81 (see the #962 link next to "master" in the top left)
You can extract the commits between your tag & the previous one and search issues (of type pull request) with each of those commit SHA using search API :
use Get tags to extract the list of tags for your repo
extract the specific tag (release tag name) you are interested to & the previous tag (release tag if existing)
get all commits between these 2 tags using compare
search issues of type pull request with the specific SHA. In this case, you can perform a query(*) with all the commits sha concatenating SHA:<commit sha> to the query
(*)Note that the search query is limited to 256 characters so you'll have to split those search API call
Example using bash, curl & jq :
#!/bin/bash
current_tag="v4.8.0"
name_with_owner="octokit/octokit.rb"
access_token="YOUR_ACCESS_TOKEN"
tags=$(curl -s -H "Authorization: Token $access_token" \
"https://api.github.com/repos/$name_with_owner/tags")
key=$(jq -r --arg current_tag $current_tag 'to_entries | .[] | select(.value.name == $current_tag) | .key' <<< "$tags")
previous_tag=$(jq -r --arg index $((key+1)) '.[$index | tonumber].name' <<< "$tags")
echo "compare between $previous_tag & $current_tag"
commits=$(curl -s -H "Authorization: Token $access_token" \
"https://api.github.com/repos/$name_with_owner/compare/$previous_tag...$current_tag" | \
jq -r '.commits[].sha')
# you can have a query of maximum of 256 character so we only process 17 sha for each request
count=0
max_per_request=17
while read sha; do
if [ $count == 0 ]; then
query="repo:$name_with_owner%20type:pr"
fi
query="$query%20SHA:%20${sha:0:7}"
count=$((count+1))
if ! (($count % $max_per_request)); then
echo "https://api.github.com/search/issues?q=$query"
curl -s -H "Authorization: Token $access_token" \
"https://api.github.com/search/issues?q=$query" | jq -r '.items[].html_url'
count=0
fi
done <<< "$commits"
I did it in Ruby:
Using octokit gem
Only looking for commits with the prefix fix or feat
Purpose:
The code looks like this:
require 'octokit'
Client = Octokit::Client.new(access_token: YOUR_GITHUB_TOKEN)
def getCommitsForTag(repo, tagName)
previousTag = getPreviousTag repo, tagName
(Client.compare repo, previousTag.name, tagName).commits
end
def getPreviousTag(repo, tagName)
tags = Client.tags repo
tags.each_with_index { |tag, index|
if tag.name == tagName
return tags[index+1]
end
}
end
def filterCommitsByPrefix(commits, commitPrefixArray)
filteredArray = []
commits.each { |commit|
commitPrefixArray.each { |commitPrefix|
if commit.commit.message.start_with?(commitPrefix)
filteredArray.push(commit)
end
}
}
filteredArray
end
def getPullRequestsByCommits(commits)
query = "SHA:"
commits.each { |commit|
query += "#{commit.sha},"
}
Client.search_issues query
end
def getPullRequestsForTag(repo, tag)
commitsForTag = getCommitsForTag repo, tag
fixCommits = filterCommitsByPrefix commitsForTag, ['fix']
featCommits = filterCommitsByPrefix commitsForTag, ['feat']
{
fixes: getPullRequestsByCommits(fixCommits).items,
features: getPullRequestsByCommits(featCommits).items
}
end
#Execute it like this:
pullRequestsForTag = getPullRequestsForTag 'octokit/octokit.rb', 'v4.8.0'
puts "Fix pull requests:"
puts pullRequestsForTag[:fixes]
puts "Feat pull requests:"
puts pullRequestsForTag[:features]

Resources