How to access GitHub API using personal access tokens in shell script? - bash

how is it going?
So I used a bash script to create a remote repository using a password to access a endpoint like this:
NEWVAR="{\"name\":\"$githubrepo\",\"private\":\"true\"}"
curl -u $USERNAME https://api.github.com/user/repos -d "$NEWVAR"
However, GitHub is going to not allow developers to access endpoints using passwords anymore. So my question is how do I create a remote repository using a personal access token?

use --header to transmit authorization:
#!/usr/bin/env sh
github_user='The GitHub user name'
github_repo='The repository name'
github_oauth_token='The GitHub API auth token'
# Create the JSON data payload arguments needed to create
# a GitHub repository.
json_data="$(
jq \
--null-input \
--compact-output \
--arg name "$github_repo" \
'{$name, "private":true}'
)"
if json_reply="$(
curl \
--fail \
--request POST \
--header 'Accept: application/vnd.github.v3+json' \
--header "Authorization: token $github_oauth_token" \
--header 'Content-Type: application/json' \
--data "$json_data" \
'https://api.github.com/user/repos'
)"; then
# Save the JSON answer of the repository creation
printf '%s' "$json_reply" >"$github_repo.json"
printf 'Successfully created the repository: %s\n' "$github_repo"
else
printf 'Could not create the repository: %s\n' "$github_repo" >&2
printf 'The GitHub API replied with this JSON:\n%s\n' "$json_reply" >&2
fi
See my answer here for a featured implementation example:
https://stackoverflow.com/a/57634322/7939871

Related

extract a value from the output of a script and store in a variable winodws?

According to the document Get Azure AD tokens for service principals:
curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' \
https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token \
-d 'client_id=<client-id>' \
-d 'grant_type=client_credentials' \
-d 'scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default' \
-d 'client_secret=<client-secret>'
Now, I could get the correct output,like:
The Azure AD access token is in the access_token value within the output of the call.
What I want is that I need the get the value of the access_token and set it to the variable, so that I could use it in next REST API scripts.
But I'm not very familiar with Bash and curl, can anyone offer advice?
use jq to extract access_token from the json, and VAR=$(...) to store it in a variable,
ACCESS_TOKEN=$(curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' \
https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token \
-d 'client_id=<client-id>' \
-d 'grant_type=client_credentials' \
-d 'scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default' \
-d 'client_secret=<client-secret>' \
| jq -r .access_token )
then you can use ACCESS_TOKEN like
curl -d access_token="$ACCESS_TOKEN"
but be wary, bash is a shitty scripting language, you should not attempt to use bash for complex logic, you should probably switch to a better scripting language like Python, Perl, or PHP, rather than implementing complex logic in Bash. (same goes for Windows's cmd and PowerShell. all 3 are languages unsuitable, but not incapable, of complex logic)

Upload data to aws s3 using curl in c++

I'm trying to upload the data to aws s3 using following script and getting below error.
Example Script
#!/usr/bin/sh
file_to_upload="/home/sraj/Hello10.txt"
bucket="mybucket"
filepath="/${bucket}/${file_to_upload}"
contentType='application\/x-compressed-tar'
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
signature_string="PUT\n\n${contentType}\n${dateValue}\n${filepath}"
s3_access_key=xxxxxxxxx
s3_secret_key=yyyyyyyyy
signature_hash=`echo -en ${signature_string} | openssl sha256 -hmac ${s3_secret_key} -binary | base64`
echo "${s3_access_key} : ${s3_secret_key} : ${signature_hash}"
curl -X PUT -T "${file_to_upload}" \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3_access_key}:${signature_hash}" \
https://${bucket}.s3.amazonaws.com/${file_to_upload}
I think this is related to I AM user KMS encryption option.
Kindly help me how to solve above issue. I'm not much idea about curl and s3 also.
It will very helpful for sample code.
Some of them mentioned this is related to Signature version4 issue but I don't have much idea about fix this one.
The error says Invalid Argument.
I found this link, I have no idea if it will help.
https://www.gyanblog.com/aws/how-upload-aws-s3-curl/#curl-the-savior`
The only difference I see is the dateValue.
I do not know if that matters, but just in case, I posted this.
# about the file
file_to_upload=<file you want to upload>
bucket=<your s3 bucket name>
filepath="/${bucket}/${file_to_upload}"
# metadata
contentType="application/x-compressed-tar"
dateValue=`date -R`
signature_string="PUT\n\n${contentType}\n${dateValue}\n${filepath}"
#s3 keys
s3_access_key=<your s3 access key>
s3_secret_key=<your s3 secret key>
#prepare signature hash to be sent in Authorization header
signature_hash=`echo -en ${signature_string} | openssl sha1 -hmac ${s3_secret_key} -binary | base64`
# actual curl command to do PUT operation on s3
curl -X PUT -T "${file_to_upload}" \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3_access_key}:${signature_hash}" \
https://${bucket}.s3.amazonaws.com/${file_to_upload}

How to use Bash command line to curl an API with token and payload as parameters

I am novice and first timer to bash. Trying to run bash under command line to invoke an API, by passing token and payload received from two different APIs and are set as parameters. Below is my command. I am trying to add this bash script to a task in AzureBatch service job.
It has 3 curl requests,
First one (Line#1 in the code snippet below)- gets payload by
calling an API. ---- This is working fine, I am able to verify the
payload using the echo statement following the first curl command.
Second one(Line#3 in the code snippet below) - gets token by
calling the token provider ----- This is working fine as well,
verified using the echo statement.
Third one (Line#5 in the code snippet below),This is the problematic command. I am trying to pass the token and payload received from the above two commands and the curl is not able to resolve them.
both token and payload or not resolving to their values..
My Bash COmmand
/bin/bash -c
"payload=$(curl --location --request GET 'http://url/OutreachData')
&& echo -e \"The value of payload is: "'$payload'"\"
&& token=$(curl --location --request POST 'https://login.microsoftonline.com/<<tenantId>>/oauth2/v2.0/token' --header 'Content-Type: application/x-www-form-urlencoded' --data-urlencode 'client_id=<<clientId>>' --data-urlencode 'scope=api://<<applicationId>>/.default' --data-urlencode 'client_secret=<<clientSecret>>' --data-urlencode 'grant_type=client_credentials' --data-urlencode 'Audience=api://<<applicationId>>'|jq -j '.access_token')
&& echo -e \"value of token is "'$token'"\n\"
&& result=$(curl --location --request POST 'https://url/api/<<Resource>>' --header 'accept: */*' --header 'Content-Type: application/json' --header 'Authorization: Bearer '"'$token'" --data-raw "'$payload'")
&& echo -e \"Result is "'$result'"\""
This is how the third Curl is resolving to, payload and token are not getting replaced as we can see in the authorization header and data-raw elements
++ curl --location --request POST https://url/api/ --header 'accept: /' --header 'Content-Type: application/json' --header 'Authorization: Bearer ' --data-raw ''''''''
There should be no need to explicitly run this with bash -c unless you are in a very constrained environment where you simply cannot run Bash by any other means.
The immediate problem is that code like
bash -c "echo "'$token'" && true"
ends up with $token being single-quoted in the shell which you run bash -c from. But the blazingly obvious fix is to not have this complex quoting in the first place.
payload=$(curl --location --request GET 'http://url/OutreachData')
echo "The value of payload is: '$payload'"
token=$(curl --location --request POST \
'https://login.microsoftonline.com/<<tenantId>>/oauth2/v2.0/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'client_id=<<clientId>>' \
--data-urlencode 'scope=api://<<applicationId>>/.default' \
--data-urlencode 'client_secret=<<clientSecret>>' \
--data-urlencode 'grant_type=client_credentials' \
--data-urlencode 'Audience=api://<<applicationId>>' |
jq -j '.access_token')
echo "value of token is "'$token'"
result=$(curl --location --request POST \
'https://url/api/<<Resource>>' --header 'accept: */*' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer '"$token" \
--data-raw "$payload")
echo "Result is "'$result'"
If your current shell is not Bash and you need these commands to be run in Bash, a simpler workaround is to put the script in a here document, which drastically simplifies the quoting needs (or if this is in an interactive session, just run bash and run these commands at the interactive Bash prompt, then exit when you no longer want to be in Bash).

Bash script to loop through remote directory and pipe files 1 at a time to CURL

I am trying to transfer all files residing in a specified directory on Server1 to Server3 via a script running on Server2.
The transfer to Server3 has to happen through an API and thus must use the following CURL call:
curl -X POST https://content.dropboxapi.com/2/files/upload \
--header "Authorization: Bearer $token" \
--header "Dropbox-API-Arg: {\"path\": \"/xfer/$name\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}" \
--header "Content-Type: application/octet-stream" \
--data-binary #$f
If it is just 1 file, I can do it successfully, but i'm trying to iterate through the directory on Server1 and send the file directly to the CURL call. So far I've got:
files="( $(ssh me#server1 ls dir/*) )"
while read f
do
name=$(basename ${f})
curl -X POST https://content.dropboxapi.com/2/files/upload \
--header "Authorization: Bearer $token" \
--header "Dropbox-API-Arg: {\"path\": \"/xfer/$name\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}" \
--header "Content-Type: application/octet-stream" \
--data-binary #$f
done <<< "$files"
The loop seems to be reading the "(" from the array of files into the 1st file name, which obviously causes a problem. I can't get beyond that to be able to tell if POSTING the current file in the loop via --data-binary will actually do what I think (or am hoping) it will.
Any ieas?
The error in the original message was enclosing the ssh command with "()". I am working on a similar issue. In the past I've used Rsync but I want a solution that doesn't require installing extra software. Here is an example that I'm working with to move files off of a Nodejs dev server to backup, running in Bash on Debian:
files=$(ssh chris#estack ls ~/tmp/gateway)
#echo $files
for FILE in $files
do
if [[ "$FILE" = "node_modules" || "$FILE" = ".git" ]]
then
echo "skip $FILE";
continue
fi
echo Copy ~/tmp/gateway/$FILE
#scp -Cpr chris#estack:~/tmp/gateway/$FILE ~/tmp/tmp
done

curl PUT using auth token header to mesosphere fails without eval

EDIT:
I have managed to make it work with
response=$(
curl -k -X PUT -d "$marathon_payload" --write-out %{http_code} --silent --output "$tmp"\
-H "Authorization: token=$dcos_token" -H "$header_content_type" $app_id_url
)
The single quotes were causing the problem. It took a few gyrations but all good.
MORAL: quotes inside the value don't matter if the value is properly quoted UNLESS you eval the whole thing, and I should have known that. Occam's wins again.
end edit
I am initiating Mesosphere microservice deployments with curl, but it won't succeed without using eval. Since I recently inherited this code I've been trying to scrub the eval out of it just as a matter of habit, but it's thwarting me.
The script initiates the deployment with
response=$(
eval curl -k -X PUT -d "'$marathon_payload'" --write-out %{http_code} --silent --output $tmp\
-H "'Authorization: token=$dcos_token'" -H "'$header_content_type'" $app_id_url
)
If it gets a 200 or a 201, it loops a curl to effectively screen-scrape the deployments page till the request disappears.
chkDeploy() { rm -f $tmp;
eval curl -k -X GET --silent --write-out %{http_code} --silent --output $tmp\
-H "'Authorization: token=$dcos_token'" -H "'$header_content_type'" $deployments_url
}
response=$( chkDeploy )
$dcos_token is a base64 encoded string.
It then checks the service with another curl loop to the info page so it can verify the version number. This one is working fine with no eval.
chkCode() {
curl -k -X GET --write-out %{http_code} --silent --output $tmp $info_url;
}
response=$( chkCode )
The first two return 401, authentication failure.
I'm guessing the auth token quoting is off.
There's no reason to use eval here; you just need to quote the arguments to -H properly.
response=$(
curl -k -X PUT -d "$marathon_payload" \
--write-out %{http_code} \
--silent --output "$tmp" \
-H "Authorization: token=$dcos_token" \
-H "$header_content_type" "$app_id_url"
)

Resources