I want to send data to a server via curl. The script has to first build the curl command by appending several parameters. Something like this:
# set other params...
param4_name="param4"
param4_value="Hello world"
params=""
# append other params to $params...
if [[ ! -z "$param4_value" ]]; then
params="$params --data-urlencode \"$param4_name=$param4_value\" "
fi
response="$(curl \
$params \
--get \
$my_url/test)"
When I use this code, however, the server receives the curl request as:
HTTP-Request: GET /test?"param4=Hello HTTP/1.1
(The " is included in the key and " world" is missing in the value.)
If I adjust the code to use:
if [[ ! -z "$param4_value" ]]; then
params="$params --data-urlencode $param4_name="$param4_value" "
fi
The server receives this:
HTTP-Request: GET /test?param4=Hello HTTP/1.1
(" world" is missing in the value.)
How do I adjust the bash code so that the server receives this request instead:
HTTP-Request: GET /test?param4=Hello%20World HTTP/1.1
I'm amenable to using this solution. But is there a "neater" way to do this without resorting to using a function, an embedded curl call, or perl/sed/awk/etc?
Related
I'd like to get a temporary GitLab token to run a simple script. I do not want a personal access token, nor project access token as that is too many clicks for the user. How do I log in or sign in? GitLab does not support password login anymore and one has to use OAuth. But that means I have to receive an HTTP redirect, running a web server!
GitLab Oauth dialogue can be very quick once authenticated in your browser. It doesn't even ask for confirmation in default configuration. The user doesn't then have to leave the command line. The little trouble is a pile of authentication browser tabs/windows left behind.
Only one HTTP request needs to be processed. We do not need a web server for that. Netcat can accept connection, give us the request and respond as well so that the user gets some feedback. Against the recommendation, this goes unencrypted. It is on a loopback device, though. Risk should be low. To up the security here, we'd need a proper browser app that can handle a custom protocol. WebView can be used to that effect, but then we're out of simple shell.
Curl (or wget) handles the necessary API calls.
This is a simple OAuth client written in a (posix?) shell:
#!/bin/sh
# fail fast
set -e
# defaults
app_id="$GITLAB_APP_ID"
app_port="8001"
app_addr="127.0.0.1"
api_v4=""
test -z "$GITLAB_HOST" || api_v4="https://${GITLAB_HOST}/api/v4"
test -z "$GITLAB_API_V4" || api_v4="$GITLAB_API_V4"
# display usage info help text
help () {
cat <<-EOHELP
Sign into GitLab on ${api_v4:-GITLAB_API_V4} with application '${app_id:-GITLAB_APP_ID}'
Requires: nc, curl, jq, x-www-browser|open, mktemp, posix(sh, cat, cut, head, rm, grep)
It is meant to work the same in a pipeline as well as on local environment.
It will use the CI_JOB_TOKEN set by GitLab runner or request OAuth from the user.
For OAuth, make the app with [api] access, not confidential, expire tokens: yes,
redirect URI should be set to http://127.0.0.1:8001
Args:
-h|--help this text
-a|--app_id set GitLab app id [GITLAB_APP_ID]
-4|--api_v4 set GitLab API v4 address [GITLAB_API_V4]
or set it with a GITLAB_HOST FQDN[:port]
-s|--server set GitLab API v4 address with a GITLAB_HOST FQDN[:port]
EOHELP
}
# parse command arguments
parse_args () {
while test "$#" -ne 0
do
case "$1" in
-h|--help) help; exit 0 ;;
-a|--app_id) app_id="$2"; shift 2 ;;
-a=*|--app_id=*) app_id="${1#*=}"; shift 1 ;;
-4|--api_v4) api_v4="$2"; shift 2 ;;
-4=*|--api_v4=*) api_v4="${1#*=}"; shift 1 ;;
-s|--server) api_v4="https://$2/api/v4"; shift 2 ;;
-s=*|--server=*) api_v4="https://${1#*=}/api/v4"; shift 1 ;;
*) ( echo "Unexpected arg '$1'"; echo; help ) >&2; exit 1 ;;
esac
done
}
auth_netcat () {
# compatible way to invoke different nc flavors: 1) gnu/bsd/mac 2) busybox
nc -l "$1" "$2" 2>/dev/null || nc -l "$1:$2"
}
auth_browse () {
# compatible way to open a default browser
x-www-browser "$#" || open "$#"
}
authenticate () {
# Use CI_JOB_TOKEN in a pipeline as gitlab_token if available or get oauth access token
if [ -n "$CI_JOB_TOKEN" ]; then
gitlab_user="gitlab-ci-token"
gitlab_token="$CI_JOB_TOKEN"
else
: "${api_v4:?specify a GitLab API v4 URL}"
: "${app_id:?specify a GitLab app ID}"
: "${app_port:?specify the app port}"
: "${app_addr:?specify the app address}"
echo "Getting token"
auth_file="$(mktemp)"
# Start netcat as a local web server to receive the auth code
auth_netcat "$app_addr" "$app_port" <<-EORSP >"$auth_file" &
HTTP/1.0 200 OK
Content-Length: 13
Authenticated
EORSP
auth_pid=$!
# If netcat started, proceed with requesting auth code
if kill -0 "$auth_pid"; then
auth_state="$auth_pid"
auth_url="${api_v4%/api/v4}/oauth/authorize?response_type=code&scope=api"
auth_url="$auth_url&client_id=$app_id"
auth_url="$auth_url&redirect_uri=http://$app_addr:$app_port"
auth_url="$auth_url&state=$auth_state"
auth_browse "$auth_url" &&
echo "Authentication window opened in your browser:" ||
echo "Authenticate:"
echo " $auth_url"
fi
# Wait for netcat to receive the code and then request access token and user name
if wait "$auth_pid" 2>/dev/null; then
auth_code="$(head -1 "$auth_file" | grep -Eo 'code=[^&]+' | cut -d= -f2)"
echo "claiming access token"
gitlab_token="$(
curl -sSLf "${api_v4%/api/v4}/oauth/token" \
-F "client_id=$app_id" \
-F "code=$auth_code" \
-F "grant_type=authorization_code" \
-F "redirect_uri=http://$app_addr:$app_port" |
jq -r '.access_token'
)"
echo "getting current user name"
gitlab_user="$(
curl -sSLf "${api_v4}/user/" \
-H "Authorization: Bearer $gitlab_token" |
jq -r '.username'
)"
rm "$auth_file"
else
echo "authentication FAILED"
fi
fi
}
if parse_args "$#" && authenticate >&2
then
echo "GITLAB_USER=${gitlab_user:?}"
echo "GITLAB_TOKEN=${gitlab_token:?}"
fi
Tbh, this Q&A is a somewhat more sophisticated gist to keep this handy and available to others. Improvements welcome!
If you copy&paste, convert spaces to tabs so the heredoc doesn't break
I am trying to connection to Neptune DB and getting vertices details using CURL command. I have shell script for it. But somehow variable data is not going through it gremlin query. I have one Orgid.txt file where tenantid is present and my shell script reading the file and passing it to "name" variable
#!/bin/bash
i=1
rm VerticesCount1
while IFS= read -r line
do
name="$line"
#echo "Orgid name is "$i": $name"
curl -X POST https://<Neptune_endpoint>:<port>/gremlin -d '{"gremlin":"g.V().has(\"system.tenantId\",\"$name\").count()"}' >> VerticesCount1
#printf "\n"
echo >> VerticesCount1
((i=i+1))
done < Orgid.txt
As with your other question I tested with a simple data file and it works fine. However, note how I changed the type of quotes used by curl.
i=1
while IFS= read -r line
do
name="$line"
curl -X POST https://mydbcluster.cluster-xxxxxxxxxxxx.us-east-1.neptune.amazonaws.com:8182/gremlin -d \
"{\"gremlin\":\"g.V().has('code','$name').count()\"}"
((i=i+1))
done < values.txt
which produces
{"requestId":"4e3e80ed-efcb-40a7-b92b-366c6f391d4e","status":{"message":"","code":200,"attributes":{"#type":"g:Map","#value":[]}},"result":{"data":{"#type":"g:List","#value":[{"#type":"g:Int64","#value":1}]},"meta":{"#type":"g:Map","#value":[]}}}{"requestId":"6a269b5b-32f6-49d2-a31d-c51dd52eba29","status":{"message":"","code":200,"attributes":{"#type":"g:Map","#value":[]}},"result":{"data":{"#type":"g:List","#value":[{"#type":"g:Int64","#value":1}]},"meta":{"#type":"g:Map","#value":[]}}}
it is working fine with this code.
while IFS= read -r line
do
name="$line"
#echo "Orgid name is "$i": $name"
curl -X POST https://<Neptune_endpoint>:<port>/gremlin -d '{"gremlin":"g.V().has(\"system.tenantId\",\"'$name'\").count()"}' >> VerticesCount1
echo >> VerticesCount1
done < Orgid.txt
I'm currently writing a shell/bash script to automate a workflow. This bash script can clone projects and create new repo's on Bitbucket, do composer install/update's and more of that stuff.
My first plan was to do this all over SSH, but there are some situations that I need HTTPS. For all thinks that go's over HTTPS I need to check the user credentials for Bitbucket first. The credentials consisting of a username and password.
Is this possible. If so, how?
As you suggested in comments, curl can do HTTP Basic Auth for you. For BitBucket, the response will be 401 Unauthorized or 200 OK, depending on the validity of username/password pair. You could test the output (or only headers) with grep, but a little bit more robust approach would be to use the -w/--write-out option, combined with a HEAD request and -silenced output:
http_status=$(curl -X HEAD -s -w '%{http_code}' \
-u "username:password" -H "Content-Type: application/json" \
https://api.bitbucket.org/2.0/repositories/$repo_owner)
Then, to test the status, you can use a simple conditional expression:
if [[ $http_status == 200 ]]; then
echo "Credentials valid"
else
if [[ $http_status == 401 ]]; then
echo "Credentials INVALID"
else
echo "Unexpected HTTP status code: $http_status"
fi
fi
Or, if you plan on testing multiple status codes, you can use the case command, for example:
case $http_status in
200) echo "Credentials valid";;
301|302) echo "API endpoint changed";;
401) echo "Credentials INVALID";;
5*) echo "BitBucket Internal server error";;
*) echo "Unexpected HTTP status code: $http_status";;
esac
Depending on certain conditions I want to use JWT else I want to provide path to certs. Thus in my shell script this is the code:
if /* some condition */
authorization='-H "'Authorization': 'Bearer' ${JWT}"'
else
authorization="--cert "${ADMIN_CERT_PATH}" --key "${ADMIN_KEY_PATH}""
Now the curl request should be:
curl -H "Authorization: Bearer 348129" for if condition
curl --cert /Users/.../admin_cert --key /Users/../admin_key .. for else path
In order to get that output I need to use the following format in my shell script for if condition
response_code="$(curl -s -o /dev/null -w "%{http_code}" "$authorization" "$status_url")"
and following format for else code:
response_code="$(curl -s -o /dev/null -w "%{http_code}" $authorization "$status_url")"
Note:
I need $authorization variable quoted in first case and unquoted in the else case.
I do not want to write 2 different curl commands instead reuse the authorization variable.
Thus, i need to modify the way I have declared my authorization variable such that I can write any one of the curl commands only once which works for both if and else cases.
curl supports a way to pass command line parameters in a file that I have used before when I have complex parameters. The idea is to place the complex command-line parameters into a simple text file and instruct curl to read parameters from it using --config parameter.
In this case the shell script would look something like the following.
#!/bin/sh
## "safely" create a temporary configuration file
curlctl=$(mktemp -q -t $(basename "$0"))
if test $? -ne 0
then
echo "$0: failed to create temporary file, exiting."
exit 75 # EX_TEMPFAIL
fi
trap 'rm "$curlctl"' 0
## write parameters used in all cases
cat>>"$curlctl" <<EOF
output = /dev/null
silent
write-out = %{http_code}
EOF
## append conditional parameters
if test "$some" = 'condition'
then
printf 'header = "Authorization: Bearer %s"\n' "$JWT" >> "$curlctl"
else
echo "cert = $ADMIN_CERT_PATH" >> "$curlctl"
echo "key = $ADMIN_KEY_PATH" >> "$curlctl"
fi
# uncomment to see what the config file looks like
# cat "$curlctl" | sed 's/^/curl config: /'
response_code=$(curl --config "$curlctl" http://httpbin.org/get)
echo "response code: $response_code"
The first few lines set up a temporary file that is deleted when the shell script exits. If you are already using trap then your cleanup will probably be more complex.
When you are using a shell that supports arrays, you can avoid the need for a temporary configuration file.
curl_opts=(-s -o /dev/null -w "%{http_code}")
if /* some condition */
curl_opts+=(-H "Authorization: Bearer $JWT")
else
curl_opts+=(--cert "$ADMIN_CERT_PATH" --key "$ADMIN_KEY_PATH")
fi
...
response_code="$(curl "${curl_opts[#]}" "$status_url")"
#!/bin/bash
currentTime=$(date +"%R")
currentdate=$(date +'%m/%d/%Y')
oldtime=$(date +%R -d "50 min ago")
echo "Current time : $currentTime"
echo "Current Date : $currentdate"
echo "Old time : $oldtime"
Response=$(curl -X GET -H "Authorization: Basic Token=" "http://targetserver.company.com:8080/v1/organizations/company/environments/environmentname/stats/apis?select=sum(is_error)&timeRange="$currentdate"%20"$currentTime"~"$currentdate"%20"$oldtime"")
echo -e "the response is: \n $Response"
Not getting any response? Please help how to use system date-time/current date-time in curl URL in shell-script.
it looks consistent, but not entirely sure what looks wrong. have you tried to echo the Response line to see what it looks like? how about trying it with wrapping the bash variable names in curly brace and removing the quotes in the string?
Response=$(curl -X GET -H "Authorization: Basic Token=" "http://targetserver.company.com:8080/v1/organizations/company/environments/environmentname/stats/apis?select=sum(is_error)&timeRange=${currentdate}%20${currentTime}~${currentdate}%20${oldtime}")