I've got the following call in a bash script I'm creating
response= curl -X POST $URL -u "$USER:$PASSWORD" --data-urlencode "key=$key" --data "label=pub_key" -o /dev/null --silent --write-out "%{http_code}"
I can see 200 being written to the console however $response always ends up null.
I've also tried the following but it was not better.
response= $(curl -X POST $URL -u "$USER:$PASSWORD" --data-urlencode "key=$key" --data "label=pub_key" -o /dev/null --silent --write-out "%{http_code}")
Any help for a bash noob would be appreciated.
Space -- the final frontier:
response=$(curl -X ...)
Note: no spaces around the =. The shell is white-space sensitive in a few places and variable assignments are one of them.
With the space, as in var= command args, you set var as empty in a one-shot assignment and then run command.
Related
I want to include --noproxy '*' with curl in a CURL definition in a bash script, i.e.
CURL='curl --noproxy \'*' --fail --max-time 10 --silent --output /dev/null --write-out '%{http_code}\\n''
But this does not correctly process '*'.
echo $CURL
curl --noproxy \* --fail --max-time 10 --silent --output /dev/null --write-out %{http_code}\n
Can anyone advise as to the correct syntax for inclusion of the single quotes with *, which noproxy requires?
If you really want to store the command somewhere, before executing it, don' use a scalar, but an array:
CURL=(curl --noproxy '*' --fail --max-time 10 --silent --output /dev/null --write-out '%{http_code}\\n')
and execute it with
"${CURL[#]}"
This not only makes quoting easier, but also works if one of the arguments contains a space.
You would need to escape both 's, but curl doesn't need the quotes. The quotes are simply to prevent the shell from expanding * before curl sees it. '*' and \* are equivalent.
Don't define a parameter to execute as a command; define a function.
CURL () {
curl --noproxy '*' --fail --max-time 10 --silent --output /dev/null --write-out '%{http_code}\n' "$#"
}
I am optionally including a user and password in a curl request as follows:
declare creds=""
if [ -n "$user" ] && [ -n "$password" ]; then
creds="-u ${user}:${password}"
fi
output=$(curl ${creds} -X PUT -v --write-out "%{http_code}" "$url" \
-H 'Content-Type: application/json' -s -o /dev/null --data "${payload}")
This seems to work fine, but I'm getting this shellcheck warning:
Double quote to prevent globbing and word splitting
https://github.com/koalaman/shellcheck/wiki/SC2086
Puting quotes around it doesn't work, e.g. if I do this:
output=$(curl "${creds}" -X PUT -v --write-out "%{http_code}" "$url" \
-H 'Content-Type: application/json' -s -o /dev/null --data "${payload}")
then when the username and password are not supplied, this results in empty double quotes in the curl request curl "" -X PUT ..., which generates a <url> malformed error.
I could use an if-else for the curl command, but I'd rather avoid the duplication. Is the above approach acceptable despite the shellcheck warning?
You were doing right in putting quotes around the variable, but shellcheck doesn't catch the issue of storing commands in a variable which has its own downfalls. Since this being a an issue with function of the shell, shellcheck can't quite catch it out-of-the-box. When you did below
creds="-u ${user}:${password}"
and quoted "$creds", it is passed as one single argument word to curl instead of being broken down to -u and "${user}:${password}" separately. The right approach should have been to use an array to store the commands and expand it, so that the words are preserved and not split by the shell (foremost reason to quote the variable, as indicated by shellcheck)
creds=(-u "${user}:${password}")
and invoke
curl "${creds[#]}" <rest-of-the-cmd>
Also explore the following
I'm trying to put a command in a variable, but the complex cases always fail!
How to store a command in a variable in a shell script?
EDIT:
I have managed to make it work with
response=$(
curl -k -X PUT -d "$marathon_payload" --write-out %{http_code} --silent --output "$tmp"\
-H "Authorization: token=$dcos_token" -H "$header_content_type" $app_id_url
)
The single quotes were causing the problem. It took a few gyrations but all good.
MORAL: quotes inside the value don't matter if the value is properly quoted UNLESS you eval the whole thing, and I should have known that. Occam's wins again.
end edit
I am initiating Mesosphere microservice deployments with curl, but it won't succeed without using eval. Since I recently inherited this code I've been trying to scrub the eval out of it just as a matter of habit, but it's thwarting me.
The script initiates the deployment with
response=$(
eval curl -k -X PUT -d "'$marathon_payload'" --write-out %{http_code} --silent --output $tmp\
-H "'Authorization: token=$dcos_token'" -H "'$header_content_type'" $app_id_url
)
If it gets a 200 or a 201, it loops a curl to effectively screen-scrape the deployments page till the request disappears.
chkDeploy() { rm -f $tmp;
eval curl -k -X GET --silent --write-out %{http_code} --silent --output $tmp\
-H "'Authorization: token=$dcos_token'" -H "'$header_content_type'" $deployments_url
}
response=$( chkDeploy )
$dcos_token is a base64 encoded string.
It then checks the service with another curl loop to the info page so it can verify the version number. This one is working fine with no eval.
chkCode() {
curl -k -X GET --write-out %{http_code} --silent --output $tmp $info_url;
}
response=$( chkCode )
The first two return 401, authentication failure.
I'm guessing the auth token quoting is off.
There's no reason to use eval here; you just need to quote the arguments to -H properly.
response=$(
curl -k -X PUT -d "$marathon_payload" \
--write-out %{http_code} \
--silent --output "$tmp" \
-H "Authorization: token=$dcos_token" \
-H "$header_content_type" "$app_id_url"
)
I wrote curl which returns only http status code :
curl --write-out %{http_code} \n
--silent \
--output /dev/null \
$URL
It works fine if I execute this from console. But after I have puted it into script, like this:
HTTP_STATUS=$(curl --write-out %{http_code} \n
--silent \
--output /dev/null \
$URL)
And try to echo $HTTP_STATUS, result is 200000000000000000000000000000000000000000000000000
How can I fix it?
I wrote curl which returns only http status code
There are couple of issues with your script.
Your use of UPPERCASE variables might override shell environment variables.
The --write-out argument of the curl can ideally be within double quotes.
status=$(curl --write-out "%{http_code}" --silent --output /dev/null "$url")
echo "$status" # Would give you just the status
Note: As pointed out in this comment you don't need the newline too since you're assigning the value to a variable.
I have a curl request like this :
curl -s -u $user:$password -X GET -H "Content-Type: application/json" $url
Which returns a json as response. So I will parse the response using jq to get some specific data. Like this :
curl -s -u $user:$password -X GET -H "Content-Type: application/json" $url | jq '<expression>'
Now if the curl request fails then obviously the parsing operation throws ugly error. I want to avoid this. How to store the response first and then later parse it if the request is successful. I don't want to display the json whole response. Also if I add -w "%{http_code}" in my request it appends the status code with the JSON response which messes up the parsing. How to solve this ? I basically want to first check if the curl request is successful or not then get the JSON response and parse it.I also want to get the status code, so that if it fails I can display the status code. But status code is now messing up with json response.
You can combine the --write and --fail options:
# separating the (verbose) curl options into an array for readability
curl_args=(
--write "%{http_code}\n"
--fail
--silent
--user "$user:$password"
--request GET
--header "Content-Type: application/json"
)
if ! output=$(curl "${curl_args[#]}" "$url"); then
echo "Failure: code=$output"
else
# remove the "http_code" line from the end of the output, and parse it
sed '$d' <<<"$output" | jq '...'
fi
Also note: quote your variables!
I found glenn jackman's answer good, but a bit confusingly written, so I rewrote it, and altered it so I can use it as a safer alternative to curl | jq.
#!/bin/bash
# call this with normal curl arguments, especially url argument, e.g.
# safecurl.sh "http://example.com:8080/something/"
# separating the (verbose) curl options into an array for readability
curl_args=(
-H 'Accept:application/json'
-H 'Content-Type:application/json'
--write '\n%{http_code}\n'
--fail
--silent
)
echo "${curl_args[#]}"
# prepend some arguments, but pass on whatever arguments this script was called with
output=$(curl "${curl_args[#]}" "$#")
return_code=$?
if [ 0 -eq $return_code ]; then
# remove the "http_code" line from the end of the output, and parse it
echo "$output" | sed '$d' | jq .
else
# echo to stderr so further piping to jq will process empty output
>&2 echo "Failure: code=$output"
fi
Note: This code does not test for services that ignore the requested content type and respond with HTML. You'd need to test for grep -l '</html>' for that.