I am optionally including a user and password in a curl request as follows:
declare creds=""
if [ -n "$user" ] && [ -n "$password" ]; then
creds="-u ${user}:${password}"
fi
output=$(curl ${creds} -X PUT -v --write-out "%{http_code}" "$url" \
-H 'Content-Type: application/json' -s -o /dev/null --data "${payload}")
This seems to work fine, but I'm getting this shellcheck warning:
Double quote to prevent globbing and word splitting
https://github.com/koalaman/shellcheck/wiki/SC2086
Puting quotes around it doesn't work, e.g. if I do this:
output=$(curl "${creds}" -X PUT -v --write-out "%{http_code}" "$url" \
-H 'Content-Type: application/json' -s -o /dev/null --data "${payload}")
then when the username and password are not supplied, this results in empty double quotes in the curl request curl "" -X PUT ..., which generates a <url> malformed error.
I could use an if-else for the curl command, but I'd rather avoid the duplication. Is the above approach acceptable despite the shellcheck warning?
You were doing right in putting quotes around the variable, but shellcheck doesn't catch the issue of storing commands in a variable which has its own downfalls. Since this being a an issue with function of the shell, shellcheck can't quite catch it out-of-the-box. When you did below
creds="-u ${user}:${password}"
and quoted "$creds", it is passed as one single argument word to curl instead of being broken down to -u and "${user}:${password}" separately. The right approach should have been to use an array to store the commands and expand it, so that the words are preserved and not split by the shell (foremost reason to quote the variable, as indicated by shellcheck)
creds=(-u "${user}:${password}")
and invoke
curl "${creds[#]}" <rest-of-the-cmd>
Also explore the following
I'm trying to put a command in a variable, but the complex cases always fail!
How to store a command in a variable in a shell script?
Related
I want to include --noproxy '*' with curl in a CURL definition in a bash script, i.e.
CURL='curl --noproxy \'*' --fail --max-time 10 --silent --output /dev/null --write-out '%{http_code}\\n''
But this does not correctly process '*'.
echo $CURL
curl --noproxy \* --fail --max-time 10 --silent --output /dev/null --write-out %{http_code}\n
Can anyone advise as to the correct syntax for inclusion of the single quotes with *, which noproxy requires?
If you really want to store the command somewhere, before executing it, don' use a scalar, but an array:
CURL=(curl --noproxy '*' --fail --max-time 10 --silent --output /dev/null --write-out '%{http_code}\\n')
and execute it with
"${CURL[#]}"
This not only makes quoting easier, but also works if one of the arguments contains a space.
You would need to escape both 's, but curl doesn't need the quotes. The quotes are simply to prevent the shell from expanding * before curl sees it. '*' and \* are equivalent.
Don't define a parameter to execute as a command; define a function.
CURL () {
curl --noproxy '*' --fail --max-time 10 --silent --output /dev/null --write-out '%{http_code}\n' "$#"
}
I have the following bash script that uses its arguments to hit a RESTful web service (via curl) and prints out both the curl request made as well as the response:
#! /bin/bash
# arguments:
# $1 - username
# $2 - password
#
# outputs:
# if the script exits with a non-zero status then something went wrong
# verify that we have all 6 required arguments and fail otherwise
if [ "$#" -ne 2 ]; then
echo "Required arguments not provided"
exit 1
fi
# set the script arguments to meaningful variable names
username=$1
password=$2
# login and fetch a valid auth token
req='curl -k -i -H "Content-Type: application/json" -X POST -d ''{"username":"$username","password":"$password"}'' https://somerepo.example.com/flimflam'
resp=$(curl -k -i -H "Content-Type: application/json" -X POST -d ''{"username":"$username","password":"$password"}'' https://somerepo.example.com/flimflam)
# echo the request for troubleshooting
echo "req = $req"
if [ -z "$resp" ]; then
echo "Login failed; unable to parse response"
exit 1
fi
echo "resp = $resp"
When I run this I get:
$ sh myscript.sh myUser 12345#45678
curl: (3) Port number ended with '"'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: 12345#45678"
100 1107 100 1093 100 14 2849 36 --:--:-- --:--:-- --:--:-- 2849
req = curl -k -i -H "Content-Type: application/json" -X POST -d {"username":"$username","password":"$password"} https://somerepo.example.com/flimflam
resp = HTTP/1.1 400 Bad Request...(rest omitted for brevity)
Obviously, I'm not escaping the various layers of single- and double-quotes inside the curl statement correctly, as is indicated by outputs like:
curl: (6) Could not resolve host: 12345#45678"
and:
req = curl -k -i -H "Content-Type: application/json" -X POST -d {"username":"$username","password":"$password"} https://somerepo.example.com/flimflam
where the username/password variables are not parsing.
In reality my script takes a lot more than 2 arguments, which is why I'm changing them to have meaningful variable names (such as $username instead of $1) so its more understandable and readable.
Can anyone spot where I'm going awry? Thanks in advance!
Update
I tried the suggestion which turns the req into:
curl -k -i -H 'Content-Type: application/json' -X POST -d "{'username':'myUser','password':'12345#45678'}" https://somerepo.example.com/flimflam
However this is still an illegal curl command and instead needs to be:
curl -k -i -H 'Content-Type: application/json' -X POST -d '{"username":"myUser","password":"12345#45678"}' https://somerepo.example.com/flimflam
First, as I said in a comment, storing commands in variables just doesn't work right. Variables are for data, not executable code. Second, you have two levels of quoting here: quotes that're part of the shell syntax (which are parsed, applied, and removed by the shell before the arguments are passed to `curl), and quotes that're part of the JSON syntax.
But the second problem is actually worse than that, because simply embedding an arbitrary string into some JSON may result in JSON syntax errors if the string contains characters that're part of JSON syntax. Which passwords are likely to do. To get the password (and username for that matter) embedded correctly in your JSON, use a tool that understands JSON syntax, like jq:
userinfo=$(jq -n -c --arg u "$username" --arg p "$password" '{"username":$u,"password":$p}')
Explanation: this uses --arg to set the jq variables u and p to the shell variables $username and $password respectively (and the double-quotes around the shell variables will keep the shell from doing anything silly to the values), and creates a JSON snippet with them embedded. jq will automatically add appropriate quoting/escaping/whatever is needed.
Then, to use it with curl, use something like this:
resp=$(curl -k -i -H "Content-Type: application/json" -X POST -d "$userinfo" https://somerepo.example.com/flimflam)
Again, the double-quotes around $userinfo keep the shell from doing anything silly. You should almost always put double-quotes around variables references in the shell.
Note that I never used the req variable to store the command. If you need to print the command (or its equivalent), use something like this:
printf '%q ' curl -k -i -H "Content-Type: application/json" -X POST -d "$userinfo" https://somerepo.example.com/flimflam
echo
The %q format specifier tells the shell to add appropriate quoting/escaping so that you could run the result as a shell command, and it'd work properly. (And the echo is there because printf doesn't automatically add a newline at the end of its output.)
try changing this:
req='curl -k -i -H "Content-Type: application/json" -X POST -d ''{"username":"$username","password":"$password"}'' https://somerepo.example.com/flimflam'
to this
req="curl -k -i -H 'Content-Type: application/json' -X POST -d \"{'username':'$username','password':'$password'}\" https://somerepo.example.com/flimflam"
and similarly for the resp
ah those pesky "curly" thingies...
how 'bout...
req="curl -k -i -H 'Content-Type: application/json' -X POST -d '{\"username\":\"$username\",\"password\":\"$password\"}' https://somerepo.example.com/flimflam"
This needs even more escaping:
With:
resp=$(curl -k -i -H "Content-Type: application/json" -X POST -d "{\"username\":\"$username\",\"password\":\"$password\"}" https://somerepo.example.com/flimflam)
In bash, the variables are still expanded when they're inside single quotes that are inside double quotes.
And you'll need the \" double quotes in the payload as per the JSON definition.
EDIT: I rerun the curl through a HTTP proxy and corrected the script line (see above, removed the single quotes). Results (in raw HTTP) are now:
POST /flimflam HTTP/1.1
Host: somerepo.example.com
User-Agent: curl/7.68.0
Accept: */*
Content-Type: application/json
Content-Length: 44
Connection: close
{"username":"user","password":"12345#abcde"}
(which should be fine)
I've got the following call in a bash script I'm creating
response= curl -X POST $URL -u "$USER:$PASSWORD" --data-urlencode "key=$key" --data "label=pub_key" -o /dev/null --silent --write-out "%{http_code}"
I can see 200 being written to the console however $response always ends up null.
I've also tried the following but it was not better.
response= $(curl -X POST $URL -u "$USER:$PASSWORD" --data-urlencode "key=$key" --data "label=pub_key" -o /dev/null --silent --write-out "%{http_code}")
Any help for a bash noob would be appreciated.
Space -- the final frontier:
response=$(curl -X ...)
Note: no spaces around the =. The shell is white-space sensitive in a few places and variable assignments are one of them.
With the space, as in var= command args, you set var as empty in a one-shot assignment and then run command.
I have written a smoke-testing script that uses BASH script & Curl to test RESTful web services we're working on. The script reads a file, and interprets each line as a URL suffix and parameters for a Curl REST call.
Unfortunately, the script gives unexpected results when I adapted it to run HTTP POST calls as well as GET calls. It does not give the same results running the command on its own, vs. in script:
The BASH Script:
IFS=$'\n' #Don't split an input URL line at spaces
RESTHOST='hostNameAndPath' #Can't give this out
URL="/activation/v2/activationInfo --header 'Content-Type:Application/xml'"
URL2="/activation/v2/activationInfo"
OUTPUT=`curl -sL -m 30 -w "%{http_code}" -o /dev/null $RESTHOST$URL -d #"./activation_post.txt" -X POST`
echo 'out:' $OUTPUT
OUTPUT2=`curl -sL -m 30 -w "%{http_code}" -o /dev/null $RESTHOST$URL2 --header 'Content-Type:Application/xml' -d #'./activation_post.txt' -X POST`
echo 'out2:' $OUTPUT2
Results Out:
out: 505
out2: 200
So, the first call fails (HTTP return code 505, HTTP Version Not Supported), and the second call succeeds (return code "OK").
Why does the first call fail, and how do I fix it? I've verified they should execute the same command (evaluating in echo). I am sure there is something basic I'm missing, as I am just NOW learning Bash scripting.
I think I have found the problem! It is caused by IFS=$'\n'! Because of this, variable expansion does not work as expected. It does not let to split the arguments specified in the URL string!
As a result the SERVER_PROTOCOL variable on the server side will be set to '--header Content-Type:Application/xml HTTP/1.1' instead of "HTTP/1.1", and the CONTENT_TYPE will be 'application/x-www-form-urlencoded' instead of 'Application/xml'.
To show the root of the problem in detail:
VAR="Solaris East"
printf "+%s+ " $VAR
echo "==="
IFS=$'\n'
printf "+%s+ " $VAR
Output:
+Solaris+ +East+ ===
+Solaris East+
So the $VAR expansion does not work as expected because of IFS=$'\n'!
Solution: Do not use IFS=$'\n' and replace space to %20 in URL!
URL=${URL2// /%20}" --header Content-Type:Application/xml"
In this case your first curl call will work properly!
If You still use IFS=$'\n' and give --header option in the command line it will not work properly if URL contains a space, because the server will fail to process it (I tested on apache)!
Even You still cannot use HEADER="--header Content-Type:Application/xml" as expanding $HEADER will result one(!) argument for curl, namely --header Content-Type:Application/xml instead of splitting them into two.
So I may suggest to replace spaces in URL to %20 anyway!
The single quotes surrounding Content-Type:Application/xml, because they are quoted in the value of URL are treated as literal quotes and not removed when $URL is expanded in that call to curl. As a result, you are passing an invalid HTTP header. Just use
URL="/activation/v2/activationInfo --header Content-Type:Application/xml"
OUTPUT=`curl -sL -m 30 -w "%{http_code}" -o /dev/null $RESTHOST$URL -d #"./activation_post.txt" -X POST`
However, it's not a great idea to rely on word-splitting like this to combine two separate pieces of the call to curl in a single variable. Try something like this instead:
URLPATH="activation/v2/activationInfo"
HEADERS=("--header" "Content-Type:Application/xml")
OUTPUT=$( curl -SL -m 30 -w "%{http_code}" -o /dev/null "$RESTHOST/$URL" "${HEADERS[#]}" -d #'./activation_post.txt' -X POST )
Hi i am trying to use curl in shell script as shown below but i am not able to substitute the variable $line in the CURL . please suggest
while read line
do
echo "allowing mac $line"
curl -X POST -d '{"src-mac": "$line"}' http://localhost:8080/wm/firewall/rules/json
curl -X POST -d '{"dst-mac": "$line"}' http://localhost:8080/wm/firewall/rules/json
done < /home/floodlight/allowedmacs
No variable substitution in single quotes. Switch to double around the expansion, like this:
curl -X POST -d '{"src-mac": "'"$line"'"}' http://localhost:8080/wm/firewall/rules/json
curl -X POST -d '{"dst-mac": "'"$line"'"}' http://localhost:8080/wm/firewall/rules/json
Or you could use double-quotes around the whole thing and escape the inner ones:
curl -X POST -d "{\"src-mac\": \"$line\"}" http://localhost:8080/wm/firewall/rules/json
curl -X POST -d "{\"dst-mac\": \"$line\"}" http://localhost:8080/wm/firewall/rules/json
Either way, can't be inside single quotes when you get to $line if you want it expanded.