I set a command line argument as below:
$TOKENARG='Authorization: Bearer 9042f9a3caacc63419be489aefec02a5eae338c33'
curl -v -X DELETE -H $(echo $TOKENARG) http://localhost:3001/api/v1/articles/3
And desired result is:
curl -v -X DELETE -H 'Authorization: Bearer 9042f9a3caacc63419be489aefec02a5eae338c33' http://localhost:3001/api/v1/articles/3
But when I run the above one it doesn't work as expected and returns the following messages:
curl: (6) Could not resolve host: Bearer
* Rebuilt URL to: 9042f9a3caacc63419be489aefec02a5eae338c33
Could not resolve host: 9042f9a3caacc63419be489aefec02a5eae338c33
How should I pass argument using a variable?
Because you don't quote your command substitution, the string is split up into three words, Authorization:, Bearer and 9042f9a3caacc63419be489aefec02a5eae338c33. To prevent that, you could quote it:
curl -v -X DELETE -H "$(echo $TOKENARG)" http://localhost:3001/api/v1/articles/3
But you don't need echo and the command substitution at all: $(command) has the same output as command, so you can use
curl -v -X DELETE -H "$TOKENARG" http://localhost:3001/api/v1/articles/3
Side note: all uppercase variable names should be used for environment variables only, see for example the POSIX spec (fourth paragraph), so better would be
$tokenarg='Authorization: Bearer 9042f9a3caacc63419be489aefec02a5eae338c33'
curl -v -X DELETE -H "$tokenarg" http://localhost:3001/api/v1/articles/3
Related
I want to pass multiple arguments through to curl. Some of these arguments are quoted and contain spaces.
I have tried like this:
ARGS="http://example.org -H 'My-Header: Foo'"
curl -vvv $ARGS
But my header is not set and I get an error at the end curl: (6) Could not resolve host: Foo'.
I have also tried quoting ARGS like this:
ARGS="http://example.org -H 'My-Header: Foo'"
curl -vvv "$ARGS"
But I get curl: (3) URL using bad/illegal format or missing URL.
If I just run curl with the arguments directly, then it works fine:
curl -vvv http://example.org -H 'My-Header: Foo'
How can I pass these arguments through to curl correctly?
There is a command called eval which evaluates all the arguments into one string and then runs it as a one big command.
Try eval curl $ARGS
I recommend you to checkout eval's man page ;)
I am optionally including a user and password in a curl request as follows:
declare creds=""
if [ -n "$user" ] && [ -n "$password" ]; then
creds="-u ${user}:${password}"
fi
output=$(curl ${creds} -X PUT -v --write-out "%{http_code}" "$url" \
-H 'Content-Type: application/json' -s -o /dev/null --data "${payload}")
This seems to work fine, but I'm getting this shellcheck warning:
Double quote to prevent globbing and word splitting
https://github.com/koalaman/shellcheck/wiki/SC2086
Puting quotes around it doesn't work, e.g. if I do this:
output=$(curl "${creds}" -X PUT -v --write-out "%{http_code}" "$url" \
-H 'Content-Type: application/json' -s -o /dev/null --data "${payload}")
then when the username and password are not supplied, this results in empty double quotes in the curl request curl "" -X PUT ..., which generates a <url> malformed error.
I could use an if-else for the curl command, but I'd rather avoid the duplication. Is the above approach acceptable despite the shellcheck warning?
You were doing right in putting quotes around the variable, but shellcheck doesn't catch the issue of storing commands in a variable which has its own downfalls. Since this being a an issue with function of the shell, shellcheck can't quite catch it out-of-the-box. When you did below
creds="-u ${user}:${password}"
and quoted "$creds", it is passed as one single argument word to curl instead of being broken down to -u and "${user}:${password}" separately. The right approach should have been to use an array to store the commands and expand it, so that the words are preserved and not split by the shell (foremost reason to quote the variable, as indicated by shellcheck)
creds=(-u "${user}:${password}")
and invoke
curl "${creds[#]}" <rest-of-the-cmd>
Also explore the following
I'm trying to put a command in a variable, but the complex cases always fail!
How to store a command in a variable in a shell script?
I have the following bash script that uses its arguments to hit a RESTful web service (via curl) and prints out both the curl request made as well as the response:
#! /bin/bash
# arguments:
# $1 - username
# $2 - password
#
# outputs:
# if the script exits with a non-zero status then something went wrong
# verify that we have all 6 required arguments and fail otherwise
if [ "$#" -ne 2 ]; then
echo "Required arguments not provided"
exit 1
fi
# set the script arguments to meaningful variable names
username=$1
password=$2
# login and fetch a valid auth token
req='curl -k -i -H "Content-Type: application/json" -X POST -d ''{"username":"$username","password":"$password"}'' https://somerepo.example.com/flimflam'
resp=$(curl -k -i -H "Content-Type: application/json" -X POST -d ''{"username":"$username","password":"$password"}'' https://somerepo.example.com/flimflam)
# echo the request for troubleshooting
echo "req = $req"
if [ -z "$resp" ]; then
echo "Login failed; unable to parse response"
exit 1
fi
echo "resp = $resp"
When I run this I get:
$ sh myscript.sh myUser 12345#45678
curl: (3) Port number ended with '"'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: 12345#45678"
100 1107 100 1093 100 14 2849 36 --:--:-- --:--:-- --:--:-- 2849
req = curl -k -i -H "Content-Type: application/json" -X POST -d {"username":"$username","password":"$password"} https://somerepo.example.com/flimflam
resp = HTTP/1.1 400 Bad Request...(rest omitted for brevity)
Obviously, I'm not escaping the various layers of single- and double-quotes inside the curl statement correctly, as is indicated by outputs like:
curl: (6) Could not resolve host: 12345#45678"
and:
req = curl -k -i -H "Content-Type: application/json" -X POST -d {"username":"$username","password":"$password"} https://somerepo.example.com/flimflam
where the username/password variables are not parsing.
In reality my script takes a lot more than 2 arguments, which is why I'm changing them to have meaningful variable names (such as $username instead of $1) so its more understandable and readable.
Can anyone spot where I'm going awry? Thanks in advance!
Update
I tried the suggestion which turns the req into:
curl -k -i -H 'Content-Type: application/json' -X POST -d "{'username':'myUser','password':'12345#45678'}" https://somerepo.example.com/flimflam
However this is still an illegal curl command and instead needs to be:
curl -k -i -H 'Content-Type: application/json' -X POST -d '{"username":"myUser","password":"12345#45678"}' https://somerepo.example.com/flimflam
First, as I said in a comment, storing commands in variables just doesn't work right. Variables are for data, not executable code. Second, you have two levels of quoting here: quotes that're part of the shell syntax (which are parsed, applied, and removed by the shell before the arguments are passed to `curl), and quotes that're part of the JSON syntax.
But the second problem is actually worse than that, because simply embedding an arbitrary string into some JSON may result in JSON syntax errors if the string contains characters that're part of JSON syntax. Which passwords are likely to do. To get the password (and username for that matter) embedded correctly in your JSON, use a tool that understands JSON syntax, like jq:
userinfo=$(jq -n -c --arg u "$username" --arg p "$password" '{"username":$u,"password":$p}')
Explanation: this uses --arg to set the jq variables u and p to the shell variables $username and $password respectively (and the double-quotes around the shell variables will keep the shell from doing anything silly to the values), and creates a JSON snippet with them embedded. jq will automatically add appropriate quoting/escaping/whatever is needed.
Then, to use it with curl, use something like this:
resp=$(curl -k -i -H "Content-Type: application/json" -X POST -d "$userinfo" https://somerepo.example.com/flimflam)
Again, the double-quotes around $userinfo keep the shell from doing anything silly. You should almost always put double-quotes around variables references in the shell.
Note that I never used the req variable to store the command. If you need to print the command (or its equivalent), use something like this:
printf '%q ' curl -k -i -H "Content-Type: application/json" -X POST -d "$userinfo" https://somerepo.example.com/flimflam
echo
The %q format specifier tells the shell to add appropriate quoting/escaping so that you could run the result as a shell command, and it'd work properly. (And the echo is there because printf doesn't automatically add a newline at the end of its output.)
try changing this:
req='curl -k -i -H "Content-Type: application/json" -X POST -d ''{"username":"$username","password":"$password"}'' https://somerepo.example.com/flimflam'
to this
req="curl -k -i -H 'Content-Type: application/json' -X POST -d \"{'username':'$username','password':'$password'}\" https://somerepo.example.com/flimflam"
and similarly for the resp
ah those pesky "curly" thingies...
how 'bout...
req="curl -k -i -H 'Content-Type: application/json' -X POST -d '{\"username\":\"$username\",\"password\":\"$password\"}' https://somerepo.example.com/flimflam"
This needs even more escaping:
With:
resp=$(curl -k -i -H "Content-Type: application/json" -X POST -d "{\"username\":\"$username\",\"password\":\"$password\"}" https://somerepo.example.com/flimflam)
In bash, the variables are still expanded when they're inside single quotes that are inside double quotes.
And you'll need the \" double quotes in the payload as per the JSON definition.
EDIT: I rerun the curl through a HTTP proxy and corrected the script line (see above, removed the single quotes). Results (in raw HTTP) are now:
POST /flimflam HTTP/1.1
Host: somerepo.example.com
User-Agent: curl/7.68.0
Accept: */*
Content-Type: application/json
Content-Length: 44
Connection: close
{"username":"user","password":"12345#abcde"}
(which should be fine)
I want to create a script to test some web service.
Here the base of my script:
XAuthToken=$(curl -i -s -X POST -H "Content-Type: application/json" -d '{"login": "apitest","password": "apitest"}' http://url:8080/rest-api-web/api/admin | grep -i '^X-Auth-Token:' | sed 's/^[Xx]-Auth-Token: //g')
curl -X GET --header 'Accept: application/json' http://url:8080/rest-api-web/api/admin/delivery_companies?token=$XAuthToken
Result of the first command is:
eyJjdHkiOiJKV1QiLCJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.KmhGpDsor4I7VVPfN1gENzcQb8Yll-wewTRorFu6qfUeaIjKNIAm5lkTkPIpuuOuSpT2U4RgXz9NhwLrYIheW45Du6LnjsyUEjjWGKB4jIW0CMO-79f_-O80pQMq0P09uwmZaqUOmARPjs8UvQYQcuCHhcHd2dB-4WMnfUl-J6USI2LdI-CeOtWDDMw5YEKHGrgnHRzxgUePNWVs-Vc-BR-tXnpkEjIfKmrYy19QUfnwxcWLEWgKisrJw5d7sz7ihfzSueyrh188GiGHTeqqiNiSIkIH8UEyZfW-5BH1Y856IkXlyUaC3abHBRYHuz-O30KwvCKhtCKkaTPVR-uqNw.89jCGtawkkWuyRtq.pQmYHaWZN-XOlbvUthjK76ewqIGd1TBF0Pt7EbA2bDnxsTsXQXyot495-u4Rsvm8-y-rscAZUZFacebMvcvZ-LhUH-jHZH6L5PALY7uvGj_SGApg_Hu8D-adNDlI1rVxNcnHjpOLuOeBdUzKAznbOytquzQNODsjP877IbLTDkWTTj0IC0OWfDu-e4rYxpf2jzZBuEbnRcC8DuvbplNB2tnbRxMjpqnKjicLY9DpYdV1T-tKUt5-sI0Bhlk6tUJ0e28g1McQReUT-sB8pO3vQreEAEsVQlysVq9rC6LiW6RNRHAZP7d6PiuhLS_D_DKVmi85junbMVimYqbvszaJ.1b7PKcZfWmVmNBbTg54nFA
But the second curl in the script, this error occurs: curl: (3) Illegal characters found in URL
And after a check, the content of "XAuthToken" is changed.
Can you help me to escape the character modification?
Thanks you for your comment.
As you can see in the following output, both declare are egal. But the oupt of the command with set -x activate, the Authtoken is changed.
Here the result :
First delare -p :
eyJjdHkiOiJKV1QiLCJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.BTri-oCE-DHYx7UZo8Kib_RoO5bx9lv7BRhEYv2yizyX2n8cwzWWhMPPA-0rDHumaESgc0LmT71NLIJEyZkwN-pXH2MgYdo2vTQhnVdFOL7rZq1rHSJvUxPvhJTbAkabnTNz-s1-Vc7WNQXmWEkngaCISdnsFQU12YfJfzhgwGZi9OOf5gmHnNtXaAxTOrDcjl84s1QI0tY5HIQ_3MEB18V4UMfcR5SSk4v_5K2yVZk-AUys3bPDw1GEZkr3rypxytNZ6wCyrOEXtGh1s8w3fFQziNnnOcgwaSkpDOdhhJWUfgN3R1Cr5tnD77vG01SZw7k844LZxiK8HXhQUj3_sQ.KAhqX5YkZPXDtVlI.d-TxeoNsuemKRfDpQe9khaYym1KACAXS9MhYfGtoIwhPomZ85hKvILeT8jXxBgIw6r4XUpgSSVNtjUzA7A-_hpu0i7ffd-Ap-YDohbCfJfCYffVO557tCuIhVvybT4qQ5EgjsX8h1V5NqIyDVIPAiDIc6hnrxWsjbwE3dmfMLxqmDLXLiYTaCUMvsxtweo-fsdIymK-REuy4DdPk8YsaITpfYj57Ee5ZphZiNfvUPixkLAXj97ycEnXbP2d6q6Aw1rYVIrE3ijGYIgW618kMs5PBvjWe"yh-76mOq9_0QnKzblZWJytHsM6DM8kbkGRHhwFhQSvjq9HUQVNmmKCI.Q4rq7-uzPW3Yewc8Wyl01A
Resut of "set -x' :
eyJjdHkiOiJKV1QiLCJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.BTri-oCE-DHYx7UZo8Kib_RoO5bx9lv7BRhEYv2yizyX2n8cwzWWhMPPA-0rDHumaESgc0LmT71NLIJEyZkwN-pXH2MgYdo2vTQhnVdFOL7rZq1rHSJvUxPvhJTbAkabnTNz-s1-Vc7WNQXmWEkngaCISdnsFQU12YfJfzhgwGZi9OOf5gmHnNtXaAxTOrDcjl84s1QI0tY5HIQ_3MEB18V4UMfcR5SSk4v_5K2yVZk-AUys3bPDw1GEZkr3rypxytNZ6wCyrOEXtGh1s8w3fFQziNnnOcgwaSkpDOdhhJWUfgN3R1Cr5tnD77vG01SZw7k844LZxiK8HXhQUj3_sQ.KAhqX5YkZPXDtVlI.d-TxeoNsuemKRfDpQe9khaYym1KACAXS9MhYfGtoIwhPomZ85hKvILeT8jXxBgIw6r4XUpgSSVNtjUzA7A-_hpu0i7ffd-Ap-YDohbCfJfCYffVO557tCuIhVvybT4qQ5EgjsX8h1V5NqIyDVIPAiDIc6hnrxWsjbwE3dmfMLxqmDLXLiYTaCUMvsxtweo-fsdIymK-REuy4DdPk8YsaITpfYj57Ee5ZphZiNfvUPixkLAXj97ycEnXbP2d6q6Aw1rYVIrE3ijGYIgW618kMs5PBvjWeTyh-76mOq9_0QnKzblZWJyt'sM6DM8kbkGRHhwFhQSvjq9HUQVNmmKCI.Q4rq7-uzPW3Yewc8Wyl01A
Second declare :
eyJjdHkiOiJKV1QiLCJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.BTri-oCE-DHYx7UZo8Kib_RoO5bx9lv7BRhEYv2yizyX2n8cwzWWhMPPA-0rDHumaESgc0LmT71NLIJEyZkwN-pXH2MgYdo2vTQhnVdFOL7rZq1rHSJvUxPvhJTbAkabnTNz-s1-Vc7WNQXmWEkngaCISdnsFQU12YfJfzhgwGZi9OOf5gmHnNtXaAxTOrDcjl84s1QI0tY5HIQ_3MEB18V4UMfcR5SSk4v_5K2yVZk-AUys3bPDw1GEZkr3rypxytNZ6wCyrOEXtGh1s8w3fFQziNnnOcgwaSkpDOdhhJWUfgN3R1Cr5tnD77vG01SZw7k844LZxiK8HXhQUj3_sQ.KAhqX5YkZPXDtVlI.d-TxeoNsuemKRfDpQe9khaYym1KACAXS9MhYfGtoIwhPomZ85hKvILeT8jXxBgIw6r4XUpgSSVNtjUzA7A-_hpu0i7ffd-Ap-YDohbCfJfCYffVO557tCuIhVvybT4qQ5EgjsX8h1V5NqIyDVIPAiDIc6hnrxWsjbwE3dmfMLxqmDLXLiYTaCUMvsxtweo-fsdIymK-REuy4DdPk8YsaITpfYj57Ee5ZphZiNfvUPixkLAXj97ycEnXbP2d6q6Aw1rYVIrE3ijGYIgW618kMs5PBvjWe"yh-76mOq9_0QnKzblZWJytHsM6DM8kbkGRHhwFhQSvjq9HUQVNmmKCI.Q4rq7-uzPW3Yewc8Wyl01A
I have written a smoke-testing script that uses BASH script & Curl to test RESTful web services we're working on. The script reads a file, and interprets each line as a URL suffix and parameters for a Curl REST call.
Unfortunately, the script gives unexpected results when I adapted it to run HTTP POST calls as well as GET calls. It does not give the same results running the command on its own, vs. in script:
The BASH Script:
IFS=$'\n' #Don't split an input URL line at spaces
RESTHOST='hostNameAndPath' #Can't give this out
URL="/activation/v2/activationInfo --header 'Content-Type:Application/xml'"
URL2="/activation/v2/activationInfo"
OUTPUT=`curl -sL -m 30 -w "%{http_code}" -o /dev/null $RESTHOST$URL -d #"./activation_post.txt" -X POST`
echo 'out:' $OUTPUT
OUTPUT2=`curl -sL -m 30 -w "%{http_code}" -o /dev/null $RESTHOST$URL2 --header 'Content-Type:Application/xml' -d #'./activation_post.txt' -X POST`
echo 'out2:' $OUTPUT2
Results Out:
out: 505
out2: 200
So, the first call fails (HTTP return code 505, HTTP Version Not Supported), and the second call succeeds (return code "OK").
Why does the first call fail, and how do I fix it? I've verified they should execute the same command (evaluating in echo). I am sure there is something basic I'm missing, as I am just NOW learning Bash scripting.
I think I have found the problem! It is caused by IFS=$'\n'! Because of this, variable expansion does not work as expected. It does not let to split the arguments specified in the URL string!
As a result the SERVER_PROTOCOL variable on the server side will be set to '--header Content-Type:Application/xml HTTP/1.1' instead of "HTTP/1.1", and the CONTENT_TYPE will be 'application/x-www-form-urlencoded' instead of 'Application/xml'.
To show the root of the problem in detail:
VAR="Solaris East"
printf "+%s+ " $VAR
echo "==="
IFS=$'\n'
printf "+%s+ " $VAR
Output:
+Solaris+ +East+ ===
+Solaris East+
So the $VAR expansion does not work as expected because of IFS=$'\n'!
Solution: Do not use IFS=$'\n' and replace space to %20 in URL!
URL=${URL2// /%20}" --header Content-Type:Application/xml"
In this case your first curl call will work properly!
If You still use IFS=$'\n' and give --header option in the command line it will not work properly if URL contains a space, because the server will fail to process it (I tested on apache)!
Even You still cannot use HEADER="--header Content-Type:Application/xml" as expanding $HEADER will result one(!) argument for curl, namely --header Content-Type:Application/xml instead of splitting them into two.
So I may suggest to replace spaces in URL to %20 anyway!
The single quotes surrounding Content-Type:Application/xml, because they are quoted in the value of URL are treated as literal quotes and not removed when $URL is expanded in that call to curl. As a result, you are passing an invalid HTTP header. Just use
URL="/activation/v2/activationInfo --header Content-Type:Application/xml"
OUTPUT=`curl -sL -m 30 -w "%{http_code}" -o /dev/null $RESTHOST$URL -d #"./activation_post.txt" -X POST`
However, it's not a great idea to rely on word-splitting like this to combine two separate pieces of the call to curl in a single variable. Try something like this instead:
URLPATH="activation/v2/activationInfo"
HEADERS=("--header" "Content-Type:Application/xml")
OUTPUT=$( curl -SL -m 30 -w "%{http_code}" -o /dev/null "$RESTHOST/$URL" "${HEADERS[#]}" -d #'./activation_post.txt' -X POST )