I am trying to run a curl command within a bash, when running the curl outside separately its working fine, but same running with Bash throws bad request 400 error. Would appreciate if someone can point where I am going wrong
#!/bin/bash
TIMESTAMP=$(date +%Y-%m-%d_%H-%M-%S)
curl http://localhost/cron/abc -H 'content-type: application/json'
echo 'Cron Executed at:' "$TIMESTAMP" | tee /var/log/cron.txt
I have the following bash script that uses its arguments to hit a RESTful web service (via curl) and prints out both the curl request made as well as the response:
#! /bin/bash
# arguments:
# $1 - username
# $2 - password
#
# outputs:
# if the script exits with a non-zero status then something went wrong
# verify that we have all 6 required arguments and fail otherwise
if [ "$#" -ne 2 ]; then
echo "Required arguments not provided"
exit 1
fi
# set the script arguments to meaningful variable names
username=$1
password=$2
# login and fetch a valid auth token
req='curl -k -i -H "Content-Type: application/json" -X POST -d ''{"username":"$username","password":"$password"}'' https://somerepo.example.com/flimflam'
resp=$(curl -k -i -H "Content-Type: application/json" -X POST -d ''{"username":"$username","password":"$password"}'' https://somerepo.example.com/flimflam)
# echo the request for troubleshooting
echo "req = $req"
if [ -z "$resp" ]; then
echo "Login failed; unable to parse response"
exit 1
fi
echo "resp = $resp"
When I run this I get:
$ sh myscript.sh myUser 12345#45678
curl: (3) Port number ended with '"'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: 12345#45678"
100 1107 100 1093 100 14 2849 36 --:--:-- --:--:-- --:--:-- 2849
req = curl -k -i -H "Content-Type: application/json" -X POST -d {"username":"$username","password":"$password"} https://somerepo.example.com/flimflam
resp = HTTP/1.1 400 Bad Request...(rest omitted for brevity)
Obviously, I'm not escaping the various layers of single- and double-quotes inside the curl statement correctly, as is indicated by outputs like:
curl: (6) Could not resolve host: 12345#45678"
and:
req = curl -k -i -H "Content-Type: application/json" -X POST -d {"username":"$username","password":"$password"} https://somerepo.example.com/flimflam
where the username/password variables are not parsing.
In reality my script takes a lot more than 2 arguments, which is why I'm changing them to have meaningful variable names (such as $username instead of $1) so its more understandable and readable.
Can anyone spot where I'm going awry? Thanks in advance!
Update
I tried the suggestion which turns the req into:
curl -k -i -H 'Content-Type: application/json' -X POST -d "{'username':'myUser','password':'12345#45678'}" https://somerepo.example.com/flimflam
However this is still an illegal curl command and instead needs to be:
curl -k -i -H 'Content-Type: application/json' -X POST -d '{"username":"myUser","password":"12345#45678"}' https://somerepo.example.com/flimflam
First, as I said in a comment, storing commands in variables just doesn't work right. Variables are for data, not executable code. Second, you have two levels of quoting here: quotes that're part of the shell syntax (which are parsed, applied, and removed by the shell before the arguments are passed to `curl), and quotes that're part of the JSON syntax.
But the second problem is actually worse than that, because simply embedding an arbitrary string into some JSON may result in JSON syntax errors if the string contains characters that're part of JSON syntax. Which passwords are likely to do. To get the password (and username for that matter) embedded correctly in your JSON, use a tool that understands JSON syntax, like jq:
userinfo=$(jq -n -c --arg u "$username" --arg p "$password" '{"username":$u,"password":$p}')
Explanation: this uses --arg to set the jq variables u and p to the shell variables $username and $password respectively (and the double-quotes around the shell variables will keep the shell from doing anything silly to the values), and creates a JSON snippet with them embedded. jq will automatically add appropriate quoting/escaping/whatever is needed.
Then, to use it with curl, use something like this:
resp=$(curl -k -i -H "Content-Type: application/json" -X POST -d "$userinfo" https://somerepo.example.com/flimflam)
Again, the double-quotes around $userinfo keep the shell from doing anything silly. You should almost always put double-quotes around variables references in the shell.
Note that I never used the req variable to store the command. If you need to print the command (or its equivalent), use something like this:
printf '%q ' curl -k -i -H "Content-Type: application/json" -X POST -d "$userinfo" https://somerepo.example.com/flimflam
echo
The %q format specifier tells the shell to add appropriate quoting/escaping so that you could run the result as a shell command, and it'd work properly. (And the echo is there because printf doesn't automatically add a newline at the end of its output.)
try changing this:
req='curl -k -i -H "Content-Type: application/json" -X POST -d ''{"username":"$username","password":"$password"}'' https://somerepo.example.com/flimflam'
to this
req="curl -k -i -H 'Content-Type: application/json' -X POST -d \"{'username':'$username','password':'$password'}\" https://somerepo.example.com/flimflam"
and similarly for the resp
ah those pesky "curly" thingies...
how 'bout...
req="curl -k -i -H 'Content-Type: application/json' -X POST -d '{\"username\":\"$username\",\"password\":\"$password\"}' https://somerepo.example.com/flimflam"
This needs even more escaping:
With:
resp=$(curl -k -i -H "Content-Type: application/json" -X POST -d "{\"username\":\"$username\",\"password\":\"$password\"}" https://somerepo.example.com/flimflam)
In bash, the variables are still expanded when they're inside single quotes that are inside double quotes.
And you'll need the \" double quotes in the payload as per the JSON definition.
EDIT: I rerun the curl through a HTTP proxy and corrected the script line (see above, removed the single quotes). Results (in raw HTTP) are now:
POST /flimflam HTTP/1.1
Host: somerepo.example.com
User-Agent: curl/7.68.0
Accept: */*
Content-Type: application/json
Content-Length: 44
Connection: close
{"username":"user","password":"12345#abcde"}
(which should be fine)
I would like to create a curl output live in a single shell command, to log a output from an Ansible job in realtime filling a log file.
I've tried this command:
curl -f -k -N -H 'Content-Type: application/json' -XPOST \
--user admin:awxsecret \
http://192.168.42.100/api/v2/jobs/1620/
...but it only returns the output generated thus far, not waiting for newly-generated content.
As #charles-duffy said: "AWX does support websockets" I will work with this solution.
I have been trying to execute curl command from a script file.
The command is as follows (the IP address is stored in variable ip):
curl -s -X POST ${ip1} -H \"content-type: application/json\" -d \''{"args":["org1","scatest'$j$i'","27-06-2018"]}'\'
It's throwing an error saying "cannot read property", if I execute from the script file.
Whereas if I execute from the command line then there is no problem.
Can anybody help me out why curl command is not getting executed from the script file?
Since the discussions under the comment has lead to a solution that works well. I'm gonna put it into answer for the record and in case anyone else runs into the same pitfall.
The problem was caused by excessive quoting which would result in extra quotes being made part of the request, correct form of the command was:
curl -s -X POST ${ip1} -H "content-type: application/json" \
-d '{"args":["org1","scatest'$j$i'","27-06-2018"]}'
I'm using curl to create several classifications. I have written the json for the many classifications and they are in one folder. I would like to create all the classifications in one go. But using curl I can only create them one at a time. How could I make them in one request?
curl -u admin:admin -H "Content-Type: application/json" -X POST -d #pii.json http://127.0.0.1:21000/api/atlas/v2/types/typedefs
The curl manual for -d says 'Multiple files can also be specified'. How can I do this? All my attempts have failed.
Do I need a bash script instead? If so, could you help me - I'm not a coder and I'm struggling without an example!
Thanks in advance.
You probably don't want to use multiple -d with JSON data since curl concatenates multiple ones with a & in between. As described in the man page for -d/--data:
If any of these options is used more than once on the same command
line, the data pieces specified will be merged together with a
separating &-symbol. Thus, using '-d name=daniel -d skill=lousy' would
generate a post chunk that looks like 'name=daniel&skill=lousy'.
You can however easily and conveniently pass several files on stdin to let curl use them all in one go:
cat a.json b.json c.json | curl -d#- -u admin:admin -H "Content-Type: application/json" http://127.0.0.1:21000/api/atlas/v2/types/typedefs
(please note that -X POST has no place on a command line that uses -d)
I found the following to work in the end:
<fileToUpload.dat xargs -I % curl -X POST -T "{%}" -u admin:admin -H "Content-Type: application/json" http://127.0.0.1:21000/api/atlas/v2/types/typedefs
Where fileToUpload.dat contained a list of the .json files.
This seemed to work over Daniel's answer, probably due to the contents of the files. Hopefully this is useful to others if Daniel's solution doesn't work for them.
I needed to upload all the *.json files from a folder via curl and I made this little script.
nfiles=*.json
echo "Enter user:"
read user
echo "Enter password:"
read -s password
for file in $nfiles
do
echo -e "\n----$file----"
curl --user $user:$password -i -X POST "https://foo.bar/foo/bar" -H "Content-Type: application/json" -d "#$file"
done
Maybe fits your needs.