I'm making a curl request to a REST API and now I want to save the HTTP answer code, just if an error occurs, to a logfile and the API answer to another file if there is no error occuring.
I'm trying it with:
error=$(curl -v -o "test.json" -H "Authorization: Basic ABCDEF" "https://api.abc.com")
and
error=$(curl --fail -o "test.json" -H "Authorization: Basic ABCDEF" "https://api.abc.com")
If I make an if [0 -eq $? ] after the curl request with --fail I can detect that an error occurred but I am not able to save the HTTP error to a log.
Thanks.
Since according man curl the option --fail
... is not fail-safe and there are occasions where non-successful response codes will slip through, especially when authentication is involved (response codes 401 and 407).
you could use --write-out and http_code.
The numerical response code that was found in the last retrieved HTTP(S) or FTP(s) transfer.
I.e.:
ERROR=$(curl --silent --fail --header "Authorization: Basic ABCDEF" "https://api.abc.com" --output "test.json" --write-out "%{http_code}")
Welcome to stackoverflow.
This should do the trick.
#send all output to file named out
curl -v -o "test.json" -H "Authorization: Basic ABCDEF" "https://api.abc.com" >out 2>&1
# find HTTP/2 code in the output
error=`grep "HTTP/2" out | tail -1 | rev | cut -c1-5 | rev`
# print the error code
echo $error
Example run:
mamuns-mac:jenkins xmrashid$ ./get_error.sh
503
mamuns-mac:jenkins xmrashid$
Good Luck.
Related
I 'd like to post directly a json object from a url(json) to another url
so the command goes as follows:
curl "<resource_link>.json" -o sample.json
curl -X POST "<my_link>" "Content-type: application/json" -d #sample.json
I 'd like to avoid this, so what is the solution? Is it something like that?
curl -X POST "<my_link>" "Content-type: application/json" -d "curl <resource_link>.json"
But it does not work? Also, this one post Stream cURL response to another cURL command posting the result
does not explain thouroughly and it is not working
Yes,
curl
manual explains the '#' but it does not explain about using another curl
Alternatievely, if I could save somewhere temporarily the 1st cURL response and use it in the other command(but not in a file)
You don't want -x POST in there so let's start with dropping that.
Send the results from the first transfer to stdout by not using -o, or telling -o to use stdout with -o-, and
Make sure your second transfer accepts the data to send on stdin, by using -d#-.
curl "<link>.json" | curl "<link2>" -H "Content-type: application/json" -d #-
With curl 7.82.0 and later
Starting with curl 7.82.0 you can do it even easier with the new --json option:
curl "<link>.json" | curl "<link2>" --json #-
This suppresses the output and just outputs the status:
curl --write-out '%{http_code}' --silent --output /dev/null --noproxy '*' http://www.google.com/
Adding a header makes the entire response print:
curl --write-out '%{http_code}' --silent --output /dev/null --noproxy '*' --header ''"'"'Host:' '192.168.0.1:2345'"'" http://www.google.com/
How can I stop the output printing to stdout when injecting a header?
Let's take a closer look at your command. Below invocation will write each separate argument in a new line.
$ printf '%q\n' curl --write-out '%{http_code}' --silent --output /dev/null --noproxy '*' --header ''"'"'Host:' '192.168.0.1:2345'"'" http://www.google.com/
curl
--write-out
%\{http_code\}
--silent
--output
/dev/null
--noproxy
\*
--header
\'Host:
192.168.0.1:2345\'
http://www.google.com/
Bingo, though Host: 192.168.0.1:2345 must be a single argument, you're providing it to curl as two separate arguments, so it tries to fetch 192.168.0.1:2345' first. And since --output is applied to only one URL, the response from http://www.google.com/ is printed.
Do it like this and it'll work.
curl --write-out '%{http_code}\n' --silent --output /dev/null --noproxy '*' --header 'Host: 192.168.0.1:2345' http://www.google.com/
EDIT:
I have managed to make it work with
response=$(
curl -k -X PUT -d "$marathon_payload" --write-out %{http_code} --silent --output "$tmp"\
-H "Authorization: token=$dcos_token" -H "$header_content_type" $app_id_url
)
The single quotes were causing the problem. It took a few gyrations but all good.
MORAL: quotes inside the value don't matter if the value is properly quoted UNLESS you eval the whole thing, and I should have known that. Occam's wins again.
end edit
I am initiating Mesosphere microservice deployments with curl, but it won't succeed without using eval. Since I recently inherited this code I've been trying to scrub the eval out of it just as a matter of habit, but it's thwarting me.
The script initiates the deployment with
response=$(
eval curl -k -X PUT -d "'$marathon_payload'" --write-out %{http_code} --silent --output $tmp\
-H "'Authorization: token=$dcos_token'" -H "'$header_content_type'" $app_id_url
)
If it gets a 200 or a 201, it loops a curl to effectively screen-scrape the deployments page till the request disappears.
chkDeploy() { rm -f $tmp;
eval curl -k -X GET --silent --write-out %{http_code} --silent --output $tmp\
-H "'Authorization: token=$dcos_token'" -H "'$header_content_type'" $deployments_url
}
response=$( chkDeploy )
$dcos_token is a base64 encoded string.
It then checks the service with another curl loop to the info page so it can verify the version number. This one is working fine with no eval.
chkCode() {
curl -k -X GET --write-out %{http_code} --silent --output $tmp $info_url;
}
response=$( chkCode )
The first two return 401, authentication failure.
I'm guessing the auth token quoting is off.
There's no reason to use eval here; you just need to quote the arguments to -H properly.
response=$(
curl -k -X PUT -d "$marathon_payload" \
--write-out %{http_code} \
--silent --output "$tmp" \
-H "Authorization: token=$dcos_token" \
-H "$header_content_type" "$app_id_url"
)
I have a simple bash file as below
#!/bin/bash
net=$(curl -s -H "Content-Type: application/json" -H "X-Auth-Token: $token" -d '{"network": {"name": "net1"}}' http://10.1.10.146:18090/network/v2.0/networks 2>&1 | awk '/id/{print $1}' | jq -r .network.id)
echo $net
Running this file gives me an error as below
parse error: Invalid numeric literal at line 2, column 0
tried making the changes according to these links
https://unix.stackexchange.com/questions/354943/setting-jq-output-to-a-bash-variable
Working with Bash and cURL
but nothing helped me, unable to figure out where i am going wrong. let me know the reason for the error and possible changes.
The curl output for the command
curl -s -H "Content-Type: application/json" -H "X-Auth-Token: $token" -d '{"network": {"name": "net1"}}' http://10.1.10.146:18090/network/v2.0/networks
output:
{"network":{"status":"ACTIVE","router:external":false,"availability_zone_hints":[],"availability_zones":[],"description":"","subnets":[],"shared":false,"tenant_id":"d0e75710820c401db3291ac6278f326f","created_at":"2018-05-15T07:37:42Z","tags":[],"ipv6_address_scope":null,"mtu":1450,"updated_at":"2018-05-15T07:37:42Z","admin_state_up":true,"revision_number":2,"ipv4_address_scope":null,"is_default":false,"port_security_enabled":true,"project_id":"d0e75710820c401db3291ac6278f326f","id":"1548df56-a35b-4232-9550-54a3c2266d60","name":"net1"}}
the idea is to get only the id from the output and store into a bash variable, to get the id i used the below command
curl -s -H "Content-Type: application/json" -H "X-Auth-Token: $token" -d '{"network": {"name": "net1"}}' http://10.1.10.146:18090/network/v2.0/networks 2>&1 | awk '/id/{print $1}' | jq -r .network.id
output:
be831582-90c1-499c-875f-9c0b0d1969a6
I have also tried removing the awk and parsing the curl json response, the same error is showing up.
thanks in advance.
The "parse error" message appears because the 2>&1 redirects any STDERR message into jq, which cannot parse it. Compare the output from these commands:
> curl http://no.such.host/network/v2.0/networks 2>&1
curl: (6) Could not resolve host: no.such.host
> curl http://no.such.host/network/v2.0/networks 2>&1 | jq '.'
parse error: Invalid numeric literal at line 1, column 4
Here are some ideas:
Separate all of the piped commands into separate commands.
Try adding set -x near the top of the script to "debug" all of the
commands.
Remove the 2>&1 because it is NOT helping you!
Add error handling to the separate commands.
We could help more if you provided the curl output. (We do not have HTTP access to the 10.1.10.146 host.)
I have a curl request like this :
curl -s -u $user:$password -X GET -H "Content-Type: application/json" $url
Which returns a json as response. So I will parse the response using jq to get some specific data. Like this :
curl -s -u $user:$password -X GET -H "Content-Type: application/json" $url | jq '<expression>'
Now if the curl request fails then obviously the parsing operation throws ugly error. I want to avoid this. How to store the response first and then later parse it if the request is successful. I don't want to display the json whole response. Also if I add -w "%{http_code}" in my request it appends the status code with the JSON response which messes up the parsing. How to solve this ? I basically want to first check if the curl request is successful or not then get the JSON response and parse it.I also want to get the status code, so that if it fails I can display the status code. But status code is now messing up with json response.
You can combine the --write and --fail options:
# separating the (verbose) curl options into an array for readability
curl_args=(
--write "%{http_code}\n"
--fail
--silent
--user "$user:$password"
--request GET
--header "Content-Type: application/json"
)
if ! output=$(curl "${curl_args[#]}" "$url"); then
echo "Failure: code=$output"
else
# remove the "http_code" line from the end of the output, and parse it
sed '$d' <<<"$output" | jq '...'
fi
Also note: quote your variables!
I found glenn jackman's answer good, but a bit confusingly written, so I rewrote it, and altered it so I can use it as a safer alternative to curl | jq.
#!/bin/bash
# call this with normal curl arguments, especially url argument, e.g.
# safecurl.sh "http://example.com:8080/something/"
# separating the (verbose) curl options into an array for readability
curl_args=(
-H 'Accept:application/json'
-H 'Content-Type:application/json'
--write '\n%{http_code}\n'
--fail
--silent
)
echo "${curl_args[#]}"
# prepend some arguments, but pass on whatever arguments this script was called with
output=$(curl "${curl_args[#]}" "$#")
return_code=$?
if [ 0 -eq $return_code ]; then
# remove the "http_code" line from the end of the output, and parse it
echo "$output" | sed '$d' | jq .
else
# echo to stderr so further piping to jq will process empty output
>&2 echo "Failure: code=$output"
fi
Note: This code does not test for services that ignore the requested content type and respond with HTML. You'd need to test for grep -l '</html>' for that.