Get Json body and Response Status from Bash Script POST - bash

Currently I am using:
#!/bin/bash
PROCESS=$(curl --location --request -v -X POST 'https://jsonplaceholder.typicode.com/posts' \
--header 'Content-Type: application/json' \
--data-raw '{"title": "foo","body": "bar","userId": "1"}')
echo "$PROCESS"
And getting:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 111 100 67 100 44 208 137 --:--:-- --:--:-- --:--:-- 344
{
"title": "foo",
"body": "bar",
"userId": "1",
"id": 101
}
But I also want the response status e.g. 201 or like this.
HTTP/2 200
date: Mon, 30 Nov 2020 14:00:56 GMT
content-type: application/json; charset=utf-8
set-cookie: __cfduid=dfda1e85d5738eb18115dc0a07311a4dd1606744856; expires=Wed, 30-Dec-20 14:00:56 GMT; path=/; domain=.typicode.com; HttpOnly; SameSite=Lax
x-powered-by: Express
x-ratelimit-limit: 1000
x-ratelimit-remaining: 999
x-ratelimit-reset: 1606702897
vary: Origin, Accept-Encoding
access-control-allow-credentials: true
cache-control: max-age=43200
pragma: no-cache
expires: -1
x-content-type-options: nosniff
etag: W/"6b80-Ybsq/K6GwwqrYkAsFxqDXGC7DoM"
via: 1.1 vegur
cf-cache-status: HIT
age: 13185
cf-request-id: 06bb0df15c0000edfbfb9b8000000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=ABBCY6aKAHfezboFKgcq%2FlsWKQZDAORup49fKMArhm%2BYl3Kb99pMLrZpLtbXsfz%2BQ6RxnutmzE0mCX5AcIVGRjmq%2FIrIja5MeNFFnmpO7WBT1725PWdN1J0KFhcqNxvNP8He2TBjfd3N"}],"group":"cf-nel","max_age":604800}
nel: {"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 5fa518fbcbdfedfb-CDG
I want to do the post and the echo out body and response code in a nice way.

Response code is sent in HTTP headers.
You may redirect headers to STDERR e.g. as described here: Report HTTP Response Headers to stderr?
So you may do this:
out=$(curl -s -D /dev/stderr http://boardreader.com 2>/tmp/headers)
# parse /tmp/headers
If you don't want to mess with temp file, you may try more complex solutions like
Capture stdout and stderr into different variables

You can only issue either a post or header request in one call and so you will need to do this in two separate calls read into the same variable and so:
PROCESS=$(curl -I 'https://jsonplaceholder.typicode.com/posts' && curl -X POST 'https://jsonplaceholder.typicode.com/posts' --header 'Content-Type: application/json' --data '{"title": "foo","body": "bar","userId": "1"}')
To me it makes sense to check the headers first and if this command is successful, get the json response with both being read into the PROCESS variable. You can of course change the order if you wish.

Related

Extract response code and part of one specific header from curl on OSX

I am writing shell scripts in macOS 11.1 Big Sur to test several endpoints thru cUrl request. To improve my scripts, I would love to do the following:
send the curl request
store the response code to a variable in an array
store part of the response payload to another variable
I can not share the exact endpoint since that would be a security breach for others to test their solutions, unfortunately.
In example, using a modified request from one of my scripts:
#!/bin/sh
EMAIL_USERNAME=`date "+%Y%m%d%H%M%S"`
curl -iX POST 'http://test.someclientcloud.com/api/online/customer/register' -H 'Connection: keep-alive' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -H 'Authorization: XXXXXXXXXXXXXXX' -H 'Accept: application/json' -H 'X-Origin-Source: WWW' -H 'Content-Type: application/json' -H 'Accept-Language: en-US,en;q=0.9,pt;q=0.8' --data-binary $'{ "emailAddress": "'$EMAIL_USERNAME'#xxxxxx.com", "firstName": "Terminate", "lastName": "ME", "opt": true, "password": "Password7654!", "phoneNumber": "555229'$TODAY'", "regCity": "Anytown", "regCountryCode": "1", "regState": "TX", "regZip": "86753", "username": "'$EMAIL_USERNAME'#xxxxx.com" }'
The curl response is:
HTTP/1.1 201 Created
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Connection: close
Date: Wed, 06 Jan 2021 18:40:38 GMT
Pragma: no-cache
Transfer-Encoding: chunked
Location: /api/customer/354190751
Expires: 0
requestDuration: 67
X-Frame-Options: DENY
X-Origin-Source: WWW
X-Content-Type-Options: nosniff
proxyUser: WWW
X-XSS-Protection: 1; mode=block
X-Application-Context: application:production
The two things I would like from that response are 201 for the response code, from the first line of the response.
And 354190751 from the end of line 7 of the response above.
I can use these responses to form a report at the end of the script of each response per request. And I can use the newly created customer number in further tests within the script to modify the account further or delete the account for clean up at the end of the test.
I suspect this can be done with clever regex? My regex level is only moderately dangerous.
To be clear, I will mention, this is Apple Shell Scripting in macOS 11.1 Big Sur. Variants of shell scripting are many.
Hopefully the question was clear, thank you for your help.
Added that this is macOS 11.1 Big Sur
Unfortunately, while curl lets you ask it to directly write the status code, that doesn't extend to individual headers -- so you need to do the pattern matching yourself. One way to do that is as follows (note the use of process substitution syntax to avoid the issues discussed in BashFAQ #24, which makes this incompatible with /bin/sh; that said, because your code uses ANSI C-like string syntax, aka $'', it isn't compliant with the sh standard regardless):
#!/usr/bin/env bash
# put your actual curl code here; using a stub here so others can test
getCurlOutput() {
cat <<'EOF'
HTTP/1.1 201 Created
Transfer-Encoding: chunked
Location: /api/customer/354190751
Expires: 0
EOF
}
code=
customer=
{
read _ code _
prefix="Location: /api/customer/"
while IFS= read -r line; do line=${line%$'\r'}
case $line in
"$prefix"*) customer=${line#$prefix};;
esac
done
} < <(getCurlOutput)
echo "Retrieved exit status $code"
if [ -n "$customer" ]; then
echo "Customer id retrieved is $customer"
fi

curl post audio data with rate limit

I am trying to post audio data with curl for a HTTP-API which allows to transmit/receive audio files.
First I tried this:
curl -vv --http1.0 -H "Content-Type: audio/basic" -H "Content-Length: 9999999" -H "Connection: Keep-Alive" -H "Cache-Control: no-cache" --data-binary #- 'http://IP/API-Endpoint.cgi'
This seems to work:
* Trying [IP]...
* TCP_NODELAY set
* Connected to [IP] ([IP]) port 80 (#0)
> POST /API-Endpoint.cgi HTTP/1.0
> Host: [IP]
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: audio/basic
> Content-Length: 9999999
> Connection: Keep-Alive
> Cache-Control: no-cache
>
* upload completely sent off: 17456 out of 17456 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/plain
< Content-Length: 0
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
< Date: Wed, 06 Jun 2018 19:38:37 GMT
< Server: lighttpd/1.4.45
But I can only hear the very last part of the Audio file. (The file has the correct audio format for the API: G.711 μ-law with 8000 Hz) My next guess is, that the audio gets transmitted too fast and has to be sent in real time to the API endpoint. So I tried the --limit-rate parameter of curl, which had no effect. Then I tried piping the data with a rate limit into curl:
cat myfile.wav | pv -L 10k | curl -vv --http1.0 -H "Content-Type: audio/basic" -H "Content-Length: 9999999" -H "Connection: Keep-Alive" -H "Cache-Control: no-cache" --data-binary #- 'http://IP/API-Endpoint.cgi'
but the result is always the same: I can only hear the last part of the audio file. It seems like curl is waiting for the piped input to complete and then sends the request as before.
Is there an option to post audio to a HTTP-API from bash in "real time"?
Update:
Without forcing HTTP 1.0 I get the following result:
curl -vv -H "Content-Type: audio/basic" --data-binary '#myfile.wav' 'http://[IP]/API-Endpoint.cgi'
* Trying [IP]...
* TCP_NODELAY set
* Connected to [IP] ([IP]) port 80 (#0)
> POST /API-Endpoint.cgi HTTP/1.1
> Host: [IP]
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: audio/basic
> Content-Length: 15087
> Expect: 100-continue
>
< HTTP/1.1 417 Expectation Failed
< Content-Type: text/html
< Content-Length: 363
< Connection: close
< Date: Wed, 06 Jun 2018 20:34:22 GMT
< Server: lighttpd/1.4.45
<
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>417 - Expectation Failed</title>
</head>
<body>
<h1>417 - Expectation Failed</h1>
</body>
</html>
* Closing connection 0
with -H "Content-Length: 9999999" you say that your audio file is exactly 9999999 bytes long (roughly 10 megabytes), but curl reports that your file is 17456 bytes:
* upload completely sent off: 17456 out of 17456 bytes
(roughly 0.02 megabytes), so either your Content-Length header is wrong (that's my best guess), or the program feeding your audio file to curl is faulty, closing stdin prematurely.
either fix your Content-Length header, or fix the program feeding curl's stdin, hopefully that should send the entire file intact.
EDIT: oh, seems that server can't handle Expect: 100-continue, to disable that header, add the argument -H 'Expect:'
(an empty Expect header will make curl omit the header entirely, instead of sending the header empty)
... but to answer the question in the title, yeah that's the --limit-rate argument.

Firefox web push "Invalid URL endpoint"

I try to send webpush to firefox
curl -i -X PUT https://updates.push.services.mozilla.com/push/gAAAAABW5EzHyop8VZSH2jm9LJ7W8ybH3ISlbZHDGnd4RwW7h2Jb0IGTuSsP2BCoBxl0kJp-kXXL164xNzhxkTEztP1-IqVf9040VOEuy_htb1nnp-24W-RGgWgjtGK1kZYAb1k3xmAS
HTTP/1.1 400 Bad Request
Access-Control-Allow-Headers: content-encoding,encryption,crypto-key,ttl,encryption-key,content-type,authorization
Access-Control-Allow-Methods: POST,PUT
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: location,www-authenticate
Content-Type: application/json
Date: Tue, 15 Mar 2016 13:04:44 GMT
Server: cyclone/1.1
Content-Length: 51
Connection: keep-alive
{"errno": 102, "code": 400, "error": "Bad Request"}
Does it mean that I have invalid registration id stored in my database and I should remove it?
The endpoint URL doesn't seem valid, it's usually something like https://updates.push.services.mozilla.com/push/v1/SOME_LONG_ID (note the v1 that your URL doesn't contain).
Indeed, this works:
curl -i -X PUT https://updates.push.services.mozilla.com/push/v1/gAAAAABW5EzHyop8VZSH2jm9LJ7W8ybH3ISlbZHDGnd4RwW7h2Jb0IGTuSsP2BCoBxl0kJp-kXXL164xNzhxkTEztP1-IqVf9040VOEuy_htb1nnp-24W-RGgWgjtGK1kZYAb1k3xmAS
Note that you might want to add the TTL header, otherwise your request might fail (you just need -H "TTL: 60"): https://blog.mozilla.org/services/2016/02/20/webpushs-new-requirement-ttl-header/.

MongoDB with Ruby driver how to create fields with curl?

I am sending the following curl request to my Ruby Driver
curl -i -X POST -H "Content-Type:application/json" -d '{ "firstName" : "Frodo", "lastName" : "Baggins" }' http://localhost:4567/new_document/?
This is the code for the POST operation in ruby.
post '/new_document/?' do
content_type :json
db = settings.mongo_db
result = db.insert_one params
db.find(:_id => result.inserted_id).to_a.first.to_json
end
I get the following response in the console.
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 43
X-Content-Type-Options: nosniff
Server: WEBrick/1.3.1 (Ruby/2.0.0/2013-11-22)
Date: Tue, 10 Nov 2015 18:38:59 GMT
Connection: Keep-Alive
{"_id":{"$oid":"564239c3e89bde194d000007"}}
As you can see the fields first and last name never get created. What am I doing wrong?
Figured it out! This creates the correct fields and their values.
curl -d 'name=adam&last=hoffman' http://localhost:4567/new_document/?

Django-rest-framework token auth doesn't work

I'm trying to POST json data to url, decorated with login_required, but django returns redirect to login page
DRF setup:
'DEFAULT_AUTHENTICATION_CLASSES': (
'rest_framework.authentication.SessionAuthentication',
'rest_framework.authentication.TokenAuthentication',
),
and rest_framework.authtoken in INSTALLED_APPS
I can obtain auth token via curl
$ curl -X POST -d "{\"username\" : 7, \"password\" : 1}" -H "Content-Type: application/json" http://127.0.0.1:9000/extapi/get-auth-token/
{"token":"bc61497d98bed02bd3a84af2235365d0b2b549ff"}
But when i POST to the view, decorated with login_required, django returns http 302 with Location header pointing to the login page.
$ curl -v -X POST -d '{"event":"14","user":"7","action":"1868","unit":"","value":"-1"}' -H "Content-Type: application/json" -H "Authorization: Token bc61497d98bed02bd3a84af2235365d0b2b549ff" http://127.0.0.1:9000/zk2015/events/actions/api/uservotejournal/7/
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9000 (#0)
> POST /zk2015/events/actions/api/uservotejournal/7/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:9000
> Accept: */*
> Content-Type: application/json
> Authorization: Token bc61497d98bed02bd3a84af2235365d0b2b549ff
> Content-Length: 64
>
* upload completely sent off: 64 out of 64 bytes
< HTTP/1.1 302 FOUND
* Server nginx/1.4.6 (Ubuntu) is not blacklisted
< Server: nginx/1.4.6 (Ubuntu)
< Date: Fri, 18 Sep 2015 11:14:31 GMT
< Content-Type: text/html; charset=utf-8
< Location: http://127.0.0.1:9000/accounts/login/?next=/zk2015/events/actions/api/uservotejournal/7/
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Cookie
< X-Frame-Options: SAMEORIGIN
< ETag: "d41d8cd98f00b204e9800998ecf8427e"
< Set-Cookie: csrftoken=G85fWrKKsIA5a2uGPIn9fS4pqKrS51jK; expires=Fri, 16-Sep-2016 11:14:31 GMT; Max-Age=31449600; Path=/
<
* Connection #0 to host 127.0.0.1 left intact
I've tried to set breakpoints in rest_framework.authentication.SessionAuthentication and rest_framework.authentication.TokenAuthentication, but they were never fired
What is wrong in my setup? Help, please.
You are not passing the Authorization in Header in the curl
curl -X POST -d "{\"username\" : 7, \"password\" : 1}" -H "Authorization: Token bc61497d98bed02bd3a84af2235365d0b2b549ff" http://127.0.0.1:9000/extapi/get-auth-token/
The point is that request.user is AnonymousUser in drf.APIView.dispatch(), but is defined as authorized user in drf.APIView.post() and other similar methods.
This differs from django: request.user is defined as authorized user in django.views.View.dispatch()
Also that is the cause, why django.contrib.auth.decorators.login_required is not compatible whith drf views.

Resources