I am making a HEAD request against this file location using httpie:
$ http HEAD https://dbeaver.io/files/dbeaver-ce_latest_amd64.deb
HTTP/1.1 302 Moved Temporarily
Connection: keep-alive
Content-Length: 169
Content-Type: text/html
Date: Mon, 09 Sep 2019 14:55:56 GMT
Location: https://dbeaver.io/files/6.2.0/dbeaver-ce_6.2.0_amd64.deb
Server: nginx/1.4.6 (Ubuntu)
I am only interested in the Location header as I want to store its value in a file to see if it the target was updated.
I tried:
http HEAD https://dbeaver.io/files/dbeaver-ce_latest_amd64.deb \
| grep Location \
| sed "s/Location: //"
yet this yields in an empty response.
I assume the output goes to stderr instead of stdout, though I don't really want to combine stdout and stderr for this.
I am rather looking for a solution directly with the http command.
You are missing the --header option:
http HEAD https://dbeaver.io/files/dbeaver-ce_latest_amd64.deb \
--headers \
| grep Location \
| sed "s/Location: //"
will as of this writing print:
https://dbeaver.io/files/6.2.0/dbeaver-ce_6.2.0_amd64.deb
Furthermore, your assumption that httpie would redirect to stderr is also wrong. Instead, it boils down to the automatically changing default behavior of the --print option. And it changes on the fact if the httpie was piped!
--print WHAT, -p WHAT
String specifying what the output should contain:
'H' request headers
'B' request body
'h' response headers
'b' response body
The default behaviour is 'hb' (i.e., the response headers and body
is printed), if standard output is not redirected. If the output is piped
to another program or to a file, then only the response body is printed
by default.
The --header/ -h option is merely a shortcut for --print=h.
Related
I just started using ncat, and playing around with simple HTTP requests, I came across the following:
Starting ncat and typing a two-line get request works fine:
$ ncat 192.168.56.20 80
GET / HTTP/1.1
Host: 192.168.56.20
HTTP/1.1 200 OK
If however the request gets echoed to ncat, it apparently breaks somewhere:
$ echo 'GET / HTTP/1.1\nHost: 192.168.56.20' | ncat 192.168.56.20 80
HTTP/1.1 400 Bad Request
I don't get it.
The \n in the string is sent literally. Use echo -e to enable interpretation of backslash escapes. Also, the newline sequence for HTTP 1.1 is \r\n (CRLF). And the header section ends with an additional end-of-line.
Try:
echo -e 'GET / HTTP/1.1\r\nHost: 192.168.56.20\r\n\r\n' | ncat 192.168.56.20 80
Alternatively, the ncat has the option to convert new lines to CRLF:
-C, --crlf Use CRLF for EOL sequence
Hence, you can write:
echo -e 'GET / HTTP/1.1\nHost: 192.168.56.20\n\n' | ncat -C 192.168.56.20 80
and you should get the same result.
This question already has answers here:
Bash variable gets butchered when being inserted into cURL command
(1 answer)
bash: injecting variable into string adds extra \r
(1 answer)
Closed 4 years ago.
What is going on here?
content=$(curl -s http://localhost:3000 -I | grep -Fi Content-Type | awk '{print $2}')
echo "${content}1234"
res:
1234ication/x-protobuf
the curl -I response is
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/x-protobuf
Content-Encoding: gzip
Content-Length: 10013
Date: Tue, 14 Aug 2018 18:00:06 GMT
Connection: keep-alive
as you can see, the webserver is a normal express app, no middlewares, all boilerplate, nothing special
further more, checking
if [ "$content" == "application/x-protobuf" ]
is always false (I've tried other variations of the condition as well)
What is the circular override business
why isn't content what is printed on the screen?
I need to get a nonce from a http service
I am using curl and later openssl to calculate the sha1 of that nonce.
but for that i need to get the nonce to a variable
1 step (done)
curl --user username:password -v -i -X POST http://192.168.0.202:8080/RPC3 -o output.txt -d #initial.txt
and now, the output file #output.txt holds the http reponse
HTTP/1.1 401 Unauthorized
Server: WinREST HTTP Server/1.0
Connection: Keep-Alive
Content-Length: 89
WWW-Authenticate: ServiceAuth realm="WinREST", nonce="/wcUEQOqUEoS64zKDHEUgg=="
<html><head><title>Unauthorized</title></head><body>Error 401: Unauthorized</body></html>
I have to get the position of "nonce=" and extract all the way to the " char.
How can I get in bash, the value of nonce ??
Regards
Pretty simple with grep using the -o/--only-matching and -P/--perl-regexp options (available in GNU grep):
$ grep -oP 'nonce="\K[^"]+' output.txt
/wcUEQOqUEoS64zKDHEUgg==
The -o option will print only matched part, which would normally include nonce=" if we had not used the reset match start escape sequence available in PCRE.
Additionally, if your output.txt (i.e. server response) can contain more than one nonce, and you are interested in only reading the first one, you can use the -m1 option (as Glenn suggests):
$ grep -oPm1 'nonce="\K[^"]+' output.txt
To store that nonce in a variable, simply use command substitution; or just pass it through openssl sha1 to get that digest you need:
$ nonce=$(grep -oPm1 'nonce="\K[^"]+' output.txt)
$ echo "$nonce"
/wcUEQOqUEoS64zKDHEUgg==
$ read hash _ <<<"$(grep -oPm1 'nonce="\K[^"]+' output.txt | openssl sha1 -r)"
$ echo "$hash"
2277ef32822c37b5c2b1018954f750163148edea
You can use GNU sed for this as below :
ubuntu$ cat output.txt
HTTP/1.1 401 Unauthorized
Server: WinREST HTTP Server/1.0
Connection: Keep-Alive
Content-Length: 89
WWW-Authenticate: ServiceAuth realm="WinREST", nonce="/wcUEQOqUEoS64zKDHEUgg=="
<html><head><title>Unauthorized</title></head><body>Error 401: Unauthorized</body></html>
ubuntu$ sed -E -n 's/(.*)(nonce="\/)([a-zA-Z0-9=]+)(")(.*)/\3/gp' output.txt
wcUEQOqUEoS64zKDHEUgg==
Regards!
I am trying to craft a script to perform curl requests on webservers and parse out the "server" and "Location." This way I can easily import it into my excel tables without having to reformat.
My current script:
curl -sD - -o /dev/null -A "Mozilla/4.0" http://site/ | sed -e '/Server/p' -e '/Location/!d' | paste - -
Expected/Desired output:
Server: Apache Location: http://www.site
Current output:
From curl:
HTTP/1.1 301 Moved permanently
Date: Sun, 16 Nov 2014 20:14:01 GMT
Server: Apache
Set-Cookie: USERNAME=;path=/
Set-Cookie: CFID=16581239;path=/
Set-Cookie: CFTOKEN=32126621;path=/
Location: http://www.site
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Piped into 'sed':
Server: Apache
Location: http://www.site
Piped into 'paste':
Server: Location: http://www.site
Why does paste immediately 'paste' after the first space? How do I get it to format correctly? I'm open to other methods, but keep in mind, the responses from the 'curl' request will be different lengths.
Thanks
Output of "curl" contains "return" i.e. \r character(s) which will cause that behaviour.
curl -sD - -o /dev/null -A "Mozilla/4.0" http://site/ | tr -d '\r'| sed -e '/Server/p' -e '/Location/!d' | paste - -
tr -d '\r' filters out all carriage return characters.
About line ends
While Linux/Unix uses "LF" (Line Feed, \n) line ends many other systems use "CR LF" (Carriage Return Line Feed \r\n) line ends. That can cause weard looking results unless you are prepared for it. Let's see some examples without \r and the same with \r.
Concatenation of strings:
a=$(echo -e "Please notice don't delete your files in /<config_dir> ")
b=$(echo -e "without hesitation ")
echo "$a""$b"
Result:
Please notice don't delete your files in /<config_dir> without hesitation
We get somewhat different result if lines end with CR LF:
a=$(echo -e "Please notice don't delete your files in /<config_dir> \r")
b=$(echo -e "without hesitation \r")
echo "$a""$b"
Result:
without hesitation delete your files in /<config_dir>
What might happen with programs which modify text only if matching string is at line end ?
Let's remove "ny" if it appears at line end:
echo "Stackoverflow is funny" | sed 's/ny$//g'
Result:
Stackoverflow is fun
The same wirh CR LF ending line:
echo -e "Stackoverflow is funny\r" | sed 's/ny$//g'
Result:
Stackoverflow is funny
sed works as designed because the line does not end to "ny" but "ny CR".
The teaching of all this is to be prepared for unexpected input data. In most cases it may be a good idea to filter out \r from data copletely since it's seldom needed for anything useful in BASH script. Filtering out unwanted character(s) is simple with "tr":
tr -d '\r'
Some research revealed a few useful stackexchange posts, namely expanding variable in CURL, but that given answer doesn't seem to properly handle bash variables that have spaces in them.
I am setting a variable to the output of awk, parsing a string for a substring (actually truncating to 150 characters). The string I am attempting to POST via curl has spaces in it.
When I use the following curl arguments, the POST variable Body is set to the part of the string before the first space.
curl -X POST 'https://api.twilio.com/2010-04-01/Accounts/GUID/SMS/Messages.xml' -d 'From=DIDfrom' -d 'To=DIDto' -d 'Body="'$smsbody'" -u SECGUID
smsbody is set as:
smsbody="$(echo $HOSTNAME$ $SERVICEDESC$ in $SERVICESTATE$\: $SERVICEOUTPUT$ | awk '{print substr($0,0,150)}')"
So the only portion of smsbody that is POSTed is $HOSTNAME$ (which happens to be a string without any space characters).
What is the curl syntax I should use to nest the bash variable properly to expand, but be taken as a single data field?
Seems pretty trivial, but I messed with quotes for a while without luck. I figure someone with better CLI-fu can handle it in a second.
Thanks!
It looks like you have an extra single quote before Body. You also need double quotes or the $smsbody won't be evaluated.
Try this:
curl -X POST 'https://api.twilio.com/2010-04-01/Accounts/GUID/SMS/Messages.xml' \
-d 'From=DIDfrom' -d 'To=DIDto' -d "Body=$smsbody" -u SECGUID
If the $s are still an issue (I don't think spaces are), try this to prepend a \ to them:
smsbody2=`echo $smsbody | sed 's/\\$/\\\\$/g'`
curl -X POST 'https://api.twilio.com/2010-04-01/Accounts/GUID/SMS/Messages.xml' \
-d 'From=DIDfrom' -d 'To=DIDto' -d "Body=$smsbody2" -u SECGUID
If I run nc -l 5000 and change the twilio address to localhost:5000, I see the smsbody variable coming in properly.
matt#goliath:~$ nc -l 5000POST / HTTP/1.1
Authorization: Basic U0VDR1VJRDphc2Q=
User-Agent: curl/7.21.6 (x86_64-apple-darwin10.7.0) libcurl/7.21.6 OpenSSL/1.0.0e zlib/1.2.5 libidn/1.20
Host: localhost:5000
Accept: */*
Content-Length: 45
Content-Type: application/x-www-form-urlencoded
From=DIDfrom&To=DIDto&Body=goliath$ $ in $: