This question already has answers here:
Bash variable gets butchered when being inserted into cURL command
(1 answer)
bash: injecting variable into string adds extra \r
(1 answer)
Closed 4 years ago.
What is going on here?
content=$(curl -s http://localhost:3000 -I | grep -Fi Content-Type | awk '{print $2}')
echo "${content}1234"
res:
1234ication/x-protobuf
the curl -I response is
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/x-protobuf
Content-Encoding: gzip
Content-Length: 10013
Date: Tue, 14 Aug 2018 18:00:06 GMT
Connection: keep-alive
as you can see, the webserver is a normal express app, no middlewares, all boilerplate, nothing special
further more, checking
if [ "$content" == "application/x-protobuf" ]
is always false (I've tried other variations of the condition as well)
What is the circular override business
why isn't content what is printed on the screen?
Related
I am making a HEAD request against this file location using httpie:
$ http HEAD https://dbeaver.io/files/dbeaver-ce_latest_amd64.deb
HTTP/1.1 302 Moved Temporarily
Connection: keep-alive
Content-Length: 169
Content-Type: text/html
Date: Mon, 09 Sep 2019 14:55:56 GMT
Location: https://dbeaver.io/files/6.2.0/dbeaver-ce_6.2.0_amd64.deb
Server: nginx/1.4.6 (Ubuntu)
I am only interested in the Location header as I want to store its value in a file to see if it the target was updated.
I tried:
http HEAD https://dbeaver.io/files/dbeaver-ce_latest_amd64.deb \
| grep Location \
| sed "s/Location: //"
yet this yields in an empty response.
I assume the output goes to stderr instead of stdout, though I don't really want to combine stdout and stderr for this.
I am rather looking for a solution directly with the http command.
You are missing the --header option:
http HEAD https://dbeaver.io/files/dbeaver-ce_latest_amd64.deb \
--headers \
| grep Location \
| sed "s/Location: //"
will as of this writing print:
https://dbeaver.io/files/6.2.0/dbeaver-ce_6.2.0_amd64.deb
Furthermore, your assumption that httpie would redirect to stderr is also wrong. Instead, it boils down to the automatically changing default behavior of the --print option. And it changes on the fact if the httpie was piped!
--print WHAT, -p WHAT
String specifying what the output should contain:
'H' request headers
'B' request body
'h' response headers
'b' response body
The default behaviour is 'hb' (i.e., the response headers and body
is printed), if standard output is not redirected. If the output is piped
to another program or to a file, then only the response body is printed
by default.
The --header/ -h option is merely a shortcut for --print=h.
This question already has answers here:
I just assigned a variable, but echo $variable shows something else
(7 answers)
Closed 4 years ago.
I am having a strange issue with storing the output of a curl into a bash variable.
If I run the following curl
curl -i -s https://google.com
I get the following output:
HTTP/2 301
location: https://www.google.com/
content-type: text/html; charset=UTF-8
date: Sat, 22 Dec 2018 03:03:59 GMT
expires: Mon, 21 Jan 2019 03:03:59 GMT
cache-control: public, max-age=2592000
server: gws
content-length: 220
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
alt-svc: quic=":443"; ma=2592000; v="44,43,39,35"
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
here.
</BODY></HTML>
Now if I try and store the same query into a bash variable
GOOGLE=$(curl -s -i https://google.com)
echo $GOOGLE
I get the following output:
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>301 Moved</TITLE></HEAD><BODY> <H1>301 Moved</H1> The document has moved here.
This is correct but the first half of the curl output is missing... does anybody know where the rest of the output has gone?
Bash nuance - needed to add double quotes around the variable when echoing
echo "$GOOGLE"
I need to get a nonce from a http service
I am using curl and later openssl to calculate the sha1 of that nonce.
but for that i need to get the nonce to a variable
1 step (done)
curl --user username:password -v -i -X POST http://192.168.0.202:8080/RPC3 -o output.txt -d #initial.txt
and now, the output file #output.txt holds the http reponse
HTTP/1.1 401 Unauthorized
Server: WinREST HTTP Server/1.0
Connection: Keep-Alive
Content-Length: 89
WWW-Authenticate: ServiceAuth realm="WinREST", nonce="/wcUEQOqUEoS64zKDHEUgg=="
<html><head><title>Unauthorized</title></head><body>Error 401: Unauthorized</body></html>
I have to get the position of "nonce=" and extract all the way to the " char.
How can I get in bash, the value of nonce ??
Regards
Pretty simple with grep using the -o/--only-matching and -P/--perl-regexp options (available in GNU grep):
$ grep -oP 'nonce="\K[^"]+' output.txt
/wcUEQOqUEoS64zKDHEUgg==
The -o option will print only matched part, which would normally include nonce=" if we had not used the reset match start escape sequence available in PCRE.
Additionally, if your output.txt (i.e. server response) can contain more than one nonce, and you are interested in only reading the first one, you can use the -m1 option (as Glenn suggests):
$ grep -oPm1 'nonce="\K[^"]+' output.txt
To store that nonce in a variable, simply use command substitution; or just pass it through openssl sha1 to get that digest you need:
$ nonce=$(grep -oPm1 'nonce="\K[^"]+' output.txt)
$ echo "$nonce"
/wcUEQOqUEoS64zKDHEUgg==
$ read hash _ <<<"$(grep -oPm1 'nonce="\K[^"]+' output.txt | openssl sha1 -r)"
$ echo "$hash"
2277ef32822c37b5c2b1018954f750163148edea
You can use GNU sed for this as below :
ubuntu$ cat output.txt
HTTP/1.1 401 Unauthorized
Server: WinREST HTTP Server/1.0
Connection: Keep-Alive
Content-Length: 89
WWW-Authenticate: ServiceAuth realm="WinREST", nonce="/wcUEQOqUEoS64zKDHEUgg=="
<html><head><title>Unauthorized</title></head><body>Error 401: Unauthorized</body></html>
ubuntu$ sed -E -n 's/(.*)(nonce="\/)([a-zA-Z0-9=]+)(")(.*)/\3/gp' output.txt
wcUEQOqUEoS64zKDHEUgg==
Regards!
I am trying to craft a script to perform curl requests on webservers and parse out the "server" and "Location." This way I can easily import it into my excel tables without having to reformat.
My current script:
curl -sD - -o /dev/null -A "Mozilla/4.0" http://site/ | sed -e '/Server/p' -e '/Location/!d' | paste - -
Expected/Desired output:
Server: Apache Location: http://www.site
Current output:
From curl:
HTTP/1.1 301 Moved permanently
Date: Sun, 16 Nov 2014 20:14:01 GMT
Server: Apache
Set-Cookie: USERNAME=;path=/
Set-Cookie: CFID=16581239;path=/
Set-Cookie: CFTOKEN=32126621;path=/
Location: http://www.site
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Piped into 'sed':
Server: Apache
Location: http://www.site
Piped into 'paste':
Server: Location: http://www.site
Why does paste immediately 'paste' after the first space? How do I get it to format correctly? I'm open to other methods, but keep in mind, the responses from the 'curl' request will be different lengths.
Thanks
Output of "curl" contains "return" i.e. \r character(s) which will cause that behaviour.
curl -sD - -o /dev/null -A "Mozilla/4.0" http://site/ | tr -d '\r'| sed -e '/Server/p' -e '/Location/!d' | paste - -
tr -d '\r' filters out all carriage return characters.
About line ends
While Linux/Unix uses "LF" (Line Feed, \n) line ends many other systems use "CR LF" (Carriage Return Line Feed \r\n) line ends. That can cause weard looking results unless you are prepared for it. Let's see some examples without \r and the same with \r.
Concatenation of strings:
a=$(echo -e "Please notice don't delete your files in /<config_dir> ")
b=$(echo -e "without hesitation ")
echo "$a""$b"
Result:
Please notice don't delete your files in /<config_dir> without hesitation
We get somewhat different result if lines end with CR LF:
a=$(echo -e "Please notice don't delete your files in /<config_dir> \r")
b=$(echo -e "without hesitation \r")
echo "$a""$b"
Result:
without hesitation delete your files in /<config_dir>
What might happen with programs which modify text only if matching string is at line end ?
Let's remove "ny" if it appears at line end:
echo "Stackoverflow is funny" | sed 's/ny$//g'
Result:
Stackoverflow is fun
The same wirh CR LF ending line:
echo -e "Stackoverflow is funny\r" | sed 's/ny$//g'
Result:
Stackoverflow is funny
sed works as designed because the line does not end to "ny" but "ny CR".
The teaching of all this is to be prepared for unexpected input data. In most cases it may be a good idea to filter out \r from data copletely since it's seldom needed for anything useful in BASH script. Filtering out unwanted character(s) is simple with "tr":
tr -d '\r'
I'm trying to play around with netcat to learn more about how HTTP works. I'd like to script some of it in bash or Perl, but I've hit upon a stumbling block early on in my testing.
If I run netcat straight from the prompt and type in a HEAD request, it works and I receive the headers for the web server I'm probing.
This works:
[romandas#localhost ~]$ nc 10.1.1.2 80
HEAD / HTTP/1.0
HTTP/1.1 200 OK
MIME-Version: 1.0
Server: Edited out
Content-length: 0
Cache-Control: public
Expires: Sat, 01 Jan 2050 18:00:00 GMT
[romandas#localhost ~]$
But when I put the same information into a text file and feed it to netcat through a pipe or via redirection, in preparation for scripting, it doesn't return the headers.
The text file consists of the HEAD request and two newlines:
HEAD / HTTP/1.0
Sending the same information via echo or printf doesn't work either.
$ printf "HEAD / HTTP/1.0\r\n"; |nc -n 10.1.1.2 80
$ /bin/echo -ne 'HEAD / HTTP/1.0\n\n' |nc 10.1.1.2 80
Any ideas what I'm doing wrong? Not sure if it's a bash problem, an echo problem, or a netcat problem.
I checked the traffic via Wireshark, and the successful request (manually typed) sends the trailing newline in a second packet, whereas the echo, printf, and text file methods keep the newline in the same packet, but I'm not sure what causes this behavior.
You need two lots of "\r\n", and also to tell netcat to wait for a response. printf "HEAD / HTTP/1.0\r\n\r\n" |nc -n -i 1 10.1.1.2 80 or similar should work.
Another way is to use what is called the 'heredoc' convention.
$ nc -n -i 1 10.1.1.2 80 <<EOF
> HEAD / HTTP/1.0
>
> EOF
Another way to get nc to wait for the response is to add a sleep to the input. e.g.
(printf 'GET / HTTP/1.0\r\n\r\n'; sleep 1) | nc HOST 80
You can use below netcat command to make your instance webserver:
MYIP=$(ifconfig eth0|grep 'inet addr'|awk -F: '{print $2}'| awk '{print $1}')
while true; do echo -e "HTTP/1.0 200 OK\r\n\r\nWelcome to $MYIP" | sudo nc -l -p 80 ; done&
This line will also work as equivalent:
echo -e "HEAD / HTTP/1.1\nHost: 10.1.1.2\nConnection: close\n\n\n\n" | netcat 10.1.1.2 80