This question already has answers here:
I just assigned a variable, but echo $variable shows something else
(7 answers)
Closed 4 years ago.
I am having a strange issue with storing the output of a curl into a bash variable.
If I run the following curl
curl -i -s https://google.com
I get the following output:
HTTP/2 301
location: https://www.google.com/
content-type: text/html; charset=UTF-8
date: Sat, 22 Dec 2018 03:03:59 GMT
expires: Mon, 21 Jan 2019 03:03:59 GMT
cache-control: public, max-age=2592000
server: gws
content-length: 220
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
alt-svc: quic=":443"; ma=2592000; v="44,43,39,35"
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
here.
</BODY></HTML>
Now if I try and store the same query into a bash variable
GOOGLE=$(curl -s -i https://google.com)
echo $GOOGLE
I get the following output:
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>301 Moved</TITLE></HEAD><BODY> <H1>301 Moved</H1> The document has moved here.
This is correct but the first half of the curl output is missing... does anybody know where the rest of the output has gone?
Bash nuance - needed to add double quotes around the variable when echoing
echo "$GOOGLE"
Related
I have 3 variables, $commonName, $expiryDate and $DaysRemInUnixEpoch. Each variable has 3 lines as below output. I want to display output of all 3 variables in to 3 different columns. I tried looking for solution using printf but no luck. Can anyone please advise if they have done this in the past using printf and how? Any help will be much appreciated.
Below are 3 variables output together in one column. I want to split in to 3 columns having 3 rows in each column.
bash-4.1$ echo -e "$commonName\n$expiryDate\n$daysRemInUnixEpoch"
mycertificate_mycert.mycomp.net
PSIN0P551
ROOTROOTCA
Feb 6 2022 11:57:32 GMT
Jan 9 2023 18:51:25 GMT
Mar 12 2035 18:24:54 GMT
682
1020
5465
bash-4.1$
desired output I am looking for is something like below
mycertificate_mycert.mycomp.net Feb 6 2022 11:57:32 GMT 682
PSIN0P551 Jan 9 2023 18:51:25 GMT 1020
ROOTROOTCA Mar 12 2035 18:24:54 GMT 5465
With bash (Process Substitution), paste and column:
paste -d ';' <(echo "$commonName") <(echo "$expiryDate") <(echo "$daysRemInUnixEpoch") | column -s ';' -t
Output:
mycertificate_mycert.mycomp.net Feb 6 2022 11:57:32 GMT 682
PSIN0P551 Jan 9 2023 18:51:25 GMT 1020
ROOTROOTCA Mar 12 2035 18:24:54 GMT 5465
I assume that your variables do not contain ;.
see: man paste and man column
I am making a HEAD request against this file location using httpie:
$ http HEAD https://dbeaver.io/files/dbeaver-ce_latest_amd64.deb
HTTP/1.1 302 Moved Temporarily
Connection: keep-alive
Content-Length: 169
Content-Type: text/html
Date: Mon, 09 Sep 2019 14:55:56 GMT
Location: https://dbeaver.io/files/6.2.0/dbeaver-ce_6.2.0_amd64.deb
Server: nginx/1.4.6 (Ubuntu)
I am only interested in the Location header as I want to store its value in a file to see if it the target was updated.
I tried:
http HEAD https://dbeaver.io/files/dbeaver-ce_latest_amd64.deb \
| grep Location \
| sed "s/Location: //"
yet this yields in an empty response.
I assume the output goes to stderr instead of stdout, though I don't really want to combine stdout and stderr for this.
I am rather looking for a solution directly with the http command.
You are missing the --header option:
http HEAD https://dbeaver.io/files/dbeaver-ce_latest_amd64.deb \
--headers \
| grep Location \
| sed "s/Location: //"
will as of this writing print:
https://dbeaver.io/files/6.2.0/dbeaver-ce_6.2.0_amd64.deb
Furthermore, your assumption that httpie would redirect to stderr is also wrong. Instead, it boils down to the automatically changing default behavior of the --print option. And it changes on the fact if the httpie was piped!
--print WHAT, -p WHAT
String specifying what the output should contain:
'H' request headers
'B' request body
'h' response headers
'b' response body
The default behaviour is 'hb' (i.e., the response headers and body
is printed), if standard output is not redirected. If the output is piped
to another program or to a file, then only the response body is printed
by default.
The --header/ -h option is merely a shortcut for --print=h.
This question already has answers here:
Bash variable gets butchered when being inserted into cURL command
(1 answer)
bash: injecting variable into string adds extra \r
(1 answer)
Closed 4 years ago.
What is going on here?
content=$(curl -s http://localhost:3000 -I | grep -Fi Content-Type | awk '{print $2}')
echo "${content}1234"
res:
1234ication/x-protobuf
the curl -I response is
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/x-protobuf
Content-Encoding: gzip
Content-Length: 10013
Date: Tue, 14 Aug 2018 18:00:06 GMT
Connection: keep-alive
as you can see, the webserver is a normal express app, no middlewares, all boilerplate, nothing special
further more, checking
if [ "$content" == "application/x-protobuf" ]
is always false (I've tried other variations of the condition as well)
What is the circular override business
why isn't content what is printed on the screen?
I need to get a nonce from a http service
I am using curl and later openssl to calculate the sha1 of that nonce.
but for that i need to get the nonce to a variable
1 step (done)
curl --user username:password -v -i -X POST http://192.168.0.202:8080/RPC3 -o output.txt -d #initial.txt
and now, the output file #output.txt holds the http reponse
HTTP/1.1 401 Unauthorized
Server: WinREST HTTP Server/1.0
Connection: Keep-Alive
Content-Length: 89
WWW-Authenticate: ServiceAuth realm="WinREST", nonce="/wcUEQOqUEoS64zKDHEUgg=="
<html><head><title>Unauthorized</title></head><body>Error 401: Unauthorized</body></html>
I have to get the position of "nonce=" and extract all the way to the " char.
How can I get in bash, the value of nonce ??
Regards
Pretty simple with grep using the -o/--only-matching and -P/--perl-regexp options (available in GNU grep):
$ grep -oP 'nonce="\K[^"]+' output.txt
/wcUEQOqUEoS64zKDHEUgg==
The -o option will print only matched part, which would normally include nonce=" if we had not used the reset match start escape sequence available in PCRE.
Additionally, if your output.txt (i.e. server response) can contain more than one nonce, and you are interested in only reading the first one, you can use the -m1 option (as Glenn suggests):
$ grep -oPm1 'nonce="\K[^"]+' output.txt
To store that nonce in a variable, simply use command substitution; or just pass it through openssl sha1 to get that digest you need:
$ nonce=$(grep -oPm1 'nonce="\K[^"]+' output.txt)
$ echo "$nonce"
/wcUEQOqUEoS64zKDHEUgg==
$ read hash _ <<<"$(grep -oPm1 'nonce="\K[^"]+' output.txt | openssl sha1 -r)"
$ echo "$hash"
2277ef32822c37b5c2b1018954f750163148edea
You can use GNU sed for this as below :
ubuntu$ cat output.txt
HTTP/1.1 401 Unauthorized
Server: WinREST HTTP Server/1.0
Connection: Keep-Alive
Content-Length: 89
WWW-Authenticate: ServiceAuth realm="WinREST", nonce="/wcUEQOqUEoS64zKDHEUgg=="
<html><head><title>Unauthorized</title></head><body>Error 401: Unauthorized</body></html>
ubuntu$ sed -E -n 's/(.*)(nonce="\/)([a-zA-Z0-9=]+)(")(.*)/\3/gp' output.txt
wcUEQOqUEoS64zKDHEUgg==
Regards!
I am trying to craft a script to perform curl requests on webservers and parse out the "server" and "Location." This way I can easily import it into my excel tables without having to reformat.
My current script:
curl -sD - -o /dev/null -A "Mozilla/4.0" http://site/ | sed -e '/Server/p' -e '/Location/!d' | paste - -
Expected/Desired output:
Server: Apache Location: http://www.site
Current output:
From curl:
HTTP/1.1 301 Moved permanently
Date: Sun, 16 Nov 2014 20:14:01 GMT
Server: Apache
Set-Cookie: USERNAME=;path=/
Set-Cookie: CFID=16581239;path=/
Set-Cookie: CFTOKEN=32126621;path=/
Location: http://www.site
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Piped into 'sed':
Server: Apache
Location: http://www.site
Piped into 'paste':
Server: Location: http://www.site
Why does paste immediately 'paste' after the first space? How do I get it to format correctly? I'm open to other methods, but keep in mind, the responses from the 'curl' request will be different lengths.
Thanks
Output of "curl" contains "return" i.e. \r character(s) which will cause that behaviour.
curl -sD - -o /dev/null -A "Mozilla/4.0" http://site/ | tr -d '\r'| sed -e '/Server/p' -e '/Location/!d' | paste - -
tr -d '\r' filters out all carriage return characters.
About line ends
While Linux/Unix uses "LF" (Line Feed, \n) line ends many other systems use "CR LF" (Carriage Return Line Feed \r\n) line ends. That can cause weard looking results unless you are prepared for it. Let's see some examples without \r and the same with \r.
Concatenation of strings:
a=$(echo -e "Please notice don't delete your files in /<config_dir> ")
b=$(echo -e "without hesitation ")
echo "$a""$b"
Result:
Please notice don't delete your files in /<config_dir> without hesitation
We get somewhat different result if lines end with CR LF:
a=$(echo -e "Please notice don't delete your files in /<config_dir> \r")
b=$(echo -e "without hesitation \r")
echo "$a""$b"
Result:
without hesitation delete your files in /<config_dir>
What might happen with programs which modify text only if matching string is at line end ?
Let's remove "ny" if it appears at line end:
echo "Stackoverflow is funny" | sed 's/ny$//g'
Result:
Stackoverflow is fun
The same wirh CR LF ending line:
echo -e "Stackoverflow is funny\r" | sed 's/ny$//g'
Result:
Stackoverflow is funny
sed works as designed because the line does not end to "ny" but "ny CR".
The teaching of all this is to be prepared for unexpected input data. In most cases it may be a good idea to filter out \r from data copletely since it's seldom needed for anything useful in BASH script. Filtering out unwanted character(s) is simple with "tr":
tr -d '\r'