I'am Developing bash script can detect web application firewall from header tags but i can find example like my idea ?
To find the request headers from bash you can simply use curl. If you're on windows you'll want the new windows bash shell or cygwin to run it.
There are dozens more tricks you can play with curl to get whatever you want in whatever format you want, lots of SO questions on it to answer any questions you come up with.
curl --head www.google.com
HTTP/1.1 200 OK
Date: Thu, 06 Apr 2017 02:07:00 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=100=IoNzfnVsz_oaEwIQE182ysgVSHoZYRVKjTqSQ5GqKrz1ewxwav2ae5GPo_bx0apr39Pnn4yvM5RfsmQnJ_QFmllVwS34ts-bNrvkzDFIfaokkDTo1BXHDDI69duBn1f9kx4sXJ_rcCK28og6; expires=Fri, 06-Oct-2017 02:07:00 GMT; path=/; domain=.google.com; HttpOnly
Transfer-Encoding: chunked
Accept-Ranges: none
Vary: Accept-Encoding
Here's an example of getting response headers using curl:
curl -D - www.google.com
HTTP/1.1 200 OK
Date: Thu, 06 Apr 2017 02:11:26 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=100=DrUalBDiHKiZkX0yETtowdWhEfjJy7ioPU0Fe7Wch9pbbYI8MeSbg8M42dHmwu-hKZmYUlnE7VIgLhJ_Zi6byG_PYpTu5s2KYUv9XjPeH-GfSOTSq22I2GnEqXZwhJv-Bdn0aYzCUugF9FHb3Q; expires=Fri, 06-Oct-2017 02:11:26 GMT; path=/; domain=.google.com; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Transfer-Encoding: chunked
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop=<cut the rest of the HTTP request>
Related
If I run the following:
sed 's/$/\r/g' <<EOF | netcat "www.google.com" 80
GET / HTTP/1.1
Host:www.google.com
Connection: close
EOF
I get the usual headers, followed by a hexadecimal number before the response body I haven't seen before (in this case, 5172, although another example is 51C8):
HTTP/1.1 200 OK
Date: Thu, 09 Sep 2021 20:03:53 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Set-Cookie: ...
Accept-Ranges: none
Vary: Accept-Encoding
Connection: close
Transfer-Encoding: chunked
5172
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en-GB"><head>...
In presumed this was content length, although the response body length did not match this by a large order.
What is this number?
Apache2.4 cache control header is missing when the content is not cached, is it because the request is HEAD instead of GET , this behavior was different on 2.2 were head and post was having cache control header.
curl -I https://localhost/test.html
HTTP/1.1 200 OK
Date: Sat, 14 Nov 2020 00:10:39 GMT
Server: Apache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Frame-Options: SAMEORIGIN
Content-Security-Policy: frame-ancestors 'self'
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Type: text/html;charset=utf-8
Content-Language: en
curl -I https://localhost/test.html
HTTP/1.1 200 OK
Date: Sat, 14 Nov 2020 00:25:24 GMT
Server: Apache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Frame-Options: SAMEORIGIN
Content-Security-Policy: frame-ancestors 'self'
Last-Modified: Sat, 14 Nov 2020 00:25:19 GMT
Accept-Ranges: bytes
Content-Length: 206004
Expires: Sun, 15 Nov 2020 00:25:24 GMT
Vary: Accept-Encoding
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: public, max-age=86400
Content-Type: text/html
Content-Language: en
Default behavior on Apache2.4 , when data is stored inside apache then header info is displayed or else not.
I have installed Varnish and configured it as per as the guideline.
And when I try it curl -I https://d-o-m-a-i-n.com I get following which seems its working correctly.
HTTP/1.1 200 OK
Date: Sat, 28 Mar 2020 03:17:02 GMT
Server: Apache/2.4.18 (Ubuntu)
Expires: Sun, 29 Mar 2020 03:17:03 GMT
Cache-Control: max-age=86400, public, s-maxage=86400
Pragma: cache
X-Magento-Tags: cms_b_porto_homeslider_3,store,cms_b,cms_p_91,cms_b_porto_custom_notice_new,cat_p,cat_c_p_30,cat_p_22,cat_p_1,cat_p_34,cat_p_21,cat_p_41,cat_p_11,cat_p_39,cat_p_35,cat_p_33,cms_b_porto_footer_top_1_for_5,cms_b_porto_footer_middle_1_for_5,cms_b_porto_footer_middle_2_for_5
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
X-UA-Compatible: IE=edge
Content-Type: text/html; charset=UTF-8
X-Varnish: 33268 3
Age: 13957
Via: 1.1 varnish (Varnish/5.2)
X-Cache: cached
Accept-Ranges: bytes
But when I check in Google Chrome header response I get followings where it shows Varnish is not caching and age is 0?
Accept-Ranges: bytes
Age: 0
Cache-Control: max-age=86400, public, s-maxage=86400
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 20027
Content-Type: text/html; charset=UTF-8
Date: Sat, 28 Mar 2020 07:13:35 GMT
Expires: Sun, 29 Mar 2020 07:13:35 GMT
Keep-Alive: timeout=5, max=100
Pragma: cache
Server: Apache/2.4.18 (Ubuntu)
Vary: Accept-Encoding
Via: 1.1 varnish (Varnish/5.2)
X-Cache: uncached
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Magento-Tags: cms_b_porto_homeslider_3,store,cms_b,cms_p_91,cms_b_porto_custom_notice_new,cat_p,cat_c_p_30,cat_p_1,cat_p_22,cat_p_21,cat_p_11,cat_p_34,cat_p_41,cat_p_39,cat_p_35,cat_p_33,cms_b_porto_footer_top_1_for_5,cms_b_porto_footer_middle_1_for_5,cms_b_porto_footer_middle_2_for_5
X-UA-Compatible: IE=edge
X-Varnish: 1704417
X-XSS-Protection: 1; mode=block
Do you have any ideas why that is?
How are you generating the key for caching(vcl_hash)?
Check it and make sure there is no user agent involved.
with Open::URI, I can do the following:
require 'open-uri'
#check status
open('http://google.com').status
#get entire html
open('http://google.com').read
Is it possible to get the HTTP headers of a request so things can be debugged, something like Curls' curl -I http://google.com?
$ curl -I google.com
HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Mon, 17 Dec 2012 14:28:17 GMT
Expires: Wed, 16 Jan 2013 14:28:17 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Is this possible?
Use the meta method of the virtual filehandle:
open('http://google.com'){|f| pp f.meta }
{"x-frame-options"=>"SAMEORIGIN",
"expires"=>"-1",
"p3p"=>
"CP=\"This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info.\"",
"content-type"=>"text/html; charset=ISO-8859-1",
"date"=>"Mon, 17 Dec 2012 14:37:29 GMT",
"server"=>"gws",
"x-xss-protection"=>"1; mode=block",
"set-cookie"=>
"PREF=ID=d2fb8a93d369bcd2:FF=0:TM=1355755049:LM=1355755049:S=ONVSP6n2jtluFgll; expires=Wed, 17-Dec-2014 14:37:29 GMT; path=/; domain=.google.com, NID=67=OFEvvHCOa3C6wScQCUIKfu_89oL9MSmnFjwN-u5LX_foP8NLsX7G9dq48NLVrf4WUXhqOA1jb38s0e9qeRp1Iwx_LT_N8IuF0Qi6dXVtR2zdvA86INqtfg5uNrKvxJfJ; expires=Tue, 18-Jun-2013 14:37:29 GMT; path=/; domain=.google.com; HttpOnly",
"cache-control"=>"private, max-age=0",
"transfer-encoding"=>"chunked"}
http://www.ruby-doc.org/stdlib-1.9.3/libdoc/open-uri/rdoc/OpenURI/Meta.html
In my web app I'm making a CURL call to Google's (unofficial) weather API at:
http://www.google.com/ig/api?weather=,,, ...
This works fine in all cases except when the page involved is accessed in Opera. When accessed in Opera, instead of the weather I get the following in the xml response:
301 Moved</H1>
The document has moved
here.
How can I fix this? I've seen some suggestions around the web that clearing the cache is a solution, but that hasn't worked for me. Note I'm particularly concerned with Opera Mini.
Thanks a lot.
Update 2012-06-20: Tested with Opera 12 and Google has fixed the sniffing issue it seems.
I do not think you can fix it. It would be interesting to know why Google does server-side user agent sniffing and redirects Opera to another URI. Could you paste the full weather URI, so we can test it at Opera ourselves? If do a get of the one you have given into Opera I get
<xml_api_reply version="1">
<weather module_id="0" tab_id="0" mobile_row="0" mobile_zipped="1" row="0" section="0">
<problem_cause data=""/>
</weather>
</xml_api_reply>
I also do not get a redirection, but I guess it is because the URI is not the one you have used.
% curl -sI "http://www.google.com/ig/api?weather=,,,"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=ISO-8859-1
Date: Fri, 02 Mar 2012 12:04:44 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=fe13590e95ceb98e:TM=1330689884:LM=1330689884:S=h1eocRzhNcZ_Kwoa; expires=Sun, 02-Mar-2014 12:04:44 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
More details will help, and I will complete here.
Update 2012-03-12 First I tried with a simple curl.
→ curl -sI "http://www.google.com/ig/api?weather=,,,40735500,-73986500"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=ISO-8859-1
Date: Mon, 12 Mar 2012 13:16:43 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=9bc71bbf2edb7ebb:TM=1331558203:LM=1331558203:S=K5Ew69E5IsYhA0s8; expires=Wed, 12-Mar-2014 13:16:43 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
Then with Opera User agent.
→ curl -sI -A "Opera/9.80 (Macintosh; Intel Mac OS X 10.7.3; U; fr) Presto/2.10.229 Version/11.61" "http://www.google.com/ig/api?weather=,,,40735500,-73986500"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Mon, 12 Mar 2012 13:17:47 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=54cc62619394059e:TM=1331558267:LM=1331558267:S=JRCO-WNJMUNMMHsO; expires=Wed, 12-Mar-2014 13:17:47 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
And finally with Firefox User Agent
→ curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0" "http://www.google.com/ig/api?weather=,,,40735500,-73986500"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Mon, 12 Mar 2012 13:20:09 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=ab709995945767a8:TM=1331558409:LM=1331558409:S=bom-8pa-x9gGY5Sb; expires=Wed, 12-Mar-2014 13:20:09 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
There is a no X-Content-Type-Options: nosniff in all cases and no redirection. What is the user agent you are using. You can type about:opera in the addressbar and the user agent string will appear.