I create one proxy cache using (tor + privoxy + squid) follow
How to create an advanced proxy server using Tor, Privoxy, and Squid
I try call to Proxy cache
curl -x 'my-ip:3400' -I 'https://github.global.ssl.fastly.net/images/modules/dashboard/bootcamp/octocat_setup.png'
curl -x 'my-ip:3400' -I http://hadoop.apache.org/images/hadoop-logo.jpg
https request is ok (through Pivoxy)
HTTP/1.1 200 Connection established
Proxy-Agent: Privoxy/3.0.21
HTTP/1.1 200 OK
Date: Sat, 08 Mar 2014 04:38:37 GMT
Server: GitHub.com
Content-Type: image/png
Last-Modified: Fri, 16 Aug 2013 09:42:18 GMT
Expires: Sun, 08 Mar 2015 04:38:37 GMT
Cache-Control: max-age=31536000, public
Timing-Allow-Origin: https://github.com
Content-Length: 10722
Accept-Ranges: bytes
Via: 1.1 varnish
Age: 0
X-Served-By: cache-am72-AMS
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1394253517.449177980,VS0,VE97
Vary: Accept-Encoding
but http requets not working
HTTP/1.0 200 OK
Date: Sat, 08 Mar 2014 04:39:19 GMT
Server: Apache/2.4.8-dev (Unix) mod_wsgi/3.4 Python/2.7.5 OpenSSL/1.0.1e
Last-Modified: Tue, 12 Feb 2008 17:38:57 GMT
ETag: "24e3-445f987fe69d8"
Accept-Ranges: bytes
Content-Length: 9443
Content-Type: image/jpeg
X-Cache: MISS from SVR226L-696.localdomain
X-Cache-Lookup: MISS from SVR226L-696.localdomain:3128
Via: 1.0 SVR226L-696.localdomain (squid)
Connection: keep-alive
Please suggest a specific solution
Related
I have installed Varnish and configured it as per as the guideline.
And when I try it curl -I https://d-o-m-a-i-n.com I get following which seems its working correctly.
HTTP/1.1 200 OK
Date: Sat, 28 Mar 2020 03:17:02 GMT
Server: Apache/2.4.18 (Ubuntu)
Expires: Sun, 29 Mar 2020 03:17:03 GMT
Cache-Control: max-age=86400, public, s-maxage=86400
Pragma: cache
X-Magento-Tags: cms_b_porto_homeslider_3,store,cms_b,cms_p_91,cms_b_porto_custom_notice_new,cat_p,cat_c_p_30,cat_p_22,cat_p_1,cat_p_34,cat_p_21,cat_p_41,cat_p_11,cat_p_39,cat_p_35,cat_p_33,cms_b_porto_footer_top_1_for_5,cms_b_porto_footer_middle_1_for_5,cms_b_porto_footer_middle_2_for_5
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
X-UA-Compatible: IE=edge
Content-Type: text/html; charset=UTF-8
X-Varnish: 33268 3
Age: 13957
Via: 1.1 varnish (Varnish/5.2)
X-Cache: cached
Accept-Ranges: bytes
But when I check in Google Chrome header response I get followings where it shows Varnish is not caching and age is 0?
Accept-Ranges: bytes
Age: 0
Cache-Control: max-age=86400, public, s-maxage=86400
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 20027
Content-Type: text/html; charset=UTF-8
Date: Sat, 28 Mar 2020 07:13:35 GMT
Expires: Sun, 29 Mar 2020 07:13:35 GMT
Keep-Alive: timeout=5, max=100
Pragma: cache
Server: Apache/2.4.18 (Ubuntu)
Vary: Accept-Encoding
Via: 1.1 varnish (Varnish/5.2)
X-Cache: uncached
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Magento-Tags: cms_b_porto_homeslider_3,store,cms_b,cms_p_91,cms_b_porto_custom_notice_new,cat_p,cat_c_p_30,cat_p_1,cat_p_22,cat_p_21,cat_p_11,cat_p_34,cat_p_41,cat_p_39,cat_p_35,cat_p_33,cms_b_porto_footer_top_1_for_5,cms_b_porto_footer_middle_1_for_5,cms_b_porto_footer_middle_2_for_5
X-UA-Compatible: IE=edge
X-Varnish: 1704417
X-XSS-Protection: 1; mode=block
Do you have any ideas why that is?
How are you generating the key for caching(vcl_hash)?
Check it and make sure there is no user agent involved.
I'm seeing the following warning in Chrome Developer Tools when I access my JHipster application at http://www.jhipster-book.com. I don't see this error when running locally.
Failed to decode downloaded font: http://www.jhipster-book.com/assets/fonts/sourcesanspro/sourcesanspro.woff
I'm using these same fonts on a client's project (that runs on GAE) and it works fine. It looks like the content-type header is not being set on Heroku. Here's what I see locally:
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/2.7.9
Date: Fri, 12 Jun 2015 21:57:23 GMT
Content-type: application/x-font-woff
Content-Length: 27248
Last-Modified: Fri, 12 Sep 2014 16:22:14 GMT
And on Heroku:
HTTP/1.1 200 OK
Connection: keep-alive
Server: Apache-Coyote/1.1
X-Application-Context: application:prod,heroku:56079
Cache-Control: max-age=2678400000, public
Pragma: cache
Expires: Mon, 13 Jul 2015 21:59:11 GMT
Last-Modified: Fri, 12 Jun 2015 21:47:41 GMT
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
Content-Length: 27247
Date: Fri, 12 Jun 2015 21:59:11 GMT
Via: 1.1 vegur
Is there something I have to configure in Heroku for this to work?
I have configured Hadoop HDFS with security enabled using Kerberos (CDH4.3). It works fine, I can do all the operations using command line tools. There are some other non hadoop cluster nodes need to access HDFS file system thru REST API, so I have webhdfs enabled with Kerberos configured.
My secure webhdfs works fine for operations that don't involve temporary redirect: ie LISTSTATUS, create directory, remove directory, remove files, all work fine. But for two step operations like create file, open file, it failed with below info:
[DEV][root#namenode ~]# curl -i --negotiate -u : -X
PUT "http: //namenode:50070/webhdfs/v1/tmp/hosts.txt?op=CREATE"
HTTP/1.1 401
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 0
Server: Jetty(6.1.26.cloudera.2)
HTTP/1.1 307 TEMPORARY_REDIRECT
Cache-Control: no-cache
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
Set-Cookie:
hadoop.auth="u=hdfs&p=hdfs#UNIXKRB&t=kerberos&e=1398429948267&s=XhEp/tfs5Pfp04Dp
9yy1moFPnEo=";Path=/
Location: http: //datanode.net:1006/webhdfs/v1/tmp/hosts.txt?
op=CREATE&delegation=HgAEaGRmcwRoZGZzAIoBRZbGSGyKAUW60sxsWo4BAhTfG8IRyDJjJ2BSR7K
hveMEo3V4dxJXRUJIREZTIGRlbGVnYXRpb24NMC4wLjAuMDo1MDA3MA&namenoderpcaddress=
namenode.net:8020&overwrite=false
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(6.1.26.cloudera.2)
[DEV][root#namenode~]# curl -i -X PUT -T hosts.txt
http: //datanode.net:1006/webhdfs/v1/tmp/hosts.txt?
op=CREATE&delegation=HgAEaGRmcwRoZGZzAIoBRZbGSGyKAUW60sxsWo4BAhTfG8IRyDJjJ2
BSR7KhveMEo3V4dxJXRUJIREZTIGRlbGVnYXRpb24NMC4wLjAuMDo1MDA3MA&namenoderpcaddress=
namenode.net:8020&overwrite=false
[2] 19047
[3] 19048
[4] 19049
[DEV][root#namenode~]# HTTP/1.1 100 Continue
HTTP/1.1 401 Unauthorized
Cache-Control: no-cache
Expires: Fri, 25 Apr 2014 02:46:26 GMT
Date: Fri, 25 Apr 2014 02:46:26 GMT
Pragma: no-cache
Expires: Fri, 25 Apr 2014 02:46:26 GMT
Date: Fri, 25 Apr 2014 02:46:26 GMT
Pragma: no-cache
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26.cloudera.2)
{"RemoteException":
{"exception":"SecurityException","javaClassName":"java.lang.SecurityException",
"message":"Failed to obtain user group information: java.io.IOException:
Security enabled but user not authenticated by filter"}}
[2] Done curl -i -X PUT -T hosts.txt
http://datanode.net:1006/webhdfs/v1/tmp/hosts.txt?op=CREATE
[3] Done
delegation=HgAEaGRmcwRoZGZzAIoBRZbGSGyKAUW60sxsWo4BAhTfG8IRyDJjJ2BSR7KhveM
Eo3V4dxJXRUJIREZTIGRlbGVnYXRpb24NMC4wLjAuMDo1MDA3MA
[4]- Done namenoderpcaddress=namenode.net:8020
[DEV][root#namenode ~]#
Is there anyone who can shed some lights?
Thanks a lot,
In my case it was working fine as below:
$ curl -i --negotiate -u : -X PUT "http://nn30.my.dom:50070/webhdfs/v1/tmp/hosts.txt?op=CREATE"
HTTP/1.1 401
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 0
Server: Jetty(7.6.10.v20130312)
HTTP/1.1 307 Temporary Redirect
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
Cache-Control: no-cache
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
Set-Cookie: hadoop.auth="u=gpadmin&p=gpadmin#MY.DOM&t=kerberos&e=1398897564735&s=p8haj7KXAsUqj5A2WzoE5VxJYA8=";Path=/
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Location: http ://dn33.my.dom:1006/webhdfs/v1/tmp/hosts.txt?op=CREATE&delegation=IgAHZ3BhZG1pbgdncGFkbWluAIoBRbKli0OKAUXWsg9DAxIUt8i6BG2xt5V4wB4vPpXSPm8MMVMSV0VCSERGUyBkZWxlZ2F0aW9uEzE5Mi4xNjguMTAuMzA6NTAwNzA&namenoderpcaddress=nn30.my.dom:8020&overwrite=false
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(7.6.10.v20130312)
$ curl -i -X PUT -T hosts.txt "http://dn33.my.dom:1006/webhdfs/v1/tmp/hosts.txt?op=CREATE&delegation=IgAHZ3BhZG1pbgdncGFkbWluAIoBRbKli0OKAUXWsg9DAxIUt8i6BG2xt5V4wB4vPpXSPm8MMVMSV0VCSERGUyBkZWxlZ2F0aW9uEzE5Mi4xNjguMTAuMzA6NTAwNzA&namenoderpcaddress=nn30.my.dom:8020&overwrite=false"
HTTP/1.1 100 Continue
HTTP/1.1 201 Created
Expires: Wed, 30 Apr 2014 12:42:24 GMT
Date: Wed, 30 Apr 2014 12:42:24 GMT
Pragma: no-cache
Cache-Control: no-cache
Expires: Wed, 30 Apr 2014 12:42:24 GMT
Date: Wed, 30 Apr 2014 12:42:24 GMT
Pragma: no-cache
Location: webhdfs ://nn30.my.dom:50070/tmp/hosts.txt
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(7.6.10.v20130312)
NOTE - if your screen capture was not the edited/modified you had few syntax issues ("") and typo things. I bet you missed "" for the http input from first command.
check here for your info - http ://hadoop.apache.org/docs/r1.0.4/webhdfs.html
BTW, I used Pivotal HD, there shouldn't be that much difference though.
with Open::URI, I can do the following:
require 'open-uri'
#check status
open('http://google.com').status
#get entire html
open('http://google.com').read
Is it possible to get the HTTP headers of a request so things can be debugged, something like Curls' curl -I http://google.com?
$ curl -I google.com
HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Mon, 17 Dec 2012 14:28:17 GMT
Expires: Wed, 16 Jan 2013 14:28:17 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Is this possible?
Use the meta method of the virtual filehandle:
open('http://google.com'){|f| pp f.meta }
{"x-frame-options"=>"SAMEORIGIN",
"expires"=>"-1",
"p3p"=>
"CP=\"This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info.\"",
"content-type"=>"text/html; charset=ISO-8859-1",
"date"=>"Mon, 17 Dec 2012 14:37:29 GMT",
"server"=>"gws",
"x-xss-protection"=>"1; mode=block",
"set-cookie"=>
"PREF=ID=d2fb8a93d369bcd2:FF=0:TM=1355755049:LM=1355755049:S=ONVSP6n2jtluFgll; expires=Wed, 17-Dec-2014 14:37:29 GMT; path=/; domain=.google.com, NID=67=OFEvvHCOa3C6wScQCUIKfu_89oL9MSmnFjwN-u5LX_foP8NLsX7G9dq48NLVrf4WUXhqOA1jb38s0e9qeRp1Iwx_LT_N8IuF0Qi6dXVtR2zdvA86INqtfg5uNrKvxJfJ; expires=Tue, 18-Jun-2013 14:37:29 GMT; path=/; domain=.google.com; HttpOnly",
"cache-control"=>"private, max-age=0",
"transfer-encoding"=>"chunked"}
http://www.ruby-doc.org/stdlib-1.9.3/libdoc/open-uri/rdoc/OpenURI/Meta.html
In my web app I'm making a CURL call to Google's (unofficial) weather API at:
http://www.google.com/ig/api?weather=,,, ...
This works fine in all cases except when the page involved is accessed in Opera. When accessed in Opera, instead of the weather I get the following in the xml response:
301 Moved</H1>
The document has moved
here.
How can I fix this? I've seen some suggestions around the web that clearing the cache is a solution, but that hasn't worked for me. Note I'm particularly concerned with Opera Mini.
Thanks a lot.
Update 2012-06-20: Tested with Opera 12 and Google has fixed the sniffing issue it seems.
I do not think you can fix it. It would be interesting to know why Google does server-side user agent sniffing and redirects Opera to another URI. Could you paste the full weather URI, so we can test it at Opera ourselves? If do a get of the one you have given into Opera I get
<xml_api_reply version="1">
<weather module_id="0" tab_id="0" mobile_row="0" mobile_zipped="1" row="0" section="0">
<problem_cause data=""/>
</weather>
</xml_api_reply>
I also do not get a redirection, but I guess it is because the URI is not the one you have used.
% curl -sI "http://www.google.com/ig/api?weather=,,,"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=ISO-8859-1
Date: Fri, 02 Mar 2012 12:04:44 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=fe13590e95ceb98e:TM=1330689884:LM=1330689884:S=h1eocRzhNcZ_Kwoa; expires=Sun, 02-Mar-2014 12:04:44 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
More details will help, and I will complete here.
Update 2012-03-12 First I tried with a simple curl.
→ curl -sI "http://www.google.com/ig/api?weather=,,,40735500,-73986500"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=ISO-8859-1
Date: Mon, 12 Mar 2012 13:16:43 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=9bc71bbf2edb7ebb:TM=1331558203:LM=1331558203:S=K5Ew69E5IsYhA0s8; expires=Wed, 12-Mar-2014 13:16:43 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
Then with Opera User agent.
→ curl -sI -A "Opera/9.80 (Macintosh; Intel Mac OS X 10.7.3; U; fr) Presto/2.10.229 Version/11.61" "http://www.google.com/ig/api?weather=,,,40735500,-73986500"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Mon, 12 Mar 2012 13:17:47 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=54cc62619394059e:TM=1331558267:LM=1331558267:S=JRCO-WNJMUNMMHsO; expires=Wed, 12-Mar-2014 13:17:47 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
And finally with Firefox User Agent
→ curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0" "http://www.google.com/ig/api?weather=,,,40735500,-73986500"
HTTP/1.1 200 OK
Content-Type: text/xml; charset=UTF-8
Date: Mon, 12 Mar 2012 13:20:09 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Set-Cookie: PREF=ID=ab709995945767a8:TM=1331558409:LM=1331558409:S=bom-8pa-x9gGY5Sb; expires=Wed, 12-Mar-2014 13:20:09 GMT; path=/; domain=.google.com
X-Content-Type-Options: nosniff
Server: igfe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
There is a no X-Content-Type-Options: nosniff in all cases and no redirection. What is the user agent you are using. You can type about:opera in the addressbar and the user agent string will appear.