I have a reverse proxy that does HTTPS on the outside, but HTTP on the inside.
This means that by default in-app URLs will have HTTP as the scheme, as this is the way it's being contacted.
How can the proxy tell the backend that HTTPS should be used?
The proxy can add extra (or overwrite) headers to requests it receives and passes through to the back-end. These can be used to communicate information to the back-end.
So far I've seen a couple used for forcing the use of https in URL scheme:
X-Forwarded-Protocol: https
X-Forwarded-Ssl: on
X-Url-Scheme: https
And wikipedia also mentions:
# a de facto standard:
X-Forwarded-Proto: https
# Non-standard header used by Microsoft applications and load-balancers:
Front-End-Https: on
This what you should add to the VirtualHost on apache: other proxies should have similar functionality
RequestHeader set X-FORWARDED-PROTOCOL https
RequestHeader set X-Forwarded-Ssl on
# etc.
I think it's best to set them all, or set one that works and remove the other known ones. To prevent evil clients messing with them.
It took me several hours of googling to find the magic setting for my environment. I have a SSL httpd Apache reverse proxy in front of a jetty app server and an apache2 http server. This answer actually gave me the information that worked. For me, adding:
RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
to the site conf file was enough for the destination to use https instead of http as the protocol when building links in the response. I tried the X-FORWARDED-PROTOCOL above, but that didn't work. Hopefully this will help in future Google searches!
Related
HTTP proxy with SSL and DNS support.
I must be lacking some key concepts about proxy-ing because I cannot grasp this. I am looking to run a simply http or https proxy without interfering with SSL. Simply, a fully transparent proxy that can passthrough all the traffic to the browser connected via HTTP or HTTPS proxy without modifying or intercepting any packets. Not able to find any code online or I'm not using the right keywords.
EX. On the browser adding server.someVPN.com:80 on the HTTP proxy field and as soon as you try to visit a website, it prompts for authentication. Then it works perfectly with any domain, any security, any ssl, no further steps needed. Most VPN providers have this.
How's this possible? it even resolves DNS itself. I thought on transparent proxy the dns relies on the client. Preferably looking for a nodeJS solution but any lang works.
Please don't propose any solutions such as SOCKS5 or sock forwarding or DNS overriding or CA based MITM. According to HTTP 1.1 which supports 'CONNECT' this should be easy.
Not looking to proxy specific domains, looking for an all inclusive solution just like most VPN Providers providers.
----Found the answer too quickly, feel free to delete this post/question admins.
The way it works is that the browser knows it is talking to a proxy server, so for example if the browser want to connect to htttp://www.example.com it sends a CONNECT www.example.com:443 HTTP/1.1 to the proxy server, the proxy server resolves wwww.example.com via DNS and then opens a TCP connection to wwww.example.com port 443 and proxies the TCP stream transparently to the client.
I don't know any solution for nodejs. Common proxy servers include Squid, Privoxy and Apache Traffic Server
See also: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT
Found the solution right after I asked...
This module works perfectly https://github.com/mpangrazzi/harrier
Does exactly what I was asking for.
Is it possible to create forwarding HTTPS Proxy (not reverse proxy) that would be able to:
block some urls based on the url regexp (ads, flash, movies, ...)
cache images based on the url regexp
It seems to me that in the usual case it is impossible because the HTTPS stream is encrypted and there's no way to process or alter it.
But, this case is special, it is a proxy for the web crawler, I don't need HTTPS at all, but some sites allow access via HTTPS only, and I have to somehow support it.
So, maybe it would be possible to do something like that?
Crawler --http--> Proxy --https--> Site
So, the proxy would be able to decode HTTPS stream and post-process it. Would it work? Is there any docs or details about such approach?
Pretty sure Apache 2.2 provides this functionality with mod_proxy in conjunction with mod_ssl and mod_cache.
Note: blocking is done using the 'ProxyBlock' directive in mod_proxy.
Can I use varnish cache with https or will this have little to no performance gain? What are the pros and cons? I've set up my vcl for http only. I want to try this with https now.
I've read this but it's from 2011:
https://www.varnish-cache.org/docs/trunk/phk/ssl.html
Varnish in itself does not support SSL and is very unlikely to do so in the overseeable future.
To use SSL and still be able to cache with varnish you have to terminate the SSL before the request is sent to varnish. This can be done efficiently by for instance HAProxy or Nginx.
To find out exactly how to configure this; a simple google search for ssl termination haproxy/nginx will yield more than enough results-
You set the X-Forwarded-For headers in HAProxy. If there is already set an X-Forwarded-For header other reverse proxies will always just add their own to it, the left-most or first address is the source address. You don't have to think about that, anything that reads and uses X-Forwarded-For headers will sort that out automagically.
You also want to set the X-Forwarded-Proto so you can do all sorts of magic in Varnish, like redirecting traffic not using TLS without hitting your backend servers and separate the caches, as Varnish doesn't talk TLS, which can lead to some interesting results, like images not being served up because they are requested over HTTP when the page is served over HTTPS.
Side question, are you using HAProxy to actually load balance between multiple backends? If not, why not just terminate the TLS connection in Apache, send that to Varnish and then back to Apache again?
How we can enable squid to cache web content (let says from firefox) for SSL connection, i mean for https URLs?
Actually SQUID can be used to access HTTPS traffic - it is in essence a man-in-the-middle attack - and there are caveats:
See: http://wiki.squid-cache.org/Features/SslBump
I have not tried cacheing this data yet, so can't say that it will work with absolute certainty. If/when I do, I'll update this post.
SSL encrypts the traffic between server and client so it cannot be read by a middle man. When using Squid as a proxy it simply cannot see the actual content in the traffic and therefore it has no means of caching it. The SSL traffic is just random bits that look different each time even if the same content is transferred multiple times and that is how encryption should work. It simply cannot be cached.
I have no problems getting Firefox (version 23.0.1 on Windows) to route SSL traffic via Squid. In Firefox Network Connection settings I just point SSL Proxy and HTTP Proxy to the same Squid installation.
After that I can successfully access https URLs in Firefox and in Squid's access_log I see entries like these:
1379660084.878 115367 10.0.0.205 TCP_MISS/200 6581 CONNECT www.gravatar.com:443 - DIRECT/68.232.35.121 -
Do you have any details about how it doesn't work for you? Squid has quite complicated possibilities to deny and allow certain types of traffic, so it is possible there is a configuration issue in Squid. Do you get any error messages in Squid's logfiles?
So I have a custom proxy that is written in ruby using mongrel to handle some fairly complex caching logic. This works great for both http and ftp requests, however since mongrel is not designed to handle https requests, I wish to front the whole thing with apache and make use of the ProxyRemote command to pass through to mongrel for https requests.
This sort of thing is easily accomplished to mirror certain site directory structures via the ProxyPass and ProxyPassReverse commands in apache, but I don't see a way to do this using ProxyRemote.
The problem is that mongrel does not handle CONNECT requests which are made to establish a secure request. So while I am able to handle https requests within the proxy itself, actually using the proxy with an https request directly is not supported.
It seems that the simplest solution would be to have apache handle the https request and then simply pass the http request itself (minus the CONNECT) to mongrel and have it handle it appropriately and return it to apache and then to the client.
So my question is, Is there a way to make ProxyRemote work the same way that ProxyPass does with HTTP requests (i.e. pass an unencrypted request to mongrel)?
Just use ProxyPass and ProxyPassReverse, the connection between your reverse proxy (apache) and your mongrel will see normal plain http :), no magic necessary (especially not CONNECT, afaik thats only possbile for forward proxies, but I'm not sure).
Hum, have you tried to do so ?
I've been using apache to do the https and just pass the requests with the old default .htaccess mod_rewrite rules.