I know if a HTTP server support CONNECT method, it could create a TCP tunnel, then we can use all protocols based on TCP.
Here is my confusion, is HTTPS proxy means a HTTP proxy support CONNECT method or a HTTP proxy over SSL?
Here is my confusion, is HTTPS proxy means a HTTP proxy support
CONNECT method or a HTTP proxy over SSL?
I think the exact meaning depends on the context. I read almost all related posts and notice that most people see "HTTPS proxy" as HTTP proxy that supports CONNECT method. And Many of them mistakenly think that there's no such thing as connecting to a proxy a server over SSL. While some others take "HTTPS proxy" as a Man-in-the-middle attack type of proxy server.
But the way connecting to a HTTP proxy server over SSL does exist. Check my question and answer here HTTPs proxy server only works in SwitchOmega , you can deploy one by yourself with tens of lines of code.
While proxy servers are usually accessed through http (meaning HTTPS_PROXY would reference an HTTP URL), note that curl 7.52+ (Dec. 2016) does support HTTPS proxy and SOCKS+HTTP(s).
That means you can also an HTTPS_PROXY referencing an https URL for a proxy.
Git 2.16 (Q1 2018) acknowledges that with commit 82b6803 (19 Dec 2017) by Wei Shuyu (``).
(Merged by Junio C Hamano -- gitster -- in commit fc4a226, 28 Dec 2017)
Git has been taught to support an https:// URL used for http.proxy
when using recent versions of libcurl.
http: support CURLPROXY_HTTPS
HTTP proxy over SSL is supported by curl since 7.52.0.
This is very useful for networks with protocol whitelist.
With Git 2.27 (Q2 2020), a handful of options to configure SSL when talking to proxies have been added.
See commit af02651, commit 88238e0 (04 Mar 2020) by Jorge Lopez Silva (jalopezsilva).
(Merged by Junio C Hamano -- gitster -- in commit aaa6255, 25 Mar 2020)
http: add client cert support for HTTPS proxies
Signed-off-by: Jorge Lopez Silva
Git supports performing connections to HTTPS proxies, but we don't support doing mutual authentication with them (through TLS).
Add the necessary options to be able to send a client certificate to the HTTPS proxy.
A client certificate can provide an alternative way of authentication instead of using 'ProxyAuthorization' or other more common methods of authentication.
Libcurl supports this functionality already, so changes are somewhat minimal.
The feature is guarded by the first available libcurl version that supports these options.
4 configuration options are added and documented, cert, key, cert password protected and CA info. The CA info should be used to specify a different CA path to validate the HTTPS proxy cert.
The Documentation/config/http.txt now includes:
http.proxySSLCert:
The pathname of a file that stores a client certificate to use to authenticate with an HTTPS proxy.
http.proxySSLKey:
The pathname of a file that stores a private key to use to authenticate with an HTTPS proxy.
http.proxySSLCertPasswordProtected:
Enable Git's password prompt for the proxy SSL certificate.
Otherwise OpenSSL will prompt the user, possibly many times, if the certificate or private key is encrypted.
http.proxySSLCAInfo:
Pathname to the file containing the certificate bundle that should be used to verify the proxy with when using an HTTPS proxy.
Proxy server handles HTTPS requests from clients is always called
HTTPS proxy server. It's similar with HTTP proxy server, the only
difference is the protocols they focus on. No matter HTTP or HTTPS
proxy server, they both can carry out caching of information
downloaded from the Internet.
More on Brief Introduction on HTTPS Proxy Server
Related
I am currently working on GO app and I wanted to know how can I detect someone using reverse proxy apps such as Charles, Wireshark and Fiddler.
I tried to read about SSL-Pinning but I couldn't find anything useful.
Charles and Fiddler are no reverse proxies but forward proxies. TLS interception in such proxies can be detected based on the certificate returned - it will not be the original server certificate but one created by the server. And certificate validation will fail unless the proxy is specifically trusted by importing the proxies CA certificate. SSL pinning helps too since it expects the server to use a specific certificate or a specific CA - which the SSL intercepting proxy cannot provide.
Wireshark is no kind of proxy but passive packet capturing, i.e. it makes no changes to the traffic. Because it is passive it cannot be detected from inside the client application and SSL pinning would not help either. Wireshark can not decrypt TLS traffic though by its own - it would need to have the connection specific secrets from either inside the client application or inside the server application.
I'm working on a desktop C++ application which uses OpenSSL sockets (a raw TLS socket, not HTTPS) to communicate with our server.
One of our clients are required to route their traffic through a proxy. The client is using ZScaler in Tunnel with Local Proxy mode.
In theory, it's possible to reconfigure ZScaler to force our traffic through a proxy chosen by ZScaler. However, I want to investigate solutions where our application uses the Windows OS-level proxy settings rather than relying on ZScaler configuration.
I've read this post:
openssl s_client using a proxy
But I'm uncertain whether those answers apply to my situation, because that user didn't mention whether they're using Windows or Linux, and they appear to be talking about an HTTP/HTTPS proxy. Also, that question appears to be asking about the s_client function, rather than simply creating a TLS socket to my server through a "Tunnel with Local Proxy" on Windows.
So, my questions are:
Can OpenSSL be used to open an SSL socket to a server through Tunnel with Local Proxy?
Can we make an OS call to determine the IP/socket for the Tunnel with Local Proxy configuration?
If this is possible, then I have another question: suppose we have a single proxy at 10.100.10.0:5000.
If one user in our client's office opens a socket to our server via their proxy, will a 2nd user be unable to connect from their office because they're bottlenecked at single proxy socket?
Put another way: what is the standard way of implementing proxy-awareness for a Windows application using OpenSSL?
Note: This question was originally posted to Network Engineering stack exchange, but it was closed because it refers to an issue above OSI layer 4.
Note: I'm looking for a solution that does not require administrator permissions on the user PC. I would prefer for our application to discover and use OS-level proxy settings without making any administrative changes to the machine, i.e. by calling netsh.
Can OpenSSL be used to open an SSL socket to a server through Tunnel with Local Proxy?
OpenSSL doesn't do it for you but OpenSSL does not prevent it either. The tunnel has to be established before you do the TLS handshake to the endpoint. Depending on what kind of proxy this is you might need to use a HTTP CONNECT method for this or might need to use the SOCKS protocol or whatever your proxy requires. In case of ZScaler this is likely the HTTP CONNECT method but you need to make sure that the connection to the target IP and port is actually allowed by the security policy.
Once you've established the tunnel to the endpoint using the proxy you can just build the SSL socket on top of the TCP socket for the tunnel. Just do the usual SSL setup (i.e. SSL_new etc) and then associate the SSL object with the existing socket using SSL_set_fd. Then proceed as usual with the handshake, i.e. SSL_connect or similar.
Can we make an OS call to determine the IP/socket for the Tunnel with Local Proxy configuration?
I don't know but Winsock use system proxy settings might answer this part.
If one user in our client's office opens a socket to our server via their proxy, will a 2nd user be unable to connect from their office because they're bottlenecked at single proxy socket?
This should not be a problem. It is perfectly normal to have multiple connections through the proxy.
I'm trying to set up a proxy server on my local mac.
http - seems to work.
But Safari is not connecting via https.
Did I miss something?
No it doesn't. You need to specify a separate https port and a ssl certificate, as documented in the squid config:
The socket address where Squid will listen for client requests made
over TLS or SSL connections. Commonly referred to as HTTPS.
This is most useful for situations where you are running squid in
accelerator mode and you want to do the TLS work at the accelerator
level.
You may specify multiple socket addresses on multiple lines, each
with their own certificate and/or options.
The tls-cert= option is mandatory on HTTPS ports.
See http_port for a list of modes and options.
http://www.squid-cache.org/Doc/config/https_port/
By design, it is quite hard to intercept https traffic:
When a browser creates a direct secure connection with an origin
server, there are no HTTP CONNECT requests. The first HTTP request
sent on such a connection is already encrypted. In most cases, Squid
is out of the loop: Squid knows nothing about that connection and
cannot block or proxy that traffic.
You also need to load the proxy settings for the browser as a PAC file, otherwise the browsers won't connect or throw a certificate warning:
Chrome The Chrome browser is able to connect to proxies over SSL
connections if configured to use one in a PAC file or command line
switch. GUI configuration appears not to be possible (yet).
More details at
http://dev.chromium.org/developers/design-documents/secure-web-proxy
Firefox The Firefox 33.0 browser is able to connect to proxies over
TLS connections if configured to use one in a PAC file. GUI
configuration appears not to be possible (yet), though there is a
config hack for embedding PAC logic.
There is still an important bug open:
Using a client certificate authentication to a proxy:
https://bugzilla.mozilla.org/show_bug.cgi?id=209312
https://wiki.squid-cache.org/Features/HTTPS
HTTP proxy with SSL and DNS support.
I must be lacking some key concepts about proxy-ing because I cannot grasp this. I am looking to run a simply http or https proxy without interfering with SSL. Simply, a fully transparent proxy that can passthrough all the traffic to the browser connected via HTTP or HTTPS proxy without modifying or intercepting any packets. Not able to find any code online or I'm not using the right keywords.
EX. On the browser adding server.someVPN.com:80 on the HTTP proxy field and as soon as you try to visit a website, it prompts for authentication. Then it works perfectly with any domain, any security, any ssl, no further steps needed. Most VPN providers have this.
How's this possible? it even resolves DNS itself. I thought on transparent proxy the dns relies on the client. Preferably looking for a nodeJS solution but any lang works.
Please don't propose any solutions such as SOCKS5 or sock forwarding or DNS overriding or CA based MITM. According to HTTP 1.1 which supports 'CONNECT' this should be easy.
Not looking to proxy specific domains, looking for an all inclusive solution just like most VPN Providers providers.
----Found the answer too quickly, feel free to delete this post/question admins.
The way it works is that the browser knows it is talking to a proxy server, so for example if the browser want to connect to htttp://www.example.com it sends a CONNECT www.example.com:443 HTTP/1.1 to the proxy server, the proxy server resolves wwww.example.com via DNS and then opens a TCP connection to wwww.example.com port 443 and proxies the TCP stream transparently to the client.
I don't know any solution for nodejs. Common proxy servers include Squid, Privoxy and Apache Traffic Server
See also: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT
Found the solution right after I asked...
This module works perfectly https://github.com/mpangrazzi/harrier
Does exactly what I was asking for.
I've seen several examples of writing an HTTP proxy in Ruby, e.g. this gist by Torsten Becker, but how would I extend it to handle HTTPS, aka for a "man in the middle" SSL proxy?
I'm looking for a simple source code framework which I can extend for my own logging and testing needs.
update
I already use Charles, a nifty HTTPS proxy app similar to Fiddler and it is essentially what I want except that it's packaged up in an app. I want to write my own because I have specific needs for filtering and presentation.
update II
Having poked around, I understand the terminology a little better. I'm NOT after a full "Man in the Middle" SSL proxy. Instead, it will run locally on my machine and so I can honor whatever SSL cert it offers. However, I need to see the decrypted contents of packets of my requests and the decrypted contents of the responses.
Just for background information, a normal HTTP proxy handles HTTPS requests via the CONNECT method: it reads the host name and port, establishes a TCP connection to this target server on this port, returns 200 OK and then merely tunnels that TCP connection to the initial client (the fact that SSL/TLS is exchanged on top of that TCP connection is barely relevant).
This is what the do_CONNECT method if WEBrick::HTTPProxyServer.
If you want a MITM proxy, i.e. if you want to be able to look inside the SSL/TLS traffic, you can certainly use WEBrick::HTTPProxyServer, but you'll need to change do_CONNECT completely:
Firstly, your proxy server will need to embed a mini CA, capable of generating certificates on the fly (failing that, you might be able to use self-signed certificates, if you're willing to bypass warning messages in the browser). You would then import that CA certificate into the browser.
When you get the CONNECT request, you'll need to generate a certificate valid for that host name (preferable with a Suject Alt. Name for that host name, or in the Subject DN's Common Name), and upgrade the socket into an SSL/TLS server socket (using that certificate). If the browser accepts to trust that certificate, what you get from thereon on this SSL/TLS socket is the plain text traffic.
You would then have to handle the requests (get the request line, headers and entity) and take it to use it via a normal HTTPS client library. You might be able to send that traffic to a second instance of WEBrick::HTTPProxyServer, but it would have to be tweaked to make outgoing HTTPS requests instead of plain HTTP requests.
Webrick can proxy ssl:
require 'webrick'
require 'webrick/httpproxy'
WEBrick::HTTPProxyServer.new(:Port => 8080).start
from my experience HTTPS is nowhere near "simple". Do you need a proxy that would catch traffic from your own machine? There are several applications, like Fiddler. Or google for alternatives. Comes with everything you need to debug the web traffic.
That blog is no way to write a proxy. It's very easy: you just accept a connection, read one line which tells you what to connect to, attempt the upstream connection, if it fails send the appropriate response and close the socket, otherwise just start copying bytes in both directions, simultaneously, until EOS has occurred in both directions. The only difference HTTPS makes is that you have to speak SSL instead of plaintext.