Is there any different between HTTP tunnel and HTTP proxy?
In some software I see two fields for proxy either HTTP and HTTP tunnel.
I see the traffic of some http proxy but I cant understand the different between http proxy and http tunnel.
The terms are often intermixed, tunnel providers are called proxies.
Originally, tunneling is the technique of using one protocol to transport data inside another protocol.
A proxy (as in proxy representative)
A proxy acts as an intermediary. It will hide your IP address from the destination (unless it adds it in a HTTP header field such as "Forward"). A proxy uses the same protocol throughout, it can alter the network flow, do caching or security scanning etc. So it's more of an extra hop on the way to the destination.
For example you can use a SOCKS proxy as a HTTP tunnel, i.e. you transport HTTP over it. This is due to the fact that SOCKS is a protocol that is designed to tunnel IP packets.
To add to the confusion, you can use a HTTP proxy to transport some other protocols such as FTP.
A good example for a tunnel is a VPN. Tunnels are often used to evade censorship or firewall rules blocking traffic.
Related
Our environment requires that all requests to the public internet go through a proxy that uses HTTP tunneling (HTTP Connect). I have a service from a vendor that is not able to use an HTTP Connect proxy.
My plan is to set up a transparent proxy that the service can send traffic to, which will redirect traffic to the HTTP Connect proxy.
service -> transparent proxy -> http connect proxy -> public internet
Is it possible to redirect this traffic without decrypting and re-encrypting the HTTPS request? I won't be able to install any custom certs on the initial service to allow this (and I don't want to view the contents of the HTTP messages anyway). Also, are there any proxies that are made specifically to do this, or will I need to write my own?
When an HTTP proxy server is used, is the number of connections negotiated between the client and the proxy reduced as compared to the client connecting directly to various http sites directly (without proxy)?
For example, when connecting directly to two different domains, it is clear that at least two connections must be made. In the case of a proxy, does the client usually use a single connect to the proxy for both "connections"?
Similarly, are there cases where a client that connecting to a single domain but accessing several resources would see a reduced number of connections using a proxy? E.g., can the proxy present a HTTP/1.1-style persistent connect even when the ultimate destination doesn't support it? Are proxies able to use longer persistent connection timeout periods?
In the case of a proxy, does the client usually use a single connect to the proxy for both "connections"?
While it would possible to use the same connection to a HTTP proxy to include HTTP requests to different targets most clients don't do it from what I've seen. Also, it would only work with HTTP and not HTTPS since in the latter case the whole TLS connection to the target is tunneled through the proxy and the close of this tunneled connection is also the close of the underlying TCP connection to the proxy. And, HTTP requests to multiple targets would only be possible with a HTTP proxy but not a SOCKS proxy since SOCKS essentially builds a tunnel to a specific target and this target is set at the beginning of the connection and can never be changed.
That said, while I've not seen it for browser to proxy connections I've seen a patched squid used (long ago) to do this in order to optimize proxy to proxy connections.
E.g., can the proxy present a HTTP/1.1-style persistent connect even when the ultimate destination doesn't support it?
While this would be possible too it is also not common. Usually the proxy does not fully decouple client and server, i.e. a server-triggered close of the connection between server and proxy usually results in close of the connection between proxy and client too. The reason is probably that it would work for only for HTTP anyway and not HTTPS and that it makes the implementation of the proxy more complex since things like repeating a request on sudden close of a persistent connection by the server between requests would now need to be handled by the proxy instead of simply forwarding the close and let the client deal with it.
There are several different approaches to security for web proxying:
No security. Client connects to proxy with HTTP, makes GET request (proxy probably uses HTTP to reach destination).
HTTPS connection to proxy, then GET request. Encryption only between proxy and client.
HTTP to proxy, CONNECT tunnel to destination (host named in clear-text), letting client HTTPS to destination. Security-wise, seems equivalent to direct HTTPS.
HTTPS to proxy, CONNECT tunnel to destination, then HTTPS to destination. (E.g., so that not even the destination host name metadata can be eavesdropped on an untrusted local network like public wifi.)
My question is about the final case. Is there any standard or RFC that describes it?
Does/must the content get double-encrypted or not? That is, on the client side, should the proxy simply forward the raw data stream (without manipulation) after the CONNECT is established, or should it still continue applying its (now-redundant) secure wrapper?
I'm building a WebClient library. Now I'm implementing a proxy feature, so I am making some research and I saw some code using the CONNECT method to request a URL.
But checking it within my web browser, it doesn't use the CONNECT method but calls the GET method instead.
So I'm confused. When I should use both methods?
TL;DR a web client uses CONNECT only when it knows it talks to a proxy and the final URI begins with https://.
When a browser says:
CONNECT www.google.com:443 HTTP/1.1
it means:
Hi proxy, please open a raw TCP connection to google; any following
bytes I write, you just repeat over that connection without any
interpretation. Oh, and one more thing. Do that only if you talk to
Google directly, but if you use another proxy yourself, instead you
just tell them the same CONNECT.
Note how this says nothing about TLS (https). In fact CONNECT is orthogonal to TLS; you can have only one, you can have other, or you can have both of them.
That being said, the intent of CONNECT is to allow end-to-end encrypted TLS session, so the data is unreadable to a proxy (or a whole proxy chain). It works even if a proxy doesn't understand TLS at all, because CONNECT can be issued inside plain HTTP and requires from the proxy nothing more than copying raw bytes around.
But the connection to the first proxy can be TLS (https) although it means a double encryption of traffic between you and the first proxy.
Obviously, it makes no sense to CONNECT when talking directly to the final server. You just start talking TLS and then issue HTTP GET. The end servers normally disable CONNECT altogether.
To a proxy, CONNECT support adds security risks. Any data can be passed through CONNECT, even ssh hacking attempt to a server on 192.168.1.*, even SMTP sending spam. Outside world sees these attacks as regular TCP connections initiated by a proxy. They don't care what is the reason, they cannot check whether HTTP CONNECT is to blame. Hence it's up to proxies to secure themselves against misuse.
A CONNECT request urges your proxy to establish an HTTP tunnel to the remote end-point.
Usually is it used for SSL connections, though it can be used with HTTP as well (used for the purposes of proxy-chaining and tunneling)
CONNECT www.google.com:443
The above line opens a connection from your proxy to www.google.com on port 443.
After this, content that is sent by the client is forwarded by the proxy to www.google.com:443.
If a user tries to retrieve a page http://www.google.com, the proxy can send the exact same request and retrieve response for him, on his behalf.
With SSL(HTTPS), only the two remote end-points understand the requests, and the proxy cannot decipher them. Hence, all it does is open that tunnel using CONNECT, and lets the two end-points (webserver and client) talk to each other directly.
Proxy Chaining:
If you are chaining 2 proxy servers, this is the sequence of requests to be issued.
GET1 is the original GET request (HTTP URL)
CONNECT1 is the original CONNECT request (SSL/HTTPS URL or Another Proxy)
User Request ==CONNECT1==> (Your_Primary_Proxy ==CONNECT==> AnotherProxy-1 ... ==CONNECT==> AnotherProxy-n) ==GET1(IF is http)/CONNECT1(IF is https)==> Destination_URL
As a rule of thumb GET is used for plain HTTP and CONNECT for HTTPS
There are more details though so you probably want to read the relevant RFC-s
http://www.ietf.org/rfc/rfc2068.txt
http://www.ietf.org/rfc/rfc2817.txt
The CONNECT method converts the request connection to a transparent TCP/IP tunnel, usually to facilitate SSL-encrypted communication (HTTPS) through an unencrypted HTTP proxy.
A Web Socket detects the presence of a proxy server and automatically sets up a tunnel to pass through the proxy. The tunnel is established by issuing an HTTP CONNECT statement to the proxy server, which requests for the proxy server to open a TCP/IP connection to a specific host and port. Once the tunnel is set up, communication can flow unimpeded through the proxy. Since HTTP/S works in a similar fashion, secure Web Sockets over SSL can leverage the same HTTP CONNECT technique. [1]
OK, sounds useful! But, in the client implementations I've seen thus far (Go [2], Java [3]) I do not see anything related to proxy detection.
Am I missing something or are these implementations just young? I know WebSockets is extremely new and client implementations may be equally young and immature. I just want to know if I'm missing something about proxy detection and handling.
[1] http://www.kaazing.org/confluence/display/KAAZING/What+is+an+HTML+5+WebSocket
[2] http://golang.org/src/pkg/websocket/client.go
[3] http://github.com/adamac/Java-WebSocket-client/raw/master/src/com/sixfire/websocket/WebSocket.java
Let me try to explain the different success rates you may have encountered. While the HTML5 Web Socket protocol itself is unaware of proxy servers and firewalls, it features an HTTP-compatible handshake so that HTTP servers can share their default HTTP and HTTPS ports (80 and 443) with a Web Sockets gateway or server.
The Web Socket protocol defines a ws:// and wss:// prefix to indicate a WebSocket and a WebSocket Secure connection, respectively. Both schemes use an HTTP upgrade mechanism to upgrade to the Web Socket protocol. Some proxy servers are harmless and work fine with Web Sockets; others will prevent Web Sockets from working correctly, causing the connection to fail. In some cases additional proxy server configuration may be required, and certain proxy servers may need to be upgraded to support Web Sockets.
If unencrypted WebSocket traffic flows through an explicit or a transparent proxy server on its way the WebSocket server, then, whether or not the proxy server behaves as it should, the connection is almost certainly bound to fail today (in the future, proxy servers may become Web Socket aware). Therefore, unencrypted WebSocket connections should be used only in the simplest topologies.
If encrypted WebSocket connection is used, then the use of Transport Layer Security (TLS) in the Web Sockets Secure connection ensures that an HTTP CONNECT command is issued when the browser is configured to use an explicit proxy server. This sets up a tunnel, which provides low-level end-to-end TCP communication through the HTTP proxy, between the Web Sockets Secure client and the WebSocket server. In the case of transparent proxy servers, the browser is unaware of the proxy server, so no HTTP CONNECT is sent. However, since the wire traffic is encrypted, intermediate transparent proxy servers may simply allow the encrypted traffic through, so there is a much better chance that the WebSocket connection will succeed if Web Sockets Secure is used. Using encryption, of course, is not free, but often provides the highest success rate.
One way to see it in action is to download and install the Kaazing WebSocket Gateway--a highly optimized, proxy-aware WebSocket gateway, which provides native WebSocket support as well as a full emulation of the standard for older browsers.
The answer is that these clients simply do not support proxies.
-Occam
The communication channel is already established by the time the WebSocket protocol enters the scene. The WebSocket is built on top of TCP and HTTP so you don't have to care about the things already done by these protocols, including proxies.
When a WebSocket connection is established it always starts with a HTTP/TCP connection which is later "upgraded" during the "handshake" phase of WebSocket. At this time the tunnel is established so the proxies are transparent, there's no need to care about them.
Regarding websocket clients and transparent proxies,
I think websocket client connections will fail most of the time for the following reasons (not tested):
If the connection is in clear, since the client does not know it is communicating with a http proxy server, it won't send the "CONNECT TO" instruction that turns the http proxy into a tcp proxy (needed for the client after the websocket handshake). It could work if the proxy supports natively websocket and handles the URL with the ws scheme differently than http.
If the connection is in SSL, the transparent proxy cannot know to which server it should connect to since it has decrypt the host name in the https request. It could by either generating a self-signed certificate on the fly (like for SSLStrip) or providing its own static certificate and decrypt the communication but if the client validates the server certificate it will fail (see https://serverfault.com/questions/369829/setting-up-a-transparent-ssl-proxy).
You mentioned Java proxies, and to respond to that I wanted to mention that Java-Websocket now supports proxies.
You can see the information about that here: http://github.com/TooTallNate/Java-WebSocket/issues/88
websocket-client, a Python package, supports proxies, at the very least over secure scheme wss:// as in that case proxy need no be aware of the traffic it forwards.
https://github.com/liris/websocket-client/commit/9f4cdb9ec982bfedb9270e883adab2e028bbd8e9