How does the browser figure out the destination in an HTTPS connection? All the headers are encrypted..
Update:No this is not a homework.. my name is student because I'll always be learning in this HUGE awesome field
When a browser views an url like https://www.gmail.com/ first thing your browser does is resolves www.gmail.com to 72.14.213.19.
Next your browser opens up a TCP connection to 72.14.213.19 on port 443.
The browser & server before ANY headers are transmitted negotiate a public key encryption scheme (RSA) based on the SSL Certificate that is digitally signed.
In this process the browser checks the certificate authenticity before communicating.
Once this trust between client & server has been established, the client now can encrypt the headers in a way the server can decrypt. It proceeds to make the HTTP request inside the SSL Tunnel.
The server decrypts the message, serves the request and encrypts it in a way that that particular client can decrypt.
The browser then decrypts the response, reads the headers and makes decisions about how to proceed from there.
This has been an overview of an HTTPS connection event. :D
The string containing https: that you typed in the browser is what the browser uses. That is not encrypted,
Probably more than you'd ever want or need to know about the The First Few Milliseconds of an HTTPS Connection I'd go into more detail explaining stuff if your username wasn't student, and this wasn't likely homework.
Related
I've read a lot of related topics in the net, but I still don't have an answer to my question.
Is it possible to implement flow described below?
Proxy receive request.
If request is encrypted and proxy cert is trusted then intercept.
If request is not encrypted, then intercept.
If request is encrypted and proxy cert is NOT trusted then pass it through without interception.
This behaviour should be default for all traffic going through the proxy.
It'd be also really nice to be able to get all possible info for passing encrypted requests (src and dst ip addresses etc.). Basically the same info which I can get with fiddler.
Not really. The main problem is that mitmproxy can not know if proxy cert is trusted by the client or not.
In the SSL/TLS protocol client starts with the CLIENT_HELLO and in response the server (in this case motmproxy) sends back the SERVER_HELLO message containing the generated server certificate.
The client now checks if the received server certificate is trusted. If not the connection is terminated. As far as I know the SSL/TLS spec does not define how to do so. Sems clients end back an SSL_ALERT message, other simply drop the connection, and a third group continues the SSL/TLS handshake but have certain internal values set in a way that always let the handshake fail.
There is a mitmproxy script that tries to identify connections that were not successful and then if the client asks for the same domain a second time bypasses interception.
Of course this requires that the client resends requests which is not always the case.
https://github.com/sociam/x-ray/blob/master/mitmproxy/examples/tls_passthrough.py
I was trying to make a website that require the user to log in to do something, but I want to know the advantage and disadvantage from HTTP and HTTPS first.
I was using a program called Fiddler that allowed you to logs all HTTP(s) traffic between your computer and the Internet
if I try to log in with the program on, I could see the username and the password that I used to log in to the website, even if it's HTTP or HTTPS using fiddler
so what's the use of HTTPS compared with HTTP?
This is what I am thinking.
The browser is supposed to enscrypt the password using the server's public key right? Then the server will descript it with the private key.
But fiddler doesn't know the server's private key. So how can it sees the plain password?
Am I wrong?
In HTTPS communication is sent over an encrypted channel, while HTTP is sent in plain-text. Most importantly his means that a 3rd party can't read information sent between the server and the browser just by sniffing network traffic, but it has other uses as well, such as ensuring that the server is who it says it is and you are who you say you are with certificates.
Fiddler2 is only able to decipher the traffic with the user's cooperation: the certificates Fiddler presents to the client are only trusted by the browser if you configure your Operating System to trust Fiddler's root certificate.
I've seen several examples of writing an HTTP proxy in Ruby, e.g. this gist by Torsten Becker, but how would I extend it to handle HTTPS, aka for a "man in the middle" SSL proxy?
I'm looking for a simple source code framework which I can extend for my own logging and testing needs.
update
I already use Charles, a nifty HTTPS proxy app similar to Fiddler and it is essentially what I want except that it's packaged up in an app. I want to write my own because I have specific needs for filtering and presentation.
update II
Having poked around, I understand the terminology a little better. I'm NOT after a full "Man in the Middle" SSL proxy. Instead, it will run locally on my machine and so I can honor whatever SSL cert it offers. However, I need to see the decrypted contents of packets of my requests and the decrypted contents of the responses.
Just for background information, a normal HTTP proxy handles HTTPS requests via the CONNECT method: it reads the host name and port, establishes a TCP connection to this target server on this port, returns 200 OK and then merely tunnels that TCP connection to the initial client (the fact that SSL/TLS is exchanged on top of that TCP connection is barely relevant).
This is what the do_CONNECT method if WEBrick::HTTPProxyServer.
If you want a MITM proxy, i.e. if you want to be able to look inside the SSL/TLS traffic, you can certainly use WEBrick::HTTPProxyServer, but you'll need to change do_CONNECT completely:
Firstly, your proxy server will need to embed a mini CA, capable of generating certificates on the fly (failing that, you might be able to use self-signed certificates, if you're willing to bypass warning messages in the browser). You would then import that CA certificate into the browser.
When you get the CONNECT request, you'll need to generate a certificate valid for that host name (preferable with a Suject Alt. Name for that host name, or in the Subject DN's Common Name), and upgrade the socket into an SSL/TLS server socket (using that certificate). If the browser accepts to trust that certificate, what you get from thereon on this SSL/TLS socket is the plain text traffic.
You would then have to handle the requests (get the request line, headers and entity) and take it to use it via a normal HTTPS client library. You might be able to send that traffic to a second instance of WEBrick::HTTPProxyServer, but it would have to be tweaked to make outgoing HTTPS requests instead of plain HTTP requests.
Webrick can proxy ssl:
require 'webrick'
require 'webrick/httpproxy'
WEBrick::HTTPProxyServer.new(:Port => 8080).start
from my experience HTTPS is nowhere near "simple". Do you need a proxy that would catch traffic from your own machine? There are several applications, like Fiddler. Or google for alternatives. Comes with everything you need to debug the web traffic.
That blog is no way to write a proxy. It's very easy: you just accept a connection, read one line which tells you what to connect to, attempt the upstream connection, if it fails send the appropriate response and close the socket, otherwise just start copying bytes in both directions, simultaneously, until EOS has occurred in both directions. The only difference HTTPS makes is that you have to speak SSL instead of plaintext.
When you GET
https://encrypted.google.com/search?q=%s
Is the %s query encrypted? Or just the response?
If it is not, why should Google serve its public content also with encryption?
The entire request is encrypted, including the URL, and even the command (GET). The only thing an intervening party such as a proxy server can glean is the destination address and port.
Note, however, that the Client Hello packet of a TLS handshake can advertise the fully qualified domain name in plaintext via the SNI extension (thanks #hafichuk), which is used by all modern mainstream browsers, though some only on newer OSes.
EDIT: (Since this just got me a "Good Answer" badge, I guess I should answer the entire question…)
The entire response is also encrypted; proxies cannot intercept any part of it.
Google serves searches and other content over https because not all of it is public, and you might also want to hide some of the public content from a MITM. In any event, it's best to let Google answer for themselves.
The URL itself is encrypted, so the parameters in the query string do not travel in plain across the wire.
However, keep in mind that URLs including the GET data are often logged by the webserver, whereas POST data seldom is. So if you're planning to do something like /login/?username=john&password=doe, then don't; use a POST instead.
HTTPS Establishes an underlying SSL conenction before any HTTP data is
transferred. This ensures that all URL data (with the exception of
hostname, which is used to establish the connection) is carried solely
within this encrypted connection and is protected from
man-in-the-middle attacks in the same way that any HTTPS data is.
The above is a part of a VERY comprehensive answer from Google Answers located here:
http://answers.google.com/answers/threadview/id/758002.html#answer
The portion of the URL after the host name is sent securely.
For example,
https://somewhere.com/index.php?NAME=FIELD
The /index.php?NAME=FIELD part is encrypted. The somewhere.com is not.
Everything is encrypted, but you need to remember, that your query will stay in server's logs and will be accessible to various log analysers etc (which is usually not the case with POST request).
The connection gets encrypted before the request is transmitted. So yes, the request is encrypted as well, including the query string.
I just connected via HTTPS to a website and passed a bunch of GET parameters. I then used wireshark to sniff the network. Using HTTP, the URL is sent unencrypted, which means I can easily see all the GET parameters in the URL. Using HTTPS, everything is encrypted and I can't even see which packet is the GET command, let alone its contents!
Yes, it is secure. SSL encrypts everything.
Excerpt from POST request:
POST /foo HTTP/1.1
... some other headers
Excerpt from GET request:
GET /foo?a=b HTTP/1.1
... some other headers
In both cases whatever is sent on the socket is encrypted. The fact that the client sees parameters in his browser during a GET request doesn't mean that a man in the middle would see the same.
The SSL takes place before the header parsing, this means:
Client creates Request
Request gets encrypted
Encrypted request gets transmitted to the Server
Server decrypts the Request
Request gets parsed
A Request looks something like this (can't remember the exact syntax, but this should be close enough):
GET /search?q=qwerty HTTP/1.1
Host: www.google.de
This is also why having different SSL Certificates for several hosts on the same IP are problematic, the requested Hostname is not known until decryption.
The GET request is encrypted when using HTTPS - in fact this is why secured websites need to have a unique IP address - there's no way to get the intended hostname (or virtual directory) from the request until after it's been decrypted.
There is a little confusion above:
despite what is said, it is NOT required, anymore, to have unique IP address since the SNI field tells the server which cert to use
everything is encrypted EXCEPT the SNI request which is VERY early. This tells the server which Name and therefore which certificate to use. That is, the Server Name Indicator IS sent unencrypted so that the server and the client can establish and then use a secure connection for everything else.
So, in answer to the original question. Everything EXCEPT the host name (and I guess port) is secured in BOTH directions.
To what extent, if at all, are HTTPS headers encrypted?
They are encrypted in transit through SSL. There is no special encryption dedicated to headers, HTTPS encrypts the entire message.
All headers are encrypted in HTTPS. You may want to look at some documentation on how SSL and TLS work.
SSL sets up a private authenticated link to the server and then sends an http request over that link. The evil-doer snooping on the link cant actually tell from just watching the connection that http is being used. SSL its self has some unencrypted headers that it uses to establish this connection though these should not be confused with the headers that are part of the http protocol.