I'm trying to extract the host name (as string) from a still encrypted HTTPS.
The host name is not encrypted due to the protocol. But I can't find the correct method to extract it. Deferent domains have deferent length and not all request are similar.
google.com (first TCP message)
�� T}#����5�;�;��O��KG��Y� ����~fM�FRH�N��7s�6w��[���ک�>�,�0�̨̩̪�+�/��$�(k�#�'g�
�9� �3��=<5/�u
google.com
3t h2http/1.11
*(
+-3&$
cϼB�Y�j ¬��b*a$��n$���}�X�.u�
example.com (first TCP message)
�ę`�ۜ����z# �X��I�&���~�� ��Ao�)���쿂�7�-�������`�l>�,�0�̨̩̪�+�/��$�(k�#�'g�
�9� �3��=<5/�u example.com
3t h2http/1.11
*(
+-3&$ a�
���a桵.3�*L_��d�N�yK
*r��
Any ideas ?
Well, we are probably talking about the TLS protocol with the SNI extension. So you basically need to have a parser that is able to understand the initial TLS packet. It is not too hard if you just implement parsing the handshake protocol, more specifically the ClientHello message. See https://www.rfc-editor.org/rfc/rfc5246 and https://www.rfc-editor.org/rfc/rfc6066#section-3.
Related
I'm using SendGrid to deliver emails and the recent test I conducted from my server, using their API, ended up having SpamAssassin flagging my email. Here is the result:
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information.
[URIs: sendgrid.net]
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 T_SPF_HELO_TEMPERROR SPF: test of HELO record failed (temperror)
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
0.0 HTML_MESSAGE BODY: HTML included in message
2.0 HTTPS_HTTP_MISMATCH BODY: No description available.
1.1 KAM_REALLYHUGEIMGSRC RAW: Spam with image tags with ridiculously huge
http urls
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
domain
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
2.6 RDNS_DYNAMIC Delivered to internal network by host with
dynamic-looking rDNS
2.0 HELO_DYNAMIC_IPADDR Relay HELO'd using suspicious hostname (IP addr 1)
X-Spam-Flag: YES
The webhosting company I'm using just came back to me saying that they fixed the problem by disabling SpamAssassin from my server. I might need a better solution.
So the main question I have is: who should I contact to get the following lines fixed? My webhost, Sendgrid or someone else?
2.6 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS
2.0 HELO_DYNAMIC_IPADDR Relay HELO'd using suspicious hostname (IP addr 1)
HELO_DYNAMIC_IPADDR indicates that the sending server connected to the receiving server and announced itself with an IP address rather than a Fully Qualified Domain Name (FQDN), and RDNS_DYNAMIC indicates that the Reverse DNS check on that IP address resolves to something that resembles the FQDN of a dynamically issued IP address.
To rectify this you would need to get the rDNS of your sending IP to resolve to a plausible FQDN, and in turn use that FQDN in the HELO / EHLO handshake. Both of these relate to the MX phase of your message exchange, and would be caused by the sending server, but maybe your sender doesn't want to announce themselves overtly to avoid other more direct SPAM rules from being applied to their perceived sender reputation ?
I would have expected SendGrid to send on your behalf using your domain name during the HELO handshake, and consequently require their sending IP ranges to be included in any SPF records for your domain.
Historically it used to be misconfigured Exchange Servers that announced the IP address during the HELO handshake, because unwitting admins entered an IP address in the FQDN field of their Send Connectors.
I'm reading the spec and trying to understand exactly when 421 might be returned. An example is given but I don't completely understand it.
Background
The spec establishes two conditions that allow for connection reuse:
For TCP connections without TLS, this depends on the host having resolved to the same IP address.
and
For https resources, connection reuse additionally depends on having a
certificate that is valid for the host in the URI.
If the certificate used in the connection has multiple subjectAltName or any of the subjectAltName is a wildcard, then the connection can be reused for any request that has a hostname that is in the list of subjectAltNames or matches any of the wildcards.
Specific Example In the spec
In some deployments, reusing a connection for multiple origins can
result in requests being directed to the wrong origin server. For
example, TLS termination might be performed by a middlebox that uses
the TLS Server Name Indication (SNI) [TLS-EXT] extension to select an
origin server. This means that it is possible for clients to send
confidential information to servers that might not be the intended
target for the request, even though the server is otherwise
authoritative.
Please explain where my understanding of this example is wrong:
An https connection is established to a middlebox with a request that has domain x.com. The middlebox has IP address 1.2.3.4 and x.com resolves to that address. Using SNI, the TLS handshake has x.com and the middlebox returns a certificate valid for that domain. All messages on this connection go from the client to the middlebox or from middlebox to client. Applicaiton level messages from client to middlebox are forwarded by middlebox to an origin on a different connection. Messages from origin to middlebox are forwarded to the client. If the connection is to be reused, meeting the two conditions discussed above is not enough. Specifically, for a request with domain y.com: if y.com resolves to 1.2.3.4 and the middlebox has a certificate valid for y.com, there can still be a problem. Because the original connection did its TLS handshake using x.com and because handshakes are only done at the beginning of new connections, there is no way establishing an https connection that would get the certificate for y.com. So the client incorrectly sends a request on the same connection to y.com. The middlebox rejects the request because the certificate associated with the connection is valid for x.com - not y.com. (The x.com certificate is only valid for x.com and the y.com certificate is only valid for y.com).
None of your examples will trigger a 421 as far as I can see.
Yes you are correct that a connection needs both the IP address and the SAN field in the certificate to be valid - without those a connection should not be reused.
So what would trigger a 421? As far as I can tell it will be mostly due to different SSL/TLS setups.
For example:
Assume website A (siteA.example.com) and website B (www.example.com) are both on same IP address. Assume website A has a wildcard cert for *.example.com and website B has a specific one. Could be a few reasons for this: for example it serves an EV cert for the main website which can't be a wildcard cert.
So cert A covers website A and website B. As does the IP address. So if you are connected to website siteA.example.com, and then try to connect to www.example.com then technically, by HTTP/2 standards, you could reuse the connection. But we wouldn't want that to happen, as we want to use our EV cert. So the server should reject with a 421. Now in this example the webserver is able to distinguish the correct host and has a valid cert for that host so could, in theory, serve the correct content under the wildcard cert, instead of sending a 421 - but since that wildcard cert is not defined for that virtualhost it should not do this.
Other examples include if you have different ciphers set up on different hosts. For example site A has super lax HTTPS config, because it's not really secure content and they want to reach even legacy browsers, but site B has super secure config and only accepts the latest TLS version and strong ciphers. Here you obviously wouldn't want them to reuse the same connection details. See here for a real would example of this.
Also this is only an issue for certain browsers, depending on how they decide to connection share. This page shows how different each of them do this (at least at the time of this blog post not not aware of anything changing since then): https://daniel.haxx.se/blog/2016/08/18/http2-connection-coalescing/
Also note that some bugs will exist with this (for example: https://bugs.chromium.org/p/chromium/issues/detail?id=546991). Best advice is: if you do not want connection sharing to happen, have a different IP address and/or ensure no overlaps in certificates.
I have a Ruby polling script that runs on a set of servers in an IP range. I very strongly prefer to do this polling by IP address, not by hostname, because:
1) I have defined IP address ranges to poll, and hostnames are arbitrary/change a lot
2) Because they change a lot, most of the hostnames do not have a reverse DNS lookup, so I can't engineer a list of hostnames from IPs
Before our web servers had no problem with this polling, but on a new server that does not accept SSLv3 communication, this is the error I get when I run my poll:
/home/dashboard/.rvm/rubies/ruby-2.1.6/lib/ruby/2.1.0/net/http.rb:923:in `connect': SSL_connect returned=1 errno=0 state=unknown state: tlsv1 unrecognized name (OpenSSL::SSL::SSLError)
On the server side, this is the error:
nss_engine_init.c(1802): start function ownSSLSNISocketConfig for SNI
nss_engine_init.c(1827): Search [val = 172.16.99.18] failed, unrecognized name
When I run the poll with hostname, everything works fine.
Here is the crux of the HTTP Client code in Ruby:
def init_http(url)
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
http.read_timeout = 10
http.use_ssl = true
#http.ssl_version = 'TLSv1'
return [http, uri]
end
As you can tell, I've been playing around with TLS and SSL version, because I figured that might be the issue. My next thought (that Google only has evidence of for Java) is, "How easy is it to just disable the SNI extension on my client?" The more general question is, "Can I keep using IP addresses with Ruby net/http while taking advantage of newer, more secure communication protocols?"
... tlsv1 unrecognized name (OpenSSL::SSL::SSLError)
This is in most cases not a problem you can solve by disabling SNI on the client side. SNI is required when you have multiple certificates on the same IP address and if you just connect by IP address and don't send the requested hostname (i.e. disabling SNI) the server will not know which certificate it should provide - which then results in the above error.
I very strongly prefer to do this polling by IP address, not by hostname, ...
If you have to deal with server which require SNI then you have to use SNI and you have to use SNI with the proper hostname, which is not necessarily the same name as you get from reverse lookup.
The easiest way to solve this is to add the patch that #steffenullrich mentioned for bug 10613.
All I did was look at the diff, and edit the file myself, but you can use the Linux patch tool if you're familiar with it.
For those who are unsure of where /net/http.rb is located, it is in the same location as the rest of your Ruby sources. For example, mine was here:
/home/myuser/.rvm/rubies/ruby-2.1.6/lib/net/http.rb
Once you patch the file, set the .disable_sni property of your HTTP object to true, and SNI will not be required, allowing the use of IP addresses in mixed-TLS communication.
I have a ruby server based on TCPSocket (non-HTTP).
I have 2 different domains, both pointing with an A-Record to my servers IP Address (the same one). So, there are clients connecting to one of those domains.
Is it possible to tell which domain a client was connecting to?
I saw that this is possible in other protocols, but I'm not sure if this is based on manually added headers or really extracted from the basic tcp/ip connection. E.g. in PHP there is $_SERVER["HTTP_HOST"] which shows to which domain a client was connecting.
At the TCP socket level, the only things that are known are the source and destination IP addresses (and ports) of the connection. How the IP address was resolved via DNS is not possible to know at this layer. Even though HTTP works on top of TCP, HTTP servers have to look at the HTTP headers from the client to know which domain they are making a request to. (That's how the HTTP_HOST value gets filled in.)
One possible solution is to configure your server to have an additional IP address. This can be by assigning an additional IP address to the NIC or adding an additional NIC. Then have each domain use a different IP address. Otherwise, this is not possible and you may want to consider your application protocol on top of TCP to convey this information.
http://ipfuck.paulds.fr/
We've been recently getting hammered by this Firefox plug-in. It sends a fake IP in the headers so when our nginx web server picks up the IP it is a fake one.
Is there any way to get a real IP address or block out requests that have this plug-in installed?
There is actually no client IP entries in any HTTP Headers. There are only some un-official proxy headers which are added to a request, so that a proxy server can tell you the real ip of the connecting client (since the tcp socket will only reveal the IP address of the proxy server).
The plugin you linked to adds those proxy headers, to "fake" a proxy request, by adding a X-Real-IP: 1.2.3.4 or X-Forwarded-For: 1.2.3.4 header to the request. But no one forces you to use that IP address (which can be fake, like the 1.2.3.4 example here), you can always use the IP address of the socket that initiated the connection - which will be the client's real IP address if he uses the mentioned plugin.
Within the location section of your nginx configuration, you get the socket IP address through the $remote_addr variable. To retrieve the "fake" IP address, you can use $http_x_forwarded_for or $http_x_real_ip variable.
If you are using any application/cgi backend, you usually can examine the full headers and the socket IP address (i.e. in PHP you should check $_REQUEST and $_HEADERS variables)